[jwrosewell]

Brings the Node pipeline builder and Sequence Element in line with the Web integration page and the Sequence Element page of the pipeline specification (both read in full at e89679e9, the specifications rc head, where every sentence quoted below still reads as quoted). Each change below was first checked against the code on rc.

The data update changes are in #52. Open drafts #45 and #46 change one line of sequenceElement.js (its accepted evidence, in the constructor). This draft changes the super(...) call just above it and the processInternal method below it, so the two should merge with a trivial conflict at most, depending on which lands first.

What changed

W2. A switch for each element.
Spec, Pipeline configuration: "For all of these elements, there MUST be configuration options to allow users to prevent them being added (or disable them) if they wish to do so."
Before, addJavaScriptBuilder turned the Sequence, JSON Builder and JavaScript Builder elements off together and useSetHeaderProperties turned Set Headers off. PipelineBuilder now also takes addSequenceElement, addJsonBuilderElement, addJavaScriptBuilderElement and addSetHeadersElement, each true by default. addJavaScriptBuilder: false and useSetHeaderProperties: false work exactly as before (they leave out their elements whatever the new switches say), so no existing configuration changes. The types are updated.

Sequence first.
Spec, Pipeline configuration: "The Sequence Element MUST be present. If not, it MUST be added as the first element in the Pipeline."
Before, addRequiredElements put it after the user's elements. It is now first, so the session id and sequence it writes to the evidence are there for every element. The JSON Builder and JavaScript Builder keep their place after the user's elements, and Set Headers after those.

Session id and sequence are read on their own.
Spec, Sequence Element, Process: a new session id is generated "if query.session-id is not present", and the sequence is the given one plus one, or one "if it is not present". The element now treats them separately, as the merged PHP change does. A session id that is not a non-empty string counts as absent and gets a new id. A sequence counts as present only when it is a whole number, or a string made only of the digits 0 to 9, leading zeros allowed, from 0 to 2147483646, and gets one added. Anything else, including an array, an object, a negative number, "abc" and NaN, counts as absent and gives 1. So a sequence with no session id now counts up, and a session id with no sequence gives 1 where it gave 2. NaN is never written.

5b. Sequence Element Data.
Spec, Sequence Element, Element Data: session-id (string) and sequence (int). Process: "Add both output values to Evidence as well".
The Sequence Element now declares both properties and sets them on its element data, as well as writing the same two values to the evidence as before. The values are the ones written to the evidence, so the way a session id and a sequence are worked out is unchanged.

Does this add a sequence section to the JSON sent to browsers? Yes, as the JSON Builder page requires.
Spec, JSON Builder, Process: "There MUST be a top-level entry for each Element Data", and only the Set Headers Element, Cloud Request Engine and Usage Sharing Element are excluded. jsonbundler.js had sequence in its exclusion list, so it is removed, and the Cloud Request Engine (cloud) and Usage Sharing (shareUsage) elements are added to it, with a test. The entry is under the key sequence, the Sequence Element's data key (the page names no key, and .NET, Go and Python use sequence). It holds session-id and sequence, for example "sequence": {"session-id": "abc-123", "sequence": 5}. One existing test that spells out the whole JSON was updated.

One more visible change follows from the element data. The "There is no element data for ..." error lists the elements that do have data, and sequence is now in that list (first, as it runs first). Two existing tests that spell out the list were updated.

Switch names. The Web integration page says only that there must be configuration options, and names none, so the names addSequenceElement, addJsonBuilderElement, addJavaScriptBuilderElement and addSetHeadersElement follow the existing addJavaScriptBuilder style.

Lite data file comments. Seventeen example comments (9 Device Detection and 6 IP Intelligence found by searching for "limited accuracy", and the two updatedatafile.js comments that said "limited capabilities") now say the Lite data file has "a small number of Properties", as the specification does, instead of "limited accuracy and capabilities".

Two effects of putting Sequence first, both found by the full build.

1. The on-premise engines passed every item in the evidence to the native evidence object, and the numeric query.sequence the Sequence Element adds made it fail ("argument 3 of type 'std::string const &'"). The Device Detection and IP Intelligence engines now pass only the evidence their own key filter accepts, which is the list of keys the native engine reads. The first full build of this draft, without that change, failed 18 tests in 9 suites for this reason: https://github.com/51Degrees/51degrees-node/actions/runs/42662212293
2. The usage sharing element runs after the Sequence Element, so the sequence it shares is the one the Sequence Element has added one to. A request that arrives with sequence 1 is shared as 2, and a request with no session now shares the new session id and a sequence of 1, where it shared neither before. The existing test that checks the shared sequence now builds its pipeline without the Sequence Element, and a new test covers the default pipeline. The format of the usage record is not changed. The cloud request engine also sends the new session id and sequence to the cloud service in the same way as it already sent ones that arrived on the request.

Tests

Written first, in pipeline/fiftyone.pipeline.core/tests/webIntegrationElements.test.js.

• Red, on rc code with the tests only, head 19b3ce9 (11 of 13 fail, and the two that pass are existing behaviours the draft must keep): https://github.com/51Degrees/51degrees-node/actions/runs/42652516081
• Red for the JSON section, tests only: https://github.com/51Degrees/51degrees-node/actions/runs/42668251106 . Red for the existing JSON builder test, which failed on the earlier head: https://github.com/51Degrees/51degrees-node/actions/runs/42668500112
• Red for the session id and sequence rules and the JSON exclusions (many cases fail), tests only: https://github.com/51Degrees/51degrees-node/actions/runs/42676590529
• Green, lint and the core, engines and usage sharing tests on the head 9701d3f: https://github.com/51Degrees/51degrees-node/actions/runs/42676738390
• The cloud cache and credential tests (cloudCache.test.js, cloudRequestEngineCredentials.test.js) build their pipelines with addSequenceElement: false, in 409c524 and 484eccd. They process with an engine that has not fetched its evidence keys, so its cache key is built from every Evidence value, the new session id included. A new test, serves a repeated request from the cache in a default pipeline, covers the behaviour rather than the test. It builds a default pipeline, so the Sequence Element is first, has the engine fetch its keys as a cloud engine does when it is added, processes the same request twice, checks the two session ids differ, and checks the cloud was asked once. It pins behaviour this change keeps rather than a fix, because the keys the cloud lists hold no session id, so it has no red run.
• Full build on the head 015e6cd, success, run 42986694165. Configure and the Ubuntu, Ubuntu ARM and Windows jobs on Node 22 and 24 pass, 1049 of 1051 tests with 2 skipped in each, Configure printed "Left out 2 macOS entries", and all 7 Setting 'PullRequestSha' lines are 015e6cd65ea94c09814a94529ad8c6a0217e6005.
  The full build this body cited before, run 42684289757, was of 9701d3f, sixteen commits back. The one after it, run 42907092247 on 484eccd, failed one test of 1050 on Windows Node 22 only, updateOnStartMissingFile.test.js "asks for the data file with no If-Modified-Since header", by exceeding jest's 10 second timeout. No cause beyond the timeout was found. This draft changes nothing in the data update service or its tests, only comments in the update examples, and the same test passed on all six jobs of run 42986694165. Since then the draft has taken rc twice, the second time for the current cxx submodule in 6d29452, and gained the cache test in 015e6cd.
  macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Not done here

• The page says the Pipeline must contain these elements and add them only if absent. The builder still adds its own even when the user has added one of the same kind. Not changed here.
• A pipeline holding a Cloud Request Engine with a cache and no cloud engine never fetches the evidence keys, because only a cloud engine calls ready() (cloudEngine.js line 93). Its cache key is then built from every Evidence value (engine.js lines 90 to 92), so with the Sequence Element first each request has a new key and is never served from the cache. The Start-up activity section says the Engine resolves its accepted Evidence when it is built, which Node does not do. That is a separate change, and it decides the cache key for #72 as well.
• #59 overlaps this draft, as both put the Sequence Element first and give it Element Data. This draft merges first. Fail in the JSON Builder when no Sequence Element has produced a sequence number #59 then takes rc and is rebuilt on top of it, so that its diff holds only what this draft does not, chiefly the JSON Builder's error when no sequence number exists.