[jwrosewell]

The problem

pipeline/Gaps-vs-spec.md lists where the Java pipeline differed from the specification. Checked against the code on rc and against the web integration page, these are still true:

Item	What the code does
W1, endpoints	PipelineFilter reads one init parameter (clientside-properties-enabled). The JavaScript path (51Degrees.core.js) and JSON path (51Degrees.core.json) are constants.
W2, optional elements	The Sequence Element and the Set Headers Element are always added when the configuration lacks them, with no way to prevent it.
5b, Sequence Element output	SequenceElement writes Evidence only and declares no properties. The specification lists Element Data as well.

Item 10 (Drift as a sum) is already fixed by #60, so it is not repeated here.

The specification is quoted at commit ba079423 of specifications rc.

What changes

1. W1. The filter takes two new init parameters, javascript-path and json-path. Intercepted Urls: "These SHOULD be configurable, but the default values are: /51Degrees.core.js ... /51dpipeline/json". The JavaScript path default stays 51Degrees.core.js. The JSON path default becomes the specification's /51dpipeline/json, as .NET and Go use, and the JavaScript the filter serves calls it. The old path 51Degrees.core.json is still answered, whatever json-path says, so a site or a cached script that calls it keeps working. FiftyOneJSServiceCore.Default gains a constructor that takes the two paths, and the existing constructor uses the defaults. Constants.CORE_JSON_NAME stays, for the old path, and Constants.DEFAULT_JSON_PATH is new.

2. W2. The filter takes sequence-element-enabled and set-headers-element-enabled, both true by default. Pipeline configuration: "For all of these elements, there MUST be configuration options to allow users to prevent them being added (or disable them) if they wish to do so." A false value stops the element being added when the configuration lacks it, and an element named in the configuration stays. With client-side properties on and the Sequence Element left out the build is refused, naming the option, as .NET does, because the JSON Builder Element fails on every request without the sequence number (json-builder.md). clientside-properties-enabled works as before and still controls only the JSON Builder Element and the JavaScript Builder Element. StartupHelpers.buildFromConfiguration gains an overload with the JSON path and the two switches, and the existing overloads call it with the defaults.

3. 5b. SequenceElement also writes the Element Data session-id and sequence, under the key sequence, as .NET, Go and Python do, and declares them as properties. The page's Element Data section gives the key: "The Element Data key is sequence." Sequence Element: the Element Data table lists both, and "Add both output values to Evidence as well" keeps the Evidence writes, which are unchanged. New class SequenceData. Process treats the two values on their own, so a session id that is not a non-empty string counts as absent and gets a new id, and a sequence counts only if it is a whole number from 0 up, or text made only of the digits 0 to 9, up to 2,147,483,646 so that adding one fits in an int, and otherwise gives 1, with no error and no overflow. specifications#127, now merged, states both rules in that section: "A query.session-id value that is not a non-empty string is not present", so an empty session id gets a new one, and "A query.sequence value that is not a whole number, or a string made only of the digits 0 to 9 with no sign or spaces, from 0 up to 2,147,483,646 is not present". A negative whole number used to be taken as a sequence, and now counts as absent. An id of "0" is kept, a sequence with no session id counts up with a new id, and a session id with no sequence gives 1. The Evidence and the Element Data are written from the same values. Java used to cast the sequence with (int), which failed on text.

4. MVC integration. FiftyOneInterceptorConfig gains getJavaScriptPath, getJsonPath, getSequenceElementEnabled and getSetHeadersElementEnabled, as default methods so existing implementations keep working, and FiftyOneInterceptorConfigDefault gains the setters. The interceptor passes them to StartupHelpers and to the new FiftyOneJSService.setPaths, and FiftyOneInterceptor.enableClientsideProperties has an overload that maps the configured paths. The JSON default is the same new default as the filter's, the old path is answered and mapped too, and both elements are on.

5. Example comments. The comments in GettingStartedOnPrem.java, MetadataOnPrem.java and OfflineProcessing.java that said the Lite file has "limited accuracy" now say it has a small number of Properties, as required-examples.md says ("the Lite data file has a small number of Properties"), and so does the one in UpdateDataFile.java. These are comments only, so there is no test. A search found no other comment with that wording.

6. Usage sharing filter. EvidenceKeyFilterShareUsage.include returns false for server.http-request before any other rule, as .NET's filter does, so the whole web request that draft Accept no Evidence for usage sharing, and give it the whole web request #68 added is never a value to share. The review of Accept no Evidence for usage sharing, and give it the whole web request #68 asked for this.

7. Exact paths. FiftyOneJSServiceCore.Default answered any request whose whole URL ended with the configured path, so a configured path of json also answered /anything/json. The path is now compared with the whole path of the request inside the application, after the context path, ignoring case and with or without a leading slash, and a request with no URI is not answered. This compares the request path rather than the whole URL, as .NET compares Request.Path, which leaves out the application's base path. .NET still uses EndsWith on that path, so the comparison here is stricter than .NET's. Intercepted Urls names the two URLs the integration intercepts and says nothing of longer paths that end the same way. The old 51Degrees.core.json path is matched the same way, and the script rc served called it at the root.

8. No JSON path from the caller. The new overload of StartupHelpers.buildFromConfiguration read jsonPath without checking it, so a caller that passed null got a null pointer exception rather than a pipeline. A null or empty path is now Constants.DEFAULT_JSON_PATH, which is what the filter and the MVC interceptor already pass in that case, so the script still calls back to a path the JSON is served from.

Does 5b add a section to the JSON sent to browsers?

Yes. The JSON Builder Element skips only the element data keys cloud-response, json-builder and set-headers, so the Sequence Element's data now appears:

"sequence": { "session-id": "...", "sequence": 1 }

JsonBuilder_SequenceElementData_InJson in JsonBuilderTests runs a Sequence Element and a JSON Builder Element in one Pipeline and reads that section. This follows json-builder.md, whose Process section says "There MUST be a top-level entry for each Element Data" and whose exclusion list names only the Set Headers Element, the Cloud Request Engine, the Usage sharing element and the JSON Builder Element itself. The Sequence Element is not on that list, so its entry, the session id included, reaches the page as the specification says it should.

Release note

The JSON that 51Degrees client-side properties read is now served from /51dpipeline/json, the path the specification names. The old path /51Degrees.core.json still works, so no change is needed to a page, a cached script or a proxy rule. A site that wants another path sets json-path in the filter, or getJsonPath in the MVC configuration. The Sequence Element's data now appears in that JSON under sequence, and the filter and the MVC integration can leave the Sequence Element and the Set Headers Element out.

Behaviour that changes for callers

• The JSON is served from /51dpipeline/json, and the script the filter and the MVC integration serve calls that path. 51Degrees.core.json is still answered. A site that sets json-path keeps its choice.
• The JSON sent to browsers gains the sequence section above.
• A site whose pages call the MVC integration is unchanged unless it sets the new options.
• A relative link to the script or the JSON from a page in a folder stops being answered. The two endpoints used to be served for any request whose path ended with 51Degrees.core.js or 51Degrees.core.json, so a page at /products/phones.html carrying <script src="51Degrees.core.js"> asked for /products/51Degrees.core.js and got the script. The paths are now matched whole, after the context path, ignoring case and a leading slash, because the specification names them as paths, Intercepted Urls: "/51Degrees.core.js - Serves the JavaScript produced by JavaScript Builder Element." Such a page links /51Degrees.core.js from the root instead, and this belongs in the release note.
• Nothing else. No public method, option or default is removed or changed. Added: SequenceData, the extra StartupHelpers.buildFromConfiguration overload, the extra FiftyOneJSServiceCore.Default constructor, the init parameter names in Constants, and the two properties on SequenceElement.

Other open drafts

Checked with git merge-tree of this head against every open draft's head.

• Accept no Evidence for usage sharing, and give it the whole web request #68, Write the usage sharing record as the specification says #81 and #93 are merged. Accept no Evidence for usage sharing, and give it the whole web request #68's accepted Evidence filter in SequenceElement is kept, and this draft's toSequence replaces its getSequence. Write the usage sharing record as the specification says #81's ReplacedString is kept as it is. Write weighted values with the lower-case keys rawweighting and value #93 added a test at the end of JsonBuilderTests, where this draft adds one, and the merge of rc (dd3bd92, rc at ff6e4c0) keeps both.
• #83 conflicts in EvidenceKeyFilterShareUsage.include. Whichever merges second keeps Add the usage sharing options to share all Evidence and all query string parameters #83's toLowerCase(Locale.ROOT) and this draft's server.http-request test ahead of Add the usage sharing options to share all Evidence and all query string parameters #83's shareAllEvidence return, so that sharing all Evidence never sends the whole web request.
• #67 also edits PipelineFilter, Constants, FiftyOneInterceptorConfig, FiftyOneInterceptorConfigDefault and FiftyOneInterceptorTests. Both add init parameters and options side by side, so whichever merges second keeps both. Cache the client-side endpoints' answers as the specification and RFC 9110 ask #67 also conflicts with rc itself.
• Authenticate a Cloud Request Engine on a license key alone, and send the properties named #97, Give each package its own version and release only what changed #63 and Build each pull request on the configurations its changes need #43 conflict with rc itself, not with this draft.

How it was checked

Tests were written first and fail on rc.

• PipelineFilterTests checks the elements the filter adds with each init parameter, and the callback endpoint.
• FiftyOneJSServiceCoreTests is new. It checks the default and configured paths. It does not compile on rc because it uses the new constructor, so the web module's tests fail to build there.
• SequenceElementTests is new, and JsonBuilderTests gains the JSON test above.
• FiftyOneInterceptorTests has four new tests for the MVC elements and paths, and NoSequence.xml is a configuration without a Sequence Element.

Run	Link	Result
Red, tests only	all tests, filter tests only	On Ubuntu and Windows: 3 of 3 SequenceElementTests, the new JsonBuilderTests test, and 3 of 9 PipelineFilterTests fail. In the first run the web module does not compile because of the new constructor, which the second run avoids by leaving FiftyOneJSServiceCoreTests out
Green	run	All pass on Ubuntu and Windows
Red, MVC tests	run	The MVC module does not compile on that head because the tests use the new options
Green, MVC	run	FiftyOneInterceptorTests (7), FiftyOneJSServiceCoreTests (4) and PipelineFilterTests (9) pass on Ubuntu and Windows
Red, key sequence	run	SequenceElementTests and the JsonBuilderTests test fail looking for the key
Green, key sequence	run	Pass on Ubuntu and Windows
Red, Sequence Element values	run	5 of 11 SequenceElementTests fail
Green, Sequence Element values	run	All 11 pass, with JsonBuilderTests and ShareUsageElementTests
Red, filter	run	webRequestIsNeverIncluded fails
Green, filter and merged Sequence Element	run	Pass on Java 8 and 21, with SequenceElementTests (11) and ShareUsageElementTests (22)
Earlier full build, Java 21 unit test	run	SessionIdAndSequenceTests expected an empty session id to be kept, and the Sequence Element now creates a new one. The test expects that now
Earlier full build	run	Failed on Ubuntu Java 21, where a constructor called an overridable method and the build treats that warning as an error. Fixed in a6303a4c
Red, JSON path	run	6 tests fail: the default endpoint, the served script, the old and new paths answered, and the MVC paths
Green, JSON path	run	Pass on Java 8 and Java 21
Red, refusal	run	The two refusal tests fail
Green, refusal	run	Pass on Java 8 and 21
Red, exact paths and the lower bound	run	The tests on the merge of rc without the fix. On Ubuntu and Windows with Java 8 and 21, SequenceNegativeSignedOrSpaced_CountsAsAbsent (1 of 13 SequenceElementTests) and ConfiguredPath_OnlyAnExactMatchIsAnswered and ContextPath_IsNotPartOfThePath (2 of 7 FiftyOneJSServiceCoreTests) fail
Green, exact paths and the lower bound	run	On Ubuntu and Windows with Java 8 and 21, all 13 SequenceElementTests, 7 FiftyOneJSServiceCoreTests, 11 PipelineFilterTests, 9 FiftyOneInterceptorTests, 32 SessionIdAndSequenceTests, 10 EvidenceKeyFilterShareUsageTests, 7 ClientsidePropertyServiceTests and 5 UsageSharingEvidenceTests pass
Red, no JSON path	run	The new test on head dd3bd92 without the fix. On Java 8 and 21, StartupHelpers_NoJsonPath_IsTheDefault fails with "java.lang.NullPointerException: Cannot invoke "String.startsWith(String)" because "jsonPath" is null", 1 error in 12 PipelineFilterTests
Green, no JSON path	run	On Java 8 and 21, all 12 PipelineFilterTests, 13 SequenceElementTests and 7 FiftyOneJSServiceCoreTests pass
Full pull request build	run	Head fb4f384, the current head. All 9 jobs without macOS passed, being Ubuntu Java 8, 11, 17 and 21, Ubuntu ARM Java 17, and Windows Java 8, 11, 17 and 21. In the Ubuntu Java 11 job the Selenium contract tests pass 9 of 9 against the cloud example and 9 of 9 against the on-premise example. Every Setting 'PullRequestSha' line, in Configure and in each of the 9 jobs, is fb4f384, and Configure printed "Left out 6 macOS entries"
Earlier full build	run	Head dd3bd92, before the fix for a caller with no JSON path. All 9 jobs without macOS passed: Ubuntu Java 8, 11, 17 and 21, Ubuntu ARM Java 17, and Windows Java 8, 11, 17 and 21. The Selenium contract tests pass, 9 of 9 against the cloud example and 9 of 9 against the on-premise example in the Ubuntu Java 11 job. In the first attempt Windows Java 11 reached its 120 minute limit as its Selenium contract tests began, after slow IP Intelligence tests on that runner, and it passed when re-run alone. Every Setting 'PullRequestSha' line, in Configure and in each of the 9 jobs, is dd3bd92, the head, and Configure printed "Left out 6 macOS entries"
Earlier full build, before the refusal and the merge of #81	run	Head 2d741ef, and every Setting 'PullRequestSha' line matches. The Ubuntu Java 11 job passes the unit tests and the example tests, then fails at the Selenium contract tests against the cloud and on-premise examples ("the first page must call the json endpoint at least once"), which stopped the other jobs. See "What remains"

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Every exit path of SequenceElement.processInternal, SequenceElement.toSequence, StartupHelpers.buildFromConfiguration, FiftyOneJSServiceCore.Default.serveJS and FiftyOneJSServiceCore.Default.serveJson was read after the change.

What remains

• Nothing in the code. The earlier full build failed at the shared Selenium contract tests, which gave Java the JSON path /51Degrees.core.json while the Java examples now call /51dpipeline/json. selenium-api-tests#23 accepts both paths for Java, and selenium-api-tests#24 has merged it into that repository's main, which the full build runs the contract tests from.