[jwrosewell]

The problem

The Cloud Request Engine sends each Evidence value under its key with the prefix removed, and resolves a clash between prefixes by precedence. It kept the form fields in a HashMap, which compares names by their exact letters, so one name given in two letter cases under two prefixes, such as header.User-Agent and query.user-agent, became two fields, User-Agent and user-agent, and both were sent. The cloud service then got two values for one field with no way to know which applied.

The other prefixes were also ranked by their spelling alone, so fiftyone beat location, which beat server, where the specification ranks server first.

What the specification says

evidence.md at e89679e9:

Keys MUST be case-insensitive and will usually be split into <prefix>.<field>.

The prefixes are listed as query, header, cookie, server, fiftyone and location, and then:

The prefixes above are in order from highest to lowest precedence. I.e. if the same Evidence value is available with two different prefixes (For example, header.user-agent and query.user-agent), then the entry whose prefix is earlier in the list above MUST be used.

If any other prefixes are present then their order of precedence is alphabetical.

cloud-request-engine.md, Processing, at e89679e9:

When Evidence is added, its prefix MUST be removed. For example, query.user-agent becomes user-agent. This means that conflicts can occur when there are Evidence values for the same key with different prefixes. Where there are conflicts, the precedence order defined in Evidence MUST be used to determine which value to send to the remote server.

What changes

In pipeline/pipeline.cloudrequestengine/src/main/java/fiftyone/pipeline/cloudrequestengine/flowelements/CloudRequestEngineDefault.java:

• getFormData keeps the form fields in a TreeMap ordered by String.CASE_INSENSITIVE_ORDER, so one name in two letter cases is one field.
• When a key of higher precedence replaces a field, addFormData removes the old field and puts the new one, so the name sent is the winning key's as well as its value. A name with no clash is sent as it was given, as before.
• addFormData orders the keys it adds lowest precedence first by the new otherPrefixRank, being server, fiftyone and location as evidence.md lists them, then any other prefix alphabetically, where before it ordered them by key alone. The conflict warning now finds conflicting keys without regard to case.
• Two other prefixes are compared as prefixes, in lower case, and the whole key only breaks a tie (commit 1f9e4d5). Comparing whole keys put ab-c.n before ab.n, because a hyphen sorts before a dot.

Python, since 51degrees-python#99, and Go already send a name once by this precedence. Python also compares two other prefixes the same way, by the prefix in lower case and then by the whole key (other_prefix_rank in cloudrequestengine.py), and Go compares the prefixes in lower case. .NET has the same fault as Java, fixed in 51degrees-dotnet#136.

Behaviour that changes for callers

• A request whose Evidence holds one name in two letter cases under two prefixes now sends one field, the winner's, where it sent two.
• A request whose Evidence holds one name under two of server, fiftyone and location now sends the server value over fiftyone, and fiftyone over location. These clashes are rare, as those prefixes carry different names in practice.

How it was checked

getFormData_OneNameInTwoCases_SentOnceByPrecedence in CloudRequestEngineTests.java adds two Evidence values and reads the form data the engine builds. It needs exactly one field of that name, ignoring case, with the winner's name and value, for the same ten rows as .NET: header against query both ways round, cookie against header, location against server, location against fiftyone, fiftyone against server, server against an unknown prefix, two unknown prefixes, ab against ab-c, and _x against a.

Each test run is a throwaway branch with a temporary workflow running CloudRequestEngineTests on Ubuntu for Java 8 and 21.

Run	Link	Result
Red, the test on rc (9170cab)	run 43025704067	All 8 new rows fail on each version, each with both spellings in the form data, for example {User-Agent=A, user-agent=B}. The other 38 tests pass
Green, with the change (21eb64f)	run 43025726318	46 of 46 pass on each version
Red, the two prefix rows (a1f1fd9)	run 43056905002	On Java 8 and 21, 1 of 48 fails, the ab-c.Value against ab.value row. The _x row passes already, because String.CASE_INSENSITIVE_ORDER compares in lower case
Green, with the prefix comparison (1f9e4d5)	run 43056926053	48 of 48 pass on each version
Full pull request build	run 43072059111	Head 2904873, which merges rc at 528e0df. All 9 jobs without macOS passed, being Ubuntu Java 8, 11, 17 and 21, Ubuntu ARM Java 17, and Windows Java 8, 11, 17 and 21. Every Setting 'PullRequestSha' line, in Configure and in each of the 9 jobs, is 2904873, and Configure printed "Left out 6 macOS entries". The first attempt failed on Ubuntu Java 11 when JavaScriptBuilderTests.Init timed out starting the browser, a test this draft does not touch, and the second attempt passed
Earlier full build	run 43039392495	Head 21eb64f, before the prefix comparison. Every job passed

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Every exit path of getFormData, addFormData and otherPrefixRank was read after the change.

What remains

• 51degrees-java#97 changes CloudRequestEngineDefault.java as well, at other lines, so whichever merges second merges rc again.

Version

No version change.