[jwrosewell]

The problem

The Cloud Request Engine sends the cloud service only the Evidence whose keys the service lists in its evidencekeys answer. It is to send every Evidence value apart from the reserved names, which is what .NET, Python, PHP and Node already do. Go has the same gap and changes separately. There is no issue for this change.

A review of this draft found two more differences from .NET on the same path.

1. Evidence whose key has no prefix, such as bare, is never sent, where .NET sends it under its whole key.
2. Prefixes are compared with their letter case, so Query.User-Agent is treated as Evidence with some other prefix, and header.User-Agent takes precedence over it with a warning. .NET compares prefixes without regard to case, and Evidence keys are case-insensitive.

The decision

On 10 November the rule was decided as "everything but reserved", because what the cloud service uses changes over time and knowing about the other Evidence is important to continued optimization. The reserved names are resource, license, licence and values, in any letter case and under any prefix.

The cause

At rc d39b6af, CloudRequestEngineDefault.isSent returns false for a key that evidenceKeyFilter.include rejects, which is any key the evidencekeys answer does not list, and for a key with no separator. getFormData builds the request only from Evidence isSent allows. #89 (merged as ff6e4c0) added the first check, following the first sentence of the specification's Processing section at the time. Java's Evidence store keeps every key added (FlowDataDefault.addEvidence), so the check is the only thing that limits the request. hasKeyPrefix tests itemKey.startsWith(prefix + "."), which is case-sensitive, and the conflict check compares the prefix with equals("query").

What the specification says

specifications#138 merged into specifications rc at 07:28 UTC today. At rc e89679e, the Processing section of the Cloud Request Engine page says the following.

The Engine processes Flow Data by making an HTTP request to the server with every Evidence value in the Flow Data, apart from the reserved names and the web request view below.

The Engine does not limit the request to the accepted Evidence that the evidencekeys request returns, so it also sends Evidence the server does not list.

Evidence under any prefix whose key strips to values, resource, license or licence, in any letter case, MUST therefore be left out of the request

The web request view that a web integration adds under server.http-request, as web integration describes, MUST also be left out of the request, in any letter case, because it is a view of the whole web request and not a value to send.

When Evidence is added, its prefix MUST be removed. For example, query.user-agent becomes user-agent. [...] Where there are conflicts, the precedence order defined in Evidence MUST be used to determine which value to send to the remote server.

A key with no prefix has nothing to remove, so it is sent under its whole key, and a reserved name with no prefix still strips to itself and is left out. The Overview of the Evidence page at the same commit says "Keys MUST be case-insensitive and will usually be split into <prefix>.<field>", so a prefix is only usual, and its precedence list puts query above header, cookie and the rest.

What .NET does

On 51degrees-dotnet rc at b3e90b4, CloudRequestEngine.GetContent adds every Evidence value apart from the web request, and IsReservedField leaves out the reserved names. It does not check the evidencekeys answer, although the engine holds that answer as its Evidence key filter.

AddQueryData takes the whole key as both the prefix and the suffix when the key has no separator, and leaves it out only when the suffix is reserved. KeyHasPrefix compares the part before the first separator, or the whole key, with InvariantCultureIgnoreCase, and the conflict check compares the query prefix the same way.

What changed

Head 85b8ad7, which merges rc at 1c2c578 into 7e8fac2. Links are at 7e8fac2, whose files the merge does not change, because it brings in only the data update service change of #98.

1. isSent no longer consults the accepted Evidence and no longer drops a key with no prefix. It leaves out only the web request (server.http-request) and a key whose name is a reserved name, compared without regard to case. Everything else is sent.
2. Two helpers, keyPrefix and keySuffix, split a key at its first separator as .NET does, and give the whole key as both where it has none. isSent, addFormData and hasKeyPrefix use them, so the reserved name filter applies to a key with no prefix the same way it applies to the name after a prefix. A name after the prefix can still hold a dot, as id.usage does in query.id.usage.
3. hasKeyPrefix and the conflict check compare the prefix without regard to case, so Query.User-Agent takes precedence over header.User-Agent with no warning. A key with no separator is its own prefix, so a bare key such as query ranks with query Evidence, as in .NET, and the comment on hasKeyPrefix now says so.
4. Evidence with no prefix ranks with Evidence of any prefix other than query, header and cookie, so it has the lowest precedence, unless the whole key is one of those prefixes. The Evidence page's precedence order orders prefixes and says nothing of a key without one, so this follows .NET, whose GetContent puts such a key with the other prefixes too. The comment on getFormData records the choice.
5. The accepted Evidence is unchanged otherwise. It is still the engine's Evidence key filter, which the Pipeline holds, the cache builds its keys from, and the web integration uses for the Vary header, as the CloudEvidenceKeyFilter comment says. When the engine sends a request is also unchanged, so no data request is sent until discovery has the accepted Evidence and the properties.

For the release notes

• The Cloud Request Engine now sends the cloud service every Evidence value apart from resource, license, licence and values and the web request, whether or not the service lists the key as accepted, and including Evidence whose key has no prefix.
• Evidence prefixes are compared without regard to case when the Cloud Request Engine decides which of two values with the same name to send.

How it was checked

Five tests in CloudRequestEngineTests post through a mocked HTTP client and read the fields of the one request sent.

• process_SendsEveryEvidenceValueButTheReservedNames replaces process_SendsOnlyAcceptedEvidence, which expected the opposite. The service lists only query.User-Agent. The request must carry User-Agent, accept-language, X-Other, session and client-ip, and must leave out other.VALUES and server.licence.
• process_SendsEvidenceWithNoPrefixUnderItsWholeKey requires bare=value in the request, and Values, license, RESOURCE and licence with no prefix to be left out.
• process_ComparesPrefixesWithoutRegardToCase sends Query.User-Agent and header.User-Agent, and requires the query value to be sent with no warning logged.
• process_RanksEvidenceWithNoPrefix sends X-Thing and header.X-Thing, and requires the header value, then a bare query and header.query, and requires the bare value.
• process_RepeatedRequestHitsTheCache builds the engine with a cache and sends two requests that differ only in header.sec-ch-ua-platform, which the service does not list. One request must be posted, carrying the first request's value, and the engine must not advertise the unlisted key. The Generation of keys section of the caching page says "All Flow Elements MUST advertize the Evidence keys that they make use of. You will need to use this to build a list of the relevant Evidence keys and values that are present in the Flow Data." FlowCacheDefault builds its keys from the engine's Evidence key filter, so sending every value does not change what a cache entry is found by. The test fails if the cache key takes every Evidence value, and it pins behaviour that already holds, so it has no red run. It came from 2321010, whose two requests were identical and so would share an entry whatever the key was built from, and 7e8fac2 makes the second differ.

The existing tests that the reserved names are left out even where the service lists them, and that the web request is never sent, are unchanged and still pass.

Each run used a temporary workflow on a tmp/ branch that builds the named modules on Ubuntu and runs the named tests.

Run	Commit	Result
Red, every value	42896508786 at e488137, the first test on rc, Java 17	Failed as expected, 1 of 34, "The request did not carry every Evidence value apart from the reserved names: [resource=resourcekey, User-Agent=iPhone]"
Green, every value	42897140667 at 4e73db5, Java 17	Passed, 0 of 82
Red, no prefix and case	42981264362 at 6cfbd89, the two new tests with the code unchanged, Java 8 and 21	Failed as expected, 2 of 72 in each job. "The request did not carry the Evidence with no prefix, or carried a reserved name: [resource=resourcekey, User-Agent=iPhone]" and "The query Evidence did not take precedence: [resource=resourcekey, User-Agent=header-value]"
Green, no prefix and case	42981264213 at e6404fc, Java 8 and 21	Passed, 0 of 72 in each job
Red, ranking	42989122523 at 6cfbd89 with the ranking test added, Java 8 and 21	Failed as expected, 3 of 73 in each job, the ranking test with "Evidence with no prefix was not ranked as .NET ranks it: [resource=resourcekey, query=header-query, X-Thing=header]" beside the two above
Green, ranking	42989275272 at e12820c, Java 8 and 21	Passed, 0 of 73 in each job
Green, cache	42995618564 at 7e8fac2, the head, Java 8 and 21	Passed, 0 of 74 in each job, process_RepeatedRequestHitsTheCache among them
Full build	43005385453 at 85b8ad7, the head	Passed, all 9 jobs

The full build's 9 jobs without macOS all passed, being Ubuntu Java 8, 11, 17 and 21, Ubuntu ARM Java 17, and Windows Java 8, 11, 17 and 21. Every Setting 'PullRequestSha' line, in Configure and in each of the 9 jobs, is 85b8ad7, the head, and Configure printed "Left out 6 macOS entries". Earlier full builds are superseded, 42897571422 at 4e73db5, and 43001749269 at 7e8fac2, which was cancelled when rc was merged in.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

The branch has rc merged in twice, at c77fb05, which moves cxx to c6d4e4d9, and at 1c2c578, which carries #98.

Other drafts

git merge-tree --write-tree of this head against every other open draft's head finds no conflict, apart from #43, #63, #67, #82 and #97, which conflict with rc itself, #82's being the cxx submodule. #98 is merged into rc and is in this branch.

What remains

Go makes the same change separately. Go, Node, Python and PHP should make the same three choices as .NET and this draft, being to send Evidence with no prefix under its whole key, to compare prefixes without regard to case, and to rank Evidence with no prefix with the other prefixes unless the whole key is query, header or cookie. Java did none of these before this change.