[jwrosewell]

Problem

Node's usage sharing element accepted, as its own evidence, every key it might
share. Accepted evidence is what the Pipeline advertises, and the ETag and the
Vary header of the client-side endpoints and the engines' cache keys are built
from it, so adding usage sharing to a pipeline changed the ETag of every
response and widened every cache key, for evidence that cannot change a single
result.

The Flow Data keeps a key only when some element accepts it, so usage sharing
accepting everything was also the only reason a header no other element reads
reached the Flow Data at all. The two could not be separated without another
route to the request, and that route is server.http-request.

The Cookie header was blocked only while shareAll was off, so turning
shareAll on shared every cookie the browser sent, including the ones the
cookie rule exists to leave out.

.NET, Java, Python, PHP, Go and Rust have all taken this change, so Node is the
last language still accepting what it shares.

What the specification says

usage-sharing-element.md, Accepted Evidence,
at the fixed commit d72eb074.

The Usage Sharing Element MUST NOT accept any Evidence. Its accepted Evidence
is empty, so it never adds a key to the Evidence a Pipeline accepts. The
features that are built from a Pipeline's accepted Evidence, being caching
and the ETag and Vary headers of the client-side endpoints, therefore never
depend on it ...

It shares instead what it finds, from two places:

• every Evidence value in the Flow Data, and
• every header other than Cookie, every cookie, query string parameter and
  session value of the web request, with its client IP address, its protocol
  and the method and path of its request line, when the web integration
  supplies the request as server.http-request ... This is how a header that
  no other Flow Element accepts is still shared.

A key found in both places is shared once, with the value from the Evidence.

Shared Evidence
at the same commit adds the Cookie header to the keys that are never shared.

• header.cookie, which is the Cookie header of the request and carries
  every cookie (RFC 6265 section 5.4). This holds for the Cookie header of
  the server.http-request view as well as for Evidence. The cookies
  starting 51D_ are shared as their own cookie. Evidence under the rules
  below, and no other cookie is shared through the header.

... The server.http-request value itself is never shared, only the values
read from it.

The same section says the session cookie is read from the same two places when
track session is on, that it is not shared unless all evidence is shared, and
that the hash the tracker compares "is of the values that would be shared, as
the element accepts no Evidence of its own for the caching logic to read".

advertize-accepted-evidence.md
at the same commit is the rule the element default follows, that an element
accepts only the Evidence it reads.
sequence-element.md, Accepted Evidence
gives the sequence element's keys as query.session-id and query.sequence,
which is what it now accepts.

Change

The web request, as evidence

evidence.js

• addFromRequest builds the whole web request as a frozen object and adds it
  under server.http-request, with every value it read under the key that
  value has as evidence, plus the whole request line as
  server.request-method, server.request-path and server.request-query.
  It also now reads the session's values as session. evidence, which the
  page lists among the parts of the request.
• The three request line values are on the view of every request, taken from
  request.method and request.url byte for byte with no decoding, with the
  method empty where the request carries none and the query empty where the
  target has no question mark, which is what
  web-integration.md, Populating Evidence
  asks for, so that an absent key always says the request line was not
  supplied rather than that the request had no query. requestLineParts
  replaces requestLinePath, which wrote an empty path as /. .NET's
  WebRequestEvidenceService.SplitRequestTarget splits the same way, at the
  first question mark with the mark dropped from the query. Both
  server.request-path and server.request-query are in the usage sharing
  element's NEVER_SHARED_KEYS, so neither is ever shared.
• add keeps server.http-request apart from the evidence store rather than
  filtering it, because it is the request rather than a value to process, and
  get gives it back. getAll leaves it out, so nothing that walks the
  evidence meets it. Both comparisons are made on the folded key, so the rule
  rc brought in, that evidence keys ignore letter case, holds for this key
  too.
• add also keeps a key that an element's new keptEvidenceKeyFilter matches,
  which is how the flow data still holds what usage sharing shares without
  usage sharing accepting it.
• The old requestEvidence field is gone, replaced by the same values under
  server.http-request.

flowElement.js

• An element with no filter of its own now accepts nothing, rather than
  everything, which is the element default the specification asks for.
• keptEvidenceKeyFilter is the evidence an element reads without it changing
  a result. The flow data keeps it, and nothing else reads the field, so it
  never reaches the accepted evidence, the ETag, the Vary header or a cache
  key.

This changes what an existing Flow Element of your own is given. On rc
an element that set no evidenceKeyFilter got the base EvidenceKeyFilter,
whose filterEvidenceKey returns true, so it accepted every key. After this
change it accepts none, and a Flow Element written outside this repository
that relied on the old default stops being given any evidence, with no error
and no warning. Anyone upgrading has to give such an element the filter naming
the keys it reads. This belongs in the release note for the version that
carries it.

helpers.js

• The client-side ETag is built from server.http-request rather than the
  removed requestEvidence, and still hashes only what an element accepts, so
  evidence kept for usage sharing takes no part.

Usage sharing

shareUsage.js

• evidenceKeyFilter is an empty BasicListEvidenceKeyFilter, so the element
  accepts nothing. The filter it had becomes keptEvidenceKeyFilter, so the
  flow data still keeps what would be shared and, when sessions are tracked,
  the session cookie.
• dataEvidenceKeyFilter, which decides what is sent, is still built with
  includeSession: false, so the session cookie is read for the tracker and
  not shared, as the page asks.
• getFoundEvidence is the two places the page names. It reads every value of
  server.http-request and then every evidence value, so a key in both takes
  the evidence value. The two are merged on the folded key, so a key whose
  letters differ between the view and the Evidence is one value under the
  Evidence spelling, which is the page's "A key found in both places is shared
  once".
• getDataFromEvidence and the tracker's key both read getFoundEvidence, so
  a header no other element accepts is shared, and the tracker compares the
  values that would be shared.

shareUsageEvidenceKeyFilter.js

• header.cookie moves from the headers blocked while shareAll is off into
  NEVER_SHARED_KEYS, so it is never shared whatever the settings.
• server.http-request itself is never shared, only the values read from it.

Other elements

The cloud request engine keeps every key until the cloud service's keys arrive,
and then narrows to them, so it no longer accepts everything for the life of a
process that never reaches the service. The set headers element accepts
nothing, because it reads the other elements' results. The sequence element
accepts query.session-id and query.sequence, which is the list its page
gives. The JavaScript builder accepts header.host and header.protocol
rather than every header.

The two on-premise engines read the keys they accept from the data file when
the engine is initialised, which waits for the download when updateOnStart
is set and there is no file yet. With an element accepting nothing by default,
evidence added before the file arrived was dropped, and the first full build on
this draft failed on
update on start-up with no data file > downloads the data file and then processes with it with
The evidence required to determine this property was not supplied. Both
deviceDetectionOnPremise.js
and
ipIntelligenceOnPremise.js
now keep every key until initEngine replaces the filter with the keys the
file lists, which is what the cloud request engine does for the keys the
service lists.

One test of rc's own moved with the behaviour

evidenceKeyCase.test.js, which
51degrees-node#71
added to rc, has a test that the JavaScript Builder finds a key whatever its
letters, and it used HEADER.User-Agent as its example because the old filter
accepted every header. The builder no longer accepts the user agent, which it
does not read, so the test now uses HEADER.Host. The point of the test, being
letter case, is unchanged.

What else this branch carries

This branch holds
51degrees-node#46,
"Accept only the evidence each element reads, and none by default", by a merge
commit, and has since it was opened. That change is the element default above
and the per element filters, and it is the same rule
51degrees-python#63
and 51degrees-php#41
merged. Usage sharing cannot accept nothing until the default is nothing, so
the two cannot be separated without undoing the merge and rebuilding both. If
this draft merges, #46 has nothing left to add and the coordinator should close
it as superseded.

The conflicts with rc

This draft was CONFLICTING when it was picked up, and rc has moved six times
since. Every resolution is below.

1. cloudRequestEngine.js. rc and this branch each added an import,
   LicenseKeys on rc and EvidenceKeyFilter here. Both are kept.
2. cloudRequestEngine.test.js. Each side appended a test at the end of the
   file, this branch's evidence set before the keys arrive is kept and rc's
   evidence never supplies resource, license or values from
   #61. Both are kept,
   in that order.
3. evidence.js, types/evidence.d.ts and javascriptbuilder.js, against
   #71. add and get
   now fold the key to lower case once and compare the folded key against
   server.http-request before rc's lookup by folded key, so both rules
   hold. JSEvidenceKeyFilter keeps rc's String(key) guard and this
   branch's narrower list. The two doc comments are joined.
4. shareUsage.js merged with no conflict against
   #36, which rc took
   at 13:30. Both features are in the merged file, the back-off and the empty
   accepted evidence, and the generated types/shareUsage.d.ts carries both.
5. The last merge, of rc at
   5228a9f9,
   conflicted in evidence.js alone, in get, against
   #72. get now
   answers server.http-request from the field add set aside, and otherwise
   uses rc's lookup, which reads the accepted store alone so that a key set
   aside is not returned. Both rules hold.
6. rc at 8093efac,
   the move of cxx to the release commit 7c4ac436 from
   #55, merged with no
   conflict. It changes the submodule and a generated wrapper, neither of
   which this draft touches.

The same change in the other languages

Language	Pull request
.NET	#100
Java	#68
Python	#61
PHP	#49
Go	#62
Rust	#334

All six are merged. All six use the key server.http-request, and .NET and PHP
also put the request's method on the view, as this does. .NET's
EvidenceKeyFilterShareUsage keeps the session cookie out of what is shared
and leaves it to the tracker's own filter, which is what
dataEvidenceKeyFilter and trackerEvidenceKeyFilter do here.

Tests

shareUsageAcceptsNoEvidence.test.js
sends real requests to a local server and reads what was shared. It checks that
a pipeline with usage sharing does not accept arbitrary headers, that the web
request is in the evidence with every part the page names, that the method is
shared while the path and the Cookie header are not even when all evidence is
shared, that a header no other element accepts is still shared from the web
request, that evidence wins over the request for the same key, that a key in
both places whose letters differ is shared once, that the request line is on
the view with an empty query where the target has none, that evidence added by
hand is shared as before, and that evidence only usage sharing reads takes no
part in the ETag while a revalidation still gets a 304.

acceptedEvidence.test.js,
elementDefaultEvidence.test.js
and
sequenceElementAccepts.test.js
cover the element default and the per element filters.
httpRequestEvidence.test.js
checks the web request is neither sent to the cloud service nor part of its
cache key. updateOnStartMissingFile.test.js in each on-premise package gains
keeps every key until the data file arrives, which names the rule the
regression above broke, so it cannot break again without a test saying so.

Red, the first round. Run 43068117316,
on tmp/node-45-red2, which is rc at 5228a9f with this branch's test files
and nothing else from it. All six suites failed, 18 of the 29 tests, being six
of the nine in shareUsageAcceptsNoEvidence.test.js, two of the nine in
shareUsageEvidenceKeyFilter.test.js, one of the two in
httpRequestEvidence.test.js, all three in acceptedEvidence.test.js, all
four in elementDefaultEvidence.test.js and both in
sequenceElementAccepts.test.js. The 11 that passed assert what rc already
does, chiefly the seven shareUsageEvidenceKeyFilter.test.js already had.

Red, the request line and the folded key.
Run 43077365630,
on tmp/node-45-red4, which is this branch with the three new assertions and
without the code that answers them. Three of the 11 failed, being the web request has every part the specification names, the request line is on the view with an empty query where there is none and a key in both places whose letters differ is shared once.

Green. Run 43077435257,
on tmp/node-45-green10, which is this head with one temporary workflow added.
The six suites above and evidenceKeyCase.test.js, evidence.test.js and
cloudRequestEngine.test.js pass, 71 of 71 in 9 suites.

Full build. Run 43077468895
on the head, 834dec1, every job a success. All seven
Setting 'PullRequestSha' lines read
834dec1, Configure printed "Left out 2 macOS
entries", and each of the six build jobs ran 1,221 tests passed with 2 skipped,
then 132 passed with 4 skipped.

macOS was left out of this draft's full build under the 9 November decision.
The rc head is built with macOS before it merges into main.

What remains

• 51degrees-node#46 is
  inside this branch and should be closed as superseded if this merges. That is
  the coordinator's call, not mine.
• 51degrees-node#48 is
  stacked on Accept only the evidence each element reads, and none by default #46's branch and needs rebasing onto rc after this merges.
• The JavaScript builder still accepts every query. key, where
  javascript-builder.md, Accepted Evidence
  names four, being header.host, header.protocol,
  query.fod-js-object-name and query.fod-js-enable-cookies, and .NET's
  JavaScriptBuilderElement has exactly those four in its whitelist. It is
  left as it is here, and the reason is worth writing down for whoever takes
  it. Step 7 of the same page says the script's parameters are "all Evidence
  values starting with query.", and .NET reads them straight from the flow
  data's evidence, which in .NET holds every value added whether or not an
  element accepts it. Node's Evidence.get answers from the accepted store
  alone, so narrowing the builder's filter without also changing where the
  parameters are read from loses the query string. Both changes together were
  tried on this branch and reverted in
  527b660, after
  run
  43069531045
  failed 17 of 664 tests across the form body, session id, sequence, letter
  case and cloud suites. Making Node match needs the question of what get
  answers for a key set aside settled first, which is a change to
  #72's model rather
  than to this draft.
• CloudRequestEngine.keysAndPropertiesFetched reads
  this.evidenceKeyFilter.length, which no evidence key filter in Node has, so
  it is always false. That is true on rc as well and is untouched here.