[jwrosewell]

For 51Degrees/specifications#53.

The problem

The web integration specification's cache header detail asks the client-side JSON and JavaScript endpoints for Cache-Control, Vary and an ETag, and a 304 when If-None-Match matches. Specifications#53 adds that the same evidence gives the same ETag in every process, and that a response to a request whose processing recorded errors carries Cache-Control: private, no-cache and no ETag, while a matching If-None-Match still gets its 304.

Node has no web integration package, so its client-side endpoints are written in each web example, and none of them did any of this. On rc (21fa119) they set only Content-Type and the body.

• Device Detection cloud gettingstarted-web, /json and /51Degrees.core.js
• Device Detection on-premise gettingstarted-web, both
• IP Intelligence cloud mixed/gettingstarted-web, both
• location webIntegration.js, /json
• the pipeline's clientSideEvidenceFlowElement.js, /json

The /json of IP Intelligence's on-premise mixed/gettingstarted-api stands in for the cloud service rather than being a client-side endpoint, so it is unchanged.

What the specifications say

Read in full, with RFC 9110 and RFC 9111 checked as current at rfc-editor.org (neither is obsoleted or updated).

• The cache header detail, with specifications#53's two paragraphs, as above.
• RFC 9110 section 13.1.2: "A recipient MUST use the weak comparison function when comparing entity tags for If-None-Match", and the origin server "MUST respond with either a) the 304 (Not Modified) status code if the request method is GET or HEAD or b) the 412 (Precondition Failed) status code for all other request methods." The JSON endpoint is posted to, so a matching If-None-Match there gets a 412, not a 304.
• RFC 9110 section 15.4.5: a 304 "MUST generate any of the following header fields that would have been sent in a 200 (OK) response to the same request", including ETag, Vary and Cache-Control, and "cannot contain content".
• RFC 9110 section 8.8.3: an entity tag is a quoted string, and one whose generation "does not satisfy all of the characteristics of a strong validator" MUST be marked weak with W/. A strong validator "changes value whenever a change occurs to the representation data" (section 8.8.1). A hash of the evidence stays the same when the answer changes, for example after a data file update, so it is weak.
• RFC 9111 section 5.2.2.4 and section 5.2.2.7, for no-cache and private, as specifications#53 cites.

The open draft 51Degrees/specifications#81 writes the RFC 9110 rules above into the cache header detail. This change follows RFC 9110 whether or not that draft is merged, as was decided for the 412 on 7 November.

What changed

Helpers.writeClientSideResponse(request, response, flowData, contentType, body, options) in fiftyone.pipeline.core writes either endpoint's response.

• ETag. W/"<32 hex digits>", the start of a SHA-256 of the evidence the request carries, keys sorted. It is built from what addFromRequest read from the request, which addFromRequest keeps in evidence.requestEvidence, hashing only the keys an element accepts, rather than from the flow data's evidence, because processing adds evidence of its own. Evidence added by hand has no such record, so its ETag comes from the flow data's evidence. The sequence element makes a new random session id for every request without one, and with that in the hash no two responses had the same ETag, which a first attempt at the fix showed (run 42397140186). The request's precondition headers (RFC 9110 section 13.1) and cache directives (Cache-Control and Pragma, RFC 9111 sections 5.2.1 and 5.4) are left out, because they say how to answer rather than what to answer, and the JavaScript builder takes every header. key as evidence, so a revalidation's own If-None-Match would otherwise change the ETag it is checked against. Nothing in the hash is chosen per process, so the same evidence gives the same ETag in every process.
• Cache-Control. private, max-age=1800, with the age set by options.maxAgeSeconds as the specification's "Max-age SHOULD be configurable" asks. After errors, private, no-cache.
• Vary. The header. evidence keys of every element whose filter lists its keys, as the Java pipeline does, such as user-agent.
• A matching If-None-Match. A GET or HEAD gets a 304 with no body and the Cache-Control, Vary and ETag a 200 would carry, which after errors means private, no-cache and no ETag. Any other method gets a 412 with no caching headers.
• Otherwise a 200 with Content-Type, Content-Length in bytes and the body, and no body for a HEAD.

The seven endpoints above call it in place of setting the status, the content type and the body.

How it was verified

Red, the tests alone on rc (run 42397458626, at 28f0425), and green, with the fix (run 42397637050, at a1048b8). Both ran on ubuntu-latest with Node 24 from a temporary branch holding this branch plus one workflow, which has been deleted. The Device Detection cloud example ran with the organisation's Bespoke resource key.

tests/clientSideCaching.test.js in fiftyone.pipeline.core serves both endpoints from a pipeline with one element that reads the User-Agent and can be made to fail.

Test	Red	Green
A response carries private, max-age=1800, Vary: user-agent, a weak ETag and the right Content-Length, on both endpoints	fails, 500 Helpers.writeClientSideResponse is not a function	passes
The same evidence gives the same ETag in two separate Node processes, and other evidence another one, through tests/clientSideETag.js	fails, no ETag	passes
If-None-Match with the ETag, its strong form, a list holding it, and * each get a 304 with no body and the same ETag, Cache-Control and Vary, and another tag gets a 200	fails	passes
A POST whose If-None-Match matches gets a 412	fails	passes
While the element fails, a response has private, no-cache and no ETag, a matching If-None-Match still gets a 304 with private, no-cache and no ETag, and once it works again the ETag is the one it was	fails	passes

Red    Tests: 5 failed, 5 total
Green  Tests: 5 passed, 5 total

The Device Detection cloud gettingstarted-web test gains cloud getting started web caching headers, which asks the real example for the script, then for it again with its ETag in If-None-Match, and posts to /json. On rc it failed with Expected: "private, max-age=1800" and Received: undefined. With the fix it passed, the second request getting a 304 with the same ETag.

Red    Tests: 1 failed, 1 passed, 2 total
Green  Tests: 2 passed, 2 total

Both runs also ran the IP Intelligence cloud mixed web example test and the location tests (7 passed), every other fiftyone.pipeline.core test (234 passed on red, all 239 on green), and eslint on every changed folder, which passed. The committed types/helpers.d.ts is what npm run tsc gave in the green run.

A later fix: the ETag no longer reads the request after processing

The full build of #45, which holds this branch, failed on macOS with Node 24 (run 42418797979). The Selenium contract tests stopped the Device Detection cloud web example with TypeError: Cannot read properties of undefined (reading 'toString') at request.connection.localAddress.toString(), called from clientSideETag. The ETag was then built by reading the request again after processing, and a browser can stop waiting while the pipeline asks the cloud, which leaves the request's socket with no addresses.

Commit 6554046 adds the ETag does not read the request again after processing, which processes a request and then takes its connection away before writing the response. It failed on run 42420811997, with Tests: 1 failed, 5 passed, 6 total. Commit 1d4caad keeps every value addFromRequest reads in requestEvidence and builds the ETag from that. All six core caching tests, both example tests and all 240 core tests passed on run 42420786946.

What remains

• Accept no evidence in usage sharing, and share the web request it finds #45, the usage sharing element accepting no evidence under specifications#99, is stacked on this branch. It makes this ETag hash only the evidence an element accepts, so values only usage sharing collects, such as a 51D_ cookie or server.host-ip, no longer change it.
• The full pull request build ran on this branch at 1d4caad, merged with rc, and passed on all eight entries (run 42539224557). It had also passed on Accept no evidence in usage sharing, and share the web request it finds #45, which holds this branch, Selenium contract tests included (run 42425263121). The Device Detection on-premise web example, which needs its native module, has no test that asks for these headers.
• Specifications#53 and Time IP Intelligence reads by name against reads by position #81 are drafts. If either changes, this follows.
• The ETag is weak here, where the Rust crates send a strong quoted string (compute_etag) and the Java pipeline an unquoted number (ClientsidePropertyServiceCore.java). The specification does not say which, and RFC 9110 section 8.8.3 asks for weak.
• A data file update gives a new answer for the same evidence under the same ETag, so a browser keeps its copy until the evidence changes. That is the specification's design, which its "This process could lead to responses that are not strictly correct" paragraph describes.