[jwrosewell]

The Java Cloud Request Engine now follows the Revised design (eager discovery at build time, transient-failure fallback) section of pipeline-specification/pipeline-elements/cloud-request-engine.md on the specifications' rc, and offers the kept copy its Resilience section asks for.

This branch carries #84 (a cloud engine whose product the resource key lacks), because the cloud aspect engines here build on the findProduct that #84 adds. #84's head, 9fced9b, was merged in, and #84 has since merged into rc as 0db9c2e. #85 (license key masking) is merged into rc as fc55341, and rc at fc55341 is merged in here with its one conflict resolved, as the "Merges" section below says. The commits of this change are c72baac, a4b6fd5, 4defa2a, 4f62134, bef79b0, be26bcf, 9f35ac9, ea5fc04, 8f709b4, 40452b7, 7bc481c, 726204b, 19cc0fa, 1e1dae8, 127b173 and 7d4cebd, with the merge commits c309b58, 66326a3, 5a6fc84 and 8a19907. The head is 8a19907.

The problem

On rc at 7299688, CloudRequestEngineDefault fetched its accessible properties and its accepted evidence keys only when something first asked for them, and threw where that was.

1. Nothing was fetched at build. An invalid resource key surfaced wherever the products or the evidence filter were first read, not when the engine was built.
2. An outage at start-up failed the Pipeline build. The Device Detection, IP Intelligence and Location cloud engines read the products when they join the Pipeline, so a cloud outage then threw out of the Pipeline build, and getEvidenceKeyFilter threw from whatever read it, such as the web integration's start.
3. A failure was kept. After a failure the cloud aspect engines logged two errors and kept no metadata, and never read it again.
4. No timeout on discovery. The two discovery requests did not use the configured timeout, so a service that did not answer could hold the build for as long as the HTTP client allowed.

What the specification says

From Revised design:

The Engine MUST attempt both discovery requests when it is built (in its builder or constructor), and MAY make them in parallel, so that in the normal case a built Engine is fully initialised and immediately ready to process Flow Data.

Definitive configuration errors - an HTTP 4xx response to a discovery request (for example an invalid Resource Key) - MUST fail the build with a critical error.

Transient failures - the service unreachable (DNS or connection failure), a timeout, or an HTTP 5xx response - MUST NOT fail the build. The Engine MUST log a warning, complete the build, and retry discovery on first use. Failures of those first-use retries are process-time failures, so SuppressProcessExceptions applies to them as to any other processing error.

From Consequences for consumers of the Engine:

After a transient discovery failure, the built Engine advertises empty accepted Evidence and empty Property metadata until a first-use retry succeeds. Consumers of this information therefore MUST tolerate it being temporarily absent and MUST NOT permanently cache an empty result obtained before discovery has completed.

From Resilience:

implementations SHOULD allow an Engine to be built from a previously obtained, persisted copy of the discovery results ... When supplied, the builder uses it and makes no cloud request ... The persisted copy is read back from the builder after a successful build.

The Java builder refuses a build with no resource key, so Discovery under a License Key does not arise. The page notes that Recovery Mode is supported by .NET only, and the Java engine has no recovery mode, so there is no failure count for discovery to add to.

What changed

Path	Java before	Java now
Successful fetch at build	Nothing fetched at build	CloudRequestEngineBuilder calls discover, which makes both requests at once, as the page allows, so a service that does not answer delays the build by one timeout. The evidence keys request runs on another thread while the accessible properties request runs on the building thread (bef79b0).
Definitive failure at build	Surfaced on first read	A 4xx answer to either request fails the build with its CloudRequestException, whose getHttpStatusCode() is the cloud's status. Both requests are made first, so a 4xx on one fails the build even when the other failed in a way that can pass. The exception is the one getPublicProperties threw before, as in .NET.
Failure that can pass, at build	Not applicable	One warning, and the build completes, keeping any list that arrived. A 5xx answer, a refused connection, a timeout and a body that cannot be read all count, as in .NET and Go, because the page makes only a 4xx definitive. A body that cannot be read passes only when the status is not 4xx. A 4xx whose body is not JSON, such as an error page from a proxy, fails the build like any other 4xx, because the CloudRequestException raised for it keeps the status (7d4cebd).
Retry on first use	Not applicable	processEngine fetches only what is missing, evidence keys first, stopping at the first failure, and sends no data request until both are known. The failure is thrown or recorded as SuppressProcessExceptions says. A request that could not be made is reported as its IOException, as a failed data request is. process-started is now set before discovery, so the cloud aspect engines report a missing response rather than a missing Cloud Request Engine.
Timeout	Not applied to discovery, and 100 seconds by default	The configured timeout applies to both discovery requests, and the default for every request is now 2 seconds, as the page's Configuration options table gives TimeoutSeconds and as .NET's CLOUD_REQUEST_TIMEOUT_DEFAULT_SECONDS is. .NET applies it to the one HttpClient that makes the discovery and the data requests (CloudRequestEngine.cs lines 265 to 272), and Java applies it to each connection.
Reading the results	getPublicProperties and getEvidenceKeyFilter fetched and threw	Neither makes a request. getPublicProperties is null until the products are known. getEvidenceKeyFilter is one CloudEvidenceKeyFilter, which includes nothing until the keys are known and reads them on each call. It derives from EvidenceKeyFilterWhitelist, so the Pipeline's EvidenceKeyFilterAggregator keeps it rather than copying its keys once, and the Pipeline's filter sees the keys as soon as a retry fetches them.
Cloud aspect engines	Two errors at build after a failure, and no metadata ever	The Device Detection, IP Intelligence and Location cloud engines, and CloudAspectEngineBase.getProperties, return empty metadata while the products are unknown, without keeping it or logging, and read it once they are known. A missing product, #84's case, still builds with empty metadata and logs once, because findProduct tells an unknown product list (null) from one without the product.
Kept copy (Resilience)	Not offered	CloudDiscoveryState holds the evidence keys and the accessibleproperties answer, with toJson and fromJson. CloudRequestEngineBuilder.getDiscoveryState() returns it after a build that knows both lists, and setDiscoveryState makes a later build use it with no discovery request.

The web integration's ClientsidePropertyServiceCore now reads any filter that is or derives from EvidenceKeyFilterWhitelist for the Vary header. It read only the exact class, so without this the engine's new filter would have dropped the cloud's headers from Vary. #67 (caching the client-side endpoints' answers) makes the same change to those lines. The two still conflict on the Vary line of the 200 answer, which this branch changes to read getHeadersAffectingJavaScript() and #67 moves into its new setCachingHeaders. Whichever of the two merges second keeps #67's setCachingHeaders and has it read the list through getHeadersAffectingJavaScript() in place of the headersAffectingJavaScript field, for both the empty check and the value, so that the 200 and the 304 both carry headers that are worked out again until every element has loaded its properties.

Compared with .NET and Go

• .NET (rc) calls WarmUp from CloudRequestEngineBuilder.Build, makes both requests in parallel, rethrows a 4xx and logs anything else as a warning. Java rethrows the same exception type and logs the same way.
• 51Degrees/51degrees-go#78 (merged into Go's rc as 38acaf4) does the same, with Builder.DiscoveryState and SetDiscoveryState for the kept copy, which getDiscoveryState and setDiscoveryState match.
• Where they differ. getPublicProperties makes no request in Java, where .NET's getter fetches on first access. In Java the cloud aspect engines and the missing property service read the products many times on each request, and a getter that fetched would make a request each time during an outage. Processing fetches them instead.

Consumers that keep what they read when the Pipeline is built

After a failure at build that can pass, the page says consumers "MUST NOT permanently cache an empty result obtained before discovery has completed". Java now meets this for every consumer that keeps what it reads from the Pipeline, being the Pipeline's map of available properties, the web integration's Vary headers and the SetHeaders element, which closes the consumer rule fully for Java. .NET meets it for the first two, and its SetHeaders element is being changed to match.

• The signal. FlowElement gains hasLoadedProperties(), true by default (FlowElement.java line 115), as .NET's aspect engines have HasLoadedProperties (CloudAspectEngineBase.cs line 247). The Cloud Request Engine returns true once both lists are known, the cloud aspect engines once their metadata is loaded, and a parallel group once all its elements have.
• The Pipeline's map of available properties. In .NET, Pipeline.ElementAvailableProperties is kept only when no element's Properties threw PropertiesNotYetLoadedException (lines 647 to 657), and is built again on the next call otherwise. Java's PipelineDefault.getElementAvailableProperties now keeps the map only once every element has loaded its properties, and builds it again on each call until then. getMetaDataForProperty already looked properties up on demand and kept only those it found, as .NET does.
• The Vary headers. .NET's ClientsidePropertyService.HeadersAffectingJavaScript keeps the headers only when no element's EvidenceKeyFilter threw. Java's ClientsidePropertyServiceCore now works them out again on each response until every element has loaded its properties.
• The SetHeaders element. SetHeadersElement.processInternal now keeps the configuration it builds from the Pipeline's available properties once every element has loaded its properties, and builds it again on each request until then. Before, it built the configuration on every request, because getOrDefault never stored it, so it was never stale but was rebuilt each time. .NET's SetHeadersElement.ProcessInternal keeps the configuration from the first request for each Pipeline without that check, and is being changed to match, so the two stay aligned.

Not changed

• The data request, its evidence, its headers and the handling of errors and warnings are as before.

For the release notes

• The Cloud Request Engine now fetches its accessible properties and accepted evidence keys when it is built. An invalid resource key, or any other 4xx answer, now fails the build with a CloudRequestException, rather than the first use.
• A cloud outage, timeout or 5xx answer when the engine is built is logged as one warning, and the build and the Pipeline build complete. The first request fetches what is missing before it sends the data request.
• The two discovery requests are made at once, and the configured timeout now applies to them as well.
• The default timeout for each request to the cloud service is now 2 seconds, as the specification gives. It was 100 seconds. Call setTimeOutSeconds to keep a longer one.
• After a cloud outage when the Pipeline is built, the Pipeline's available properties, the web integration's Vary header and the SetHeaders element's response headers gain the cloud's properties and headers once a retry fetches them.
• The SetHeaders element now builds its configuration once for each Pipeline, rather than on every request.
• FlowElement.hasLoadedProperties() says whether an element knows all its property metadata and accepted evidence. It is true by default.
• getPublicProperties and getEvidenceKeyFilter no longer make requests. getPublicProperties returns null until the properties are known.
• An answer from the cloud service whose body is not JSON now raises a CloudRequestException that carries the HTTP status.
• CloudRequestEngineBuilder.getDiscoveryState and setDiscoveryState, with CloudDiscoveryState.toJson and fromJson, let a host keep what one build discovered and build later with no discovery request.

How to verify

No test calls a live service, apart from the existing CloudRequestEngineTests.HttpDataSetInException.

mvn -pl pipeline/pipeline.cloudrequestengine,pipeline/pipeline.engines.fiftyone,pipeline/pipeline.web.packages/pipeline.web,device-detection/device-detection.cloud,ip-intelligence/ip-intelligence.cloud,location/geo-location.cloud -am test -Dtest='SetHeadersElementTests,CloudRequestEngineDiscoveryTests,CloudRequestEngineTests,CloudRequestEngineBuilderTests,CloudAspectEngineBaseTests,ClientsidePropertyServiceTests,ProductMissingTests,GeoLocationCloudEngineTests,MissingPropertyHandlingTests,IPIntelligenceCloudEngineTests' -Dsurefire.failIfNoSpecifiedTests=false

Class	Test methods on rc (fc55341)	At #84's head (9fced9b)	This branch (8a19907)
CloudRequestEngineDiscoveryTests	no class	no class	14
CloudRequestEngineTests	19	18	19
CloudAspectEngineBaseTests	8	11	12
ClientsidePropertyServiceTests (pipeline.web)	7	7	8
SetHeadersElementTests (pipeline.engines.fiftyone)	6	6	8
ProductMissingTests (device-detection.cloud)	no class	4	5
ProductMissingTests (ip-intelligence.cloud)	no class	4	5
GeoLocationCloudEngineTests (geo-location.cloud)	no class	4	5

CloudRequestEngineTests has one more method on rc than at #84's head, because #85 added Process_LicenseKeyInCloudError_Masked.

CloudRequestEngineDiscoveryTests.Build_4xx_FailsTheBuild runs 12 cases, being 400, 401, 403, 404, 408 and 429 from each request, Build_4xxWithBodyNotJson_FailsTheBuild runs two, and Build_TransientFailure_CompletesWithOneWarning runs seven. The class uses a stand-in HTTP client that counts the requests to each endpoint.

• c72baac holds the tests alone, on Build a cloud engine whose product the resource key lacks, and log that once #84's code. These are written to fail there:

  • All nine methods in CloudRequestEngineDiscoveryTests at that commit.
  • In CloudRequestEngineTests, the three cases of validateErrorHandling, getPublicProperties_Server_Unavailable_At_Build and getEvidenceKeyFilter_Server_Unavailable_At_Build. These used to expect a throw on first read and now expect a build that completes with one warning.
  • CloudAspectEngineBaseTests.loadProperties_notLoaded, which used to expect a throw.
  • DiscoveryNotComplete_LoadsTheMetadataLater in each of the three cloud aspect engine classes.

• a4b6fd5 holds the fix.

• 4defa2a adds the kept copy with its two tests, DiscoveryState_KeptAndUsedWithNoRequest and DiscoveryState_NoneWithoutBothLists. They are in the same commit because they use the new class and methods.

• 4f62134 holds three more tests alone, on 4defa2a's code, each written to fail there:

  • Build_MakesBothRequestsAtOnce. Each discovery request waits for the other to start, so made one after the other the first waits 5 seconds and fails.
  • Build_DefaultTimeoutIsTwoSeconds, which finds 100 seconds there.
  • CloudAspectEngineBaseTests.elementAvailableProperties_ReadAgainOnceKnown, where the Pipeline kept the empty map it built.

  It also gives each request in the shared test client a connection of its own, as the requests can now run at the same time.

• bef79b0, holds the three fixes and ClientsidePropertyServiceTests.JavaScript_VaryHeader_ReadAgainUntilLoaded. That test is in the same commit because it uses hasLoadedProperties.

• be26bcf holds two SetHeaders tests alone, each written to fail on bef79b0's code, where the configuration was built on every request: setHeadersElement_ConfigKeptOnceLoaded and setHeadersElement_ConfigBuiltAgainUntilLoaded. They count the reads of the Pipeline's available properties.

• 9f35ac9 holds the SetHeaders fix.

Later commits

• ea5fc04 gives the SetHeaders test mocks their type parameters, because the build treats raw types as errors.
• c309b58 merges Build a cloud engine whose product the resource key lacks, and log that once #84 at 33a13fe, which marks the mocked product map unchecked.
• 8f709b4 stubs the mocked product map with two single-value thenReturn calls in the three cloud engine tests, because thenReturn(null, products) made an unchecked generic array, which the build treats as an error.
• 40452b7 and 7bc481c made a 408 or 429 at discovery transient, following specifications#126. The specification on rc and .NET (CloudRequestEngine.cs lines 496 to 501 at 5b4d06e) both treat every 4xx at discovery as fatal, and specifications#126 was closed without being merged, so 726204b and 19cc0fa revert them.
• 1e1dae8 adds 408 beside 429 to the statuses that Build_4xx_FailsTheBuild expects to fail the build, as the specification on rc makes every 4xx at discovery definitive.
• 127b173 holds Build_4xxWithBodyNotJson_FailsTheBuild alone, on 1e1dae8's code. It answers one discovery request with a 403 and an HTML body, and expects the build to fail with that status.
• 7d4cebd holds the fix. validateResponse now raises a CloudRequestException that keeps the HTTP status when the body cannot be read as JSON, as .NET does, so a 4xx at discovery fails the build whatever its body holds. The message names only the status, and the cause is the JSONException.

Merges

• 66326a3 merges rc at fc55341, which now holds Show a license key only by its last 4 characters in logs and error messages #85 (license key masking). The one conflict is in CloudRequestEngineDefault.java, where Show a license key only by its last 4 characters in logs and error messages #85 adds getLicenseKeys() in the place where this branch adds setTimeouts(HttpURLConnection) and renames getCloudProperties to fetchPublicProperties. Both helpers are kept, and the accessible properties request keeps this branch's name, fetchPublicProperties, which every call site uses. validateResponse masks the license key in the cloud's messages as Show a license key only by its last 4 characters in logs and error messages #85 does, and the message this branch adds for a body that is not JSON names only the status.
• 5a6fc84 merges Build a cloud engine whose product the resource key lacks, and log that once #84's head, 9fced9b, which logs a missing product as a warning, treats a missing response part as no values and removes isProductMissing. The conflicts are in the Device Detection and IP Intelligence ProductMissingTests and in GeoLocationCloudEngineTests. Each takes the file as it is at 9fced9b and keeps this branch's DiscoveryNotComplete_LoadsTheMetadataLater test. After the merge nothing calls isProductMissing or getCloudProperties.
• 8a19907 merges rc at 24c6f25, after Build a cloud engine whose product the resource key lacks, and log that once #84 merged as a merge commit and Write weighted values with the lower-case keys rawweighting and value #93 as a squash. GitHub reported a conflict from the two merge bases this left, although git merges the two cleanly. The merge brings in only Write weighted values with the lower-case keys rawweighting and value #93's JsonBuilderElement.java and JsonBuilderTests.java, which this branch never touched, and every file of this branch is unchanged.

Neither merge deletes a file.

Red runs

Run 42729019549 runs each set of tests on the code before its fix, in five jobs, and each job fails as expected.

• c72baac's tests on Build a cloud engine whose product the resource key lacks, and log that once #84's code fail all nine CloudRequestEngineDiscoveryTests methods, the three validateErrorHandling cases, both _Server_Unavailable_At_Build tests, loadProperties_notLoaded and the three DiscoveryNotComplete_LoadsTheMetadataLater tests.
• 4f62134's three tests fail on 4defa2a's code.
• be26bcf's two tests fail on bef79b0's code.
• 1e1dae8's test fails cases 9 to 12, being 408 and 429 from each request, on 7bc481c's code, which made those two statuses transient.
• 127b173's test fails on 1e1dae8's code for accessibleproperties only, because the evidencekeys request did not read its body as JSON and already raised a CloudRequestException for a 4xx.

Full build

The full build and test matrix passed every job other than macOS on head 8a19907 in run 42828763704 on 10 November 2026, run as a dry run so nothing merged. Configure and all nine build jobs passed, and all 10 Setting 'PullRequestSha' lines are 8a19907. The earlier build on 5a6fc84 was run 42800334955. macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.