[jwrosewell]

Java's usage sharing has no back-off, so every message is sent, and lost, while the service is failing. This adds the back-off that 51degrees-dotnet#88 and 51degrees-go#84 already have, with the same five settings and defaults.

Problem

ShareUsageElement.sendUsageData sent every batch it took from the queue, and a failed send was logged and the batch dropped. During an outage every batch is lost, and each one is another request to a failing service. .NET had a back-off from pipeline-dotnet#218, but it stopped processing as well as sending and was removed in pipeline-dotnet#261.

What the specification says

The Back-off after failed sends section of usage-sharing-element.md, pinned at 8aa2de01 on specifications rc, came in with specifications#97, which has merged. Its six rules and where each one lives in Java:

Rule	Where
1. The failures number within the window pauses sending, and successes between them do not reset the count	ShareUsageBackOff.failed
2. The first send after a pause is a trial, and one failure multiplies the delay up to the maximum	ShareUsageBackOff.failed
3. A success ends the back-off, and the next pause starts from the initial delay	ShareUsageBackOff.succeeded
4. Evidence is still queued during a pause, and dropped only when the queue is full	ShareUsageBase.processInternal
5. When a pause ends, sending starts again without waiting for a request, and runs until fewer than the minimum entries are queued	ShareUsageElement.sendUsageData with ShareUsageBase.waitWhilePaused
6. A warning when sending pauses or a pause is lengthened, information when it resumes	ShareUsageBase.recordSendResult

The section also says a send fails when the request cannot be made, has no answer within the time the implementation allows, or is answered with anything other than 200, and that the batch that failed is dropped rather than put back on the queue.

What changed

Setting	Default	Accepted values
setFailuresToEnterRecovery	10	1 to 100
setFailuresWindowSeconds	100	1 or more
setInitialRecoveryDelaySeconds	2.0	more than 0
setMaxRecoveryDelaySeconds	300.0	more than 0
setRecoveryMultiplier	2.0	1 or more

• ShareUsageBackOff holds the rules. Each method takes the time from the caller, on System.nanoTime through ShareUsageBackOff.now, so a change to the system clock cannot lengthen or skip a pause, and the tests set the time.
• ShareUsageBase gains the protected waitWhilePaused and recordSendResult, which sendUsageData calls before and after each message. A class derived from ShareUsageBase with its own consumer gets no back-off until it calls them.
• The consumer waits out a pause on a CountDownLatch that closing releases, so closing ends a pause at once and sends nothing more, and it logs how many entries were not sent.
• New constructors on ShareUsageBase and ShareUsageElement take the five settings, and the existing ones pass the defaults, so code built against them is unchanged. The defaults are in Constants under the .NET names.
• sendAsXML reports the status and the content of a failed answer, which the specification requires to aid troubleshooting, and DataUploaderHttp gains getResponseContent and a read timeout so a send that is never answered ends rather than stopping all sending.
• A full queue is logged for the first entry discarded and then every QUEUE_FULL_LOG_EVERY entries, never one message per entry, which is what the Error handling section forbids.

The branch also carries the usage sharing options to share all Evidence and all query string parameters, to read keys without regard to case, and to keep one tracker cache.

Tests

The tests were written first, in ShareUsageElementBackOffTests.java, ShareUsageBackOffTests.java, ShareUsageOptionsTests.java, ShareUsageSendTimeoutTests.java and DataUploaderHttpTests.java.

• Red, on rc with only the two tests that compile there: run 42347851766. ShareUsageElement_BackOff_PausesThenDrainsQueue saw thirteen sends 0.7 seconds after the tenth failed send where ten were expected, and ShareUsageElement_BackOff_CloseWhilePausedDoesNotWaitOrSend saw a send during the pause. The other 27 usage sharing tests passed, with one skipped as before.
• Green: run 42348489883, all 39 usage sharing tests pass on Ubuntu and Windows, with the same one skipped.

The two tests that set the delays need the new constructor, so they were added with the change, as were ShareUsageBackOffTests and the setter checks.

Two findings fixed since those runs

The empty deque

ShareUsageBackOff.failed tested failures.peekFirst() in the loop condition, which unboxes a Long and would throw a NullPointerException on an empty deque. The deque is now tested for emptiness first, and the comment says the constructor refuses a window below one second, so the case cannot be reached. There is no red run for it, because nothing the constructor allows reaches the branch and a test that reached it would have to weaken the constructor first. 51degrees-python#53 has the same guard for the same reason.

A new test broke an existing one, and the mechanism is worth knowing

The first three builds of this branch failed on HttpClientTests.VerifyHeaders, with the cloud answering 401 "This resource key is not authorised for the domain ''". The same test passed in another draft's build on the same rc at 12:46, 12:49 and 12:51, spanning the first failure, so the service and the key were working and this branch was the difference.

HttpClientDefault.connect sets sun.net.http.allowRestrictedHeaders to true and then opens the connection. Origin is on the JDK's restricted header list, and sun.net.www.protocol.http.HttpURLConnection reads that property once, in its static initialiser, into a static final field. Whichever code first loads that class therefore decides for the whole run whether Origin is sent.

The new DataUploaderHttpTests opens an HttpURLConnection and did not set the property. It sorts before HttpClientTests and the log shows it running sixteen seconds earlier in the same JVM, so it loaded the class first, the property was unset, Origin was dropped from then on, and the cloud saw an empty domain. The test now sets the same property in a @BeforeClass, so the class is loaded with the value HttpClientDefault needs whichever test runs first.

The three failures, for the record: 43020842228 twice and 43031127409. In the build below HttpClientTests passes two of two in every one of the nine jobs.

Full build

Run 43033176993, success, on head 17f66db227773271f461d7e2a45e265d98751208. Configure, the nine build and test jobs (Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu ARM with Java 17, and Windows with Java 8, 11, 17 and 21) and Complete all passed. All 10 Setting 'PullRequestSha' lines are 17f66db227773271f461d7e2a45e265d98751208, the head, and the Configure log reads "Left out 6 macOS entries".

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

The branch takes rc at ce406b2, with no conflict.

What remains

Neither is a defect in this change, and both want an issue of their own.

1. The Origin fragility is older than this change. DataUploaderHttp already opened connections before this draft, so an application that shares usage before anything calls HttpClientDefault.connect is already losing Origin. Fixing it properly means HttpClientDefault not depending on which class loads first. This change only stops the tests tripping it.
2. ShareUsageBuilderBaseTest uses JUnit 4 annotations and did not run in any of the runs above, which ran every other usage sharing test class. Whether the build runs JUnit 4 tests at all has not been checked.

For review

1. Two commits in this branch add and then remove jdkhttp.java, a file copied into the clone by mistake while working out the Origin failure. The tree is clean and the diff against rc never held it, and tidying the history would have needed a force push.
2. The full queue is logged on a count where .NET logs once each time the queue fills and again only after an entry is accepted. Both satisfy the rule that a message must not be logged for each entry discarded, so this is a difference rather than a fault.

The same back-off in the other languages

Language	Change
.NET	#88, merged
Go	#84, merged
Python	#53, open beside this one
Node	#36, open

Related

• #61 makes Java share a repeated evidence set at most once in each interval.

Still to do

• Review by someone other than the author.