[jwrosewell]

Problem

Node's usage sharing had no back-off. While the service was down or
overloaded, every batch was sent and lost, and the queue was unbounded, so a
busy server could hold a message for every request it served. Four smaller
faults sat with it.

• The queue had no maximum, so there was nothing to log when it filled.
• A message was built with shareData.join(), which is join(','), so every
  message carried a comma between each <Device> element.
• A message took the whole queue however large it had grown.
• A send that was answered with anything other than 200 was logged at debug as
  Usage data sent, so a service answering 500 to everything looked like a
  service working.

.NET and Go already back off, so the same outage cost Node users their usage
data and cost .NET and Go users nothing.

What the specification says

usage-sharing-element.md, Back-off after failed sends,
at the fixed commit 8aa2de01 on rc.

Repeated failed sends MUST pause sending, so that a service that is down or
overloaded is not sent a message for every batch, and so that usage data
waits in the queue rather than being sent and lost. Only sending pauses.
Processing MUST NOT wait for, or be stopped by, the back-off.

A send fails when the request cannot be made, has no answer within the time
the implementation allows, or is answered with anything other than 200. The
batch that failed is dropped. It is not put back on the queue.

1. When the failures to enter recovery number of sends have failed within
   the last failures window seconds, sending MUST pause for the initial
   recovery delay. Failures older than the window MUST NOT be counted.
   Successful sends between the failures do not reset the count ...
2. The first send after a pause ends is a trial. If it fails, the delay MUST
   be multiplied by the recovery multiplier, up to the maximum recovery
   delay, and sending MUST pause again for the new delay. One failed trial
   is enough.
3. A successful send MUST end the back-off. The count of failures starts
   again from zero, and the next pause starts again from the initial delay.
4. During a pause, Evidence MUST still be added to the queue. New Evidence
   is discarded only when the queue is full.
5. When a pause ends and at least the minimum entries per message are
   queued, sending MUST start again without waiting for more requests to be
   processed, and MUST continue until fewer than the minimum entries per
   message are queued ...
6. A message SHOULD be logged as a warning when sending pauses or a pause is
   lengthened, giving the delay, and as information when sending resumes.

The configuration options
table at the same commit gives the five recovery options and their defaults
(10, 100 seconds, 2.0 seconds, 300.0 seconds, 2.0), says an implementation
MUST reject a value outside the range given, gives the maximum queue size as
1000 or ten times the minimum entries per message if that is larger, and says
a message holds from the minimum entries per message to twice that number.

Session tracking
at the same commit settles the repeat interval boundary.

• Evidence matching an entry shared less than the interval ago MUST NOT be
  shared, and the entry's time MUST NOT change.
• Evidence matching an entry shared the interval or more ago MUST be shared,
  and the entry's time MUST be set to the time it is shared.

Change

shareUsageBackOff.js,
a new class holding the rules and nothing else.

• The constructor takes the five options with the defaults above and throws a
  RangeError for a value outside the range.
• failed(now) records a failure, drops failures older than the window, and
  when the count reaches failuresToEnterRecovery empties the list and starts
  a pause at the initial delay. While a pause is set, a failure is the trial
  failing, so the delay is multiplied by the multiplier, capped at the
  maximum. It returns the new pause, or zero when the failure started nothing.
• succeeded() clears the delay and the resume time and says whether a
  back-off ended.
• remaining(now) gives what is left of the pause.
• Each method takes the time from the caller, and ShareUsageBackOff.now()
  reads performance.now(), which is monotonic, so a change to the system
  clock cannot lengthen or skip a pause.

shareUsage.js

• The five recovery options, maximumQueueSize and requestTimeout are
  settings of the element. maximumQueueSize defaults to 1000 or ten times
  requestedPackageSize, whichever is larger, and a value below
  requestedPackageSize throws, because the queue could then never hold a
  message.
• addToShareUsage drops a new entry when the queue is full and logs a
  warning for the first one dropped and then once every 1000, never one for
  each. The count starts again once the queue has room. Adding is the
  processing path, so processing never waits for a pause.
• sendShareUsage starts nothing while a message is in flight, and while a
  pause is running it sets a timer for the end of the pause instead. The
  timer is unrefed, so it never holds the process open, and a pause longer
  than about 24 days is waited out in parts.
• sendNext takes at most twice requestedPackageSize entries, sends them,
  and on success sends the next until fewer than requestedPackageSize are
  left, so a queue that filled during a pause drains at once.
• post is the send. It gives the request a timeout, treats an answer other
  than 200 as a failure and logs the status with the first 500 characters of
  the content, treats a request that cannot be made or that throws as a
  failure, and calls back exactly once. The entries it took are not put back,
  as the page says.
• The message is built with join(''), so there is no longer a comma between
  the <Device> elements.

shareUsageTracker.js

• match compares with >= rather than >, so evidence first shared exactly
  one interval ago is shared again, which is what the session tracking section
  quoted above asks for. The time is still only moved on when the repeat is
  shared.

The types in types/shareUsage.d.ts and the new types/shareUsageBackOff.d.ts
are regenerated to match.

The same change in the other languages

Language	Pull request
.NET	#88
Go	#84, which also fills a message up to twice the minimum

The option names and the five defaults are the same in all three.
specifications#97, which
made the back-off a requirement, names this draft as the Node change. Java
#62 and Python
#53 carry it for those
two languages and are not mine.

Tests

shareUsageBackOff.test.js
runs a local HTTP server on its own port and sets what that server answers, so
nothing leaves the runner. Eleven tests drive the element (pauses then drains
the queue, the delay grows to the maximum, a full queue drops new evidence and
logs without a message per entry, a request with no answer does not stop
sending, a failed answer logs the status and the content, the message has no
text between devices and holds up to twice the package size, a request that
throws counts as failed, now follows the monotonic clock, and invalid
settings are rejected). Six more drive the rules in ShareUsageBackOff
directly.

shareUsageTracker.test.js
is new and holds the interval boundary, with Date.now mocked, at the
interval less one, at the interval and at the interval plus one, and that a
repeat inside the interval leaves the stored time alone.

Red. Run 43020800406,
on tmp/node-36-red. The branch is rc at 8b9fc3d with the two test files
and shareUsageBackOff.js taken from this head, and nothing else. The new
class file is there because a test file cannot load a module that does not
exist, and nothing in rc calls it, so rc's element is unchanged. 11 of the
21 tests failed, being nine of the eleven that drive the element and two of
the four tracker tests.

The 10 that passed are the six unit tests of the new class, which is the code
under test in them and is present on the red branch, now follows the monotonic clock, which is also a test of the class, the two tracker tests
either side of the boundary that > and >= answer the same way, and
request with no answer does not stop sending, which passes on rc because
rc starts a send on every call to addToShareUsage whatever came before.

Green. Run 43023230776,
on tmp/node-36-green, which is this head. 21 of 21 passed.

Full build. Run 43021227015,
on tmp/pr-matrix-36, success. Every one of its seven
Setting 'PullRequestSha' lines reads
77c6ab113d61f96a036dc06e648e1c47b24dabf0, which is this head. Configure
printed "Left out 2 macOS entries". The branch it built from was rc at
8b9fc3d, and rc has moved on since. Nothing rc has taken since touches
the three files this changes, and the draft merges into the current rc
with no conflict.

macOS was left out of this draft's full build under the 9 November decision.
The rc head is built with macOS before it merges into main.

What remains

• This draft and
  51degrees-node#45
  both change shareUsage.js in the constructor and in the two methods that
  read the evidence, so whichever merges second needs the other merged into it
  first. Neither depends on the other for its behaviour.
• The cleanup rule in the same section of the page, that cleanup must not wait
  for a pause and must not send, has nothing to attach to in Node, because
  Node elements have no cleanup step.
  specifications#97
  records that and does not change it.