Renew the registry token when a layer download fails mid-import

[100milliongold]

Fixes #278.

The problem

docker::_download authenticates once (src/docker.sh:145) and turns the result into a -K file
that every layer request reuses. The token's lifetime belongs to the registry: 600 s on nvcr.io,
300 s on Docker Hub. A blob request answers 307 and the transfer runs against a signed CDN URL, so
the token is only needed at the moment each layer's redirect is requested. On an image with many
layers, a request issued near the end of a long download happens after the token has expired and
fails with 401, and the import aborts with no image produced.

Neither retry layer recovers from it: curl --retry does not retry a 401, and the
parallel --retries 2 pass re-reads the same unrefreshed -K file.

The change

Wrap the layer download in a loop that renews the token and resumes when a pass fails. Layers
already in the cache exit early on the next pass (src/docker.sh:103), so the download continues
where it stopped rather than starting over.

Resuming is allowed only while the cache keeps growing. That keeps the loop bounded by the number of
layers without adding a retry count, and a download failing for any other reason still stops with an
error, as it does today.

No new configuration, and nothing changes on the success path: a run that does not fail
authenticates once, exactly as before.

Testing

Enroot 4.2.1 from the release .deb, with this docker.sh mounted over /usr/lib/enroot/docker.sh,
importing docker://library/python:3.12 (7 layers, 900 MB squashfs). Waiting for a real token to
expire is not reproducible, so the token file was overwritten with an invalid one three seconds into
the layer download, which produces the same 401 the issue reports.

Run	Connections	Injected fault	Exit	Image	Authentication succeeded	Renewals	401s
unpatched 4.2.1	2	token invalidated once	3	none	1	n/a	3
patched	2	token invalidated once	0	899850240 bytes	2	1	5
patched	3	token invalidated once	0	899850240 bytes	2	1	4
patched	2	none	0	899850240 bytes	1	0	0
patched	2	token invalidated every 0.2 s	0	899850240 bytes	3	2	4
patched	10	token invalidated once	0	899850240 bytes	1	0	0
patched	2	blob CDN host blackholed after the download starts	1	none	2	1	0
unpatched 4.2.1	10	CDN blackholed and token invalidated together, so the retry pass carries the dead token	7	none	1	n/a	7
patched	10	same	1	none	2	1	3

The unpatched run fails with the reported symptom:

[INFO] Downloading 7 missing layers...
curl: (22) The requested URL returned error: 401

The patched run recovers:

[INFO] Downloading 7 missing layers...
curl: (22) The requested URL returned error: 401
[INFO] Download interrupted, renewing the registry token and resuming
[INFO] Authentication succeeded
[INFO] Extracting image layers...

The ENROOT_MAX_CONNECTIONS 10 row is not a null result. With ten connections and seven layers,
every redirect is requested before the token is invalidated, so the run cannot be affected however
long the transfers take afterwards. That is the same property the issue describes: what matters is
when a layer's 307 is requested, not how long the image takes to download. It also means the bug
needs more layers than connections, which is the shape of the reported failure (35 layers, 10
connections by default).

The last row is the bound: with the CDN unreachable, the second pass adds nothing to the cache and
the import stops instead of renewing forever.

[INFO] Download interrupted, renewing the registry token and resuming
[INFO] Authentication succeeded
[ERROR] Could not download all the layers of registry-1.docker.io/library/python

Raising ENROOT_MAX_CONNECTIONS is not a workaround

Since the failure needs more layers than connections, the obvious question is whether opening more
connections avoids it. It does not, and the last two rows of the table are that case: connections
above the layer count, every 307 requested before the token was invalidated, and the import still
died with seven 401s on 4.2.1. What carried the dead token was the retry. A transfer that fails for
any transient reason is re-run by parallel, and the re-run starts with a fresh 307.

Three more reasons it is not a knob to turn:

• The layer count is not known until the manifest is read, which happens after
  ENROOT_MAX_CONNECTIONS has taken effect, so "set it above the layer count" is not something a
  caller can act on.
• Matching it to the layer count means 35 simultaneous connections for a 35-layer image, which
  splits the available bandwidth and runs into whatever concurrency limit the registry enforces.
• A token can arrive with most of its life already spent. Ten consecutive anonymous requests to
  nvcr.io returned 600 379 600 378 98 598 600 596 96 595 seconds of remaining validity, because
  the server hands out cached tokens. With 96 s left, the first layer's redirect can already be past
  expiry, and the connection count has no bearing on that.

The patched / 10 row is worth reading precisely: the injected CDN outage outlives the renewal, so
the second pass adds nothing to the cache and the import stops with the bound message. The renewal
is not a way to survive an unreachable registry, only an expired token.

Interaction with the existing --retries

parallel --retries 2 runs a failing job twice before giving up, locally as well as over
--sshlogin; a joblog on this machine confirms two attempts per job. Those attempts happen before
control returns to the caller, so on an expiry each still-pending layer sends one more 401 before
the token is renewed. That is visible in the counts above: three connections and seven layers give
four 401s, or two attempts each for the two layers that had not been requested yet.

The cost is bounded by the number of pending layers and it buys a change that does not touch the
worker. Refreshing inside the worker would avoid the extra attempts, at the price described below.

An alternative I did not take

The refresh could live inside docker::_download_extract, next to the failing request, with a lock
on the token file so that ENROOT_MAX_CONNECTIONS workers do not each request a token. It needs
more plumbing: that function receives only curl arguments, so the registry, user, and manifest URL
would have to be passed in, and the token path could not be recomputed from $$ because the workers
are separate processes. Renewing in the caller needs none of that and reuses the existing resume
behavior. Happy to switch if you prefer the refresh closer to the request.