Skip to content

Retry transient read failures at the failing chunk - #3

Open
owenpkent wants to merge 3 commits into
mainfrom
retry-resume
Open

Retry transient read failures at the failing chunk#3
owenpkent wants to merge 3 commits into
mainfrom
retry-resume

Conversation

@owenpkent

Copy link
Copy Markdown
Owner

Implements the roadmap item "Retry the source, not just the read": a transient read failure no longer restarts the whole file.

What changed

  • The read-ahead thread in _copy_fanout now retries a failing chunk in place: it reopens the source (after an I/O error the handle's buffered state cannot be trusted), seeks back to the last chunk boundary it delivered, and reads on. One marginal sector costs a re-read of 8 MiB, not of a 79 GB clip.
  • No hasher rewind turned out to be needed, which the roadmap had anticipated. The hashers only ever see chunks that were read successfully, so the running checksum state is already at the resume point.
  • The whole-file retry in run() stays as the outer fallback for everything the chunk retry cannot reach: opening a target, a write to a blipping network destination, or a chunk that never reads good within its attempts. retry.py is unchanged, as the roadmap specified.
  • A chunk-level recovery is still reported, aggregated into one warning per file so a card with many marginal sectors cannot flood the report. Backoff sleeps run in slices so pause and cancel still take effect promptly.

Tests

Suite is 412 tests (was 409), all passing, ruff clean. New tests assert:

  • A mid-file failure causes exactly one reopen with a seek to the failed chunk boundary, the first chunk is never re-read, and the destination verifies byte-for-byte.
  • A chunk-level recovery still produces the "source may be failing" warning.
  • Exhausted chunk retries fall back to a whole-file restart, total attempts stay bounded at attempts x attempts, and no partial file is left behind.
  • Existing guarantees hold: permanent errors are never retried, progress is not double-counted, no partials survive.

Existing test fakes grew seek/close because the reader now manages handles directly instead of a with block.

Docs

docs/data-safety.md documented the old restart-the-file behaviour and is rewritten; ROADMAP moves the item to Done; CHANGELOG entry added; README flag table and CLI help no longer say "per file"; docs/performance.md's robocopy comparison notes the retry now resumes at the failing chunk.

A transient read no longer restarts the whole file. The read-ahead
thread reopens the source, seeks back to the last chunk it delivered
and reads on; the hashers only ever see chunks that were read
successfully, so the running checksum needs no rewind and one marginal
sector costs a re-read of 8 MiB rather than of the whole clip.

The whole-file retry stays as the outer fallback for everything the
chunk retry cannot reach: opening a target, a write to a blipping
network destination, or a chunk that never reads good within its
attempts. A chunk-level recovery is still reported as a job warning,
aggregated per file so a card with many marginal sectors cannot flood
the report.
The CLI help and the README flag table still described --retries as
attempts per file, and the robocopy comparison in docs/performance.md
did not mention that the adopted retry now resumes at the failing
chunk.
The security review of this branch confirmed the integrity guarantees
hold across the chunk retry, and surfaced two things worth recording
rather than fixing:

- The retry budget is per chunk, so a card with many marginal sectors
  accrues attempts x chunks reads in the worst case, not attempts x
  attempts. That is deliberate recovery behaviour and now stated in
  data-safety.md and the changelog, pinned by a test that a shared
  per-file budget would fail, and the exhaustion test no longer
  overstates its bound.
- A source modified mid-copy, or swapped between a retry's close and
  reopen, was never protected against by any single-pass copy tool,
  including this one before the chunk retry. It is now listed under
  'What is still not protected'.
@owenpkent

Copy link
Copy Markdown
Owner Author

Security review of this branch found no vulnerabilities. The integrity guarantees hold across the chunk retry: the resume offset counts only bytes actually delivered, the hashers and destination writes consume one stream, a shrunken or swapped-shorter source is caught by the size check, and the retry tally has no concurrent access (the reader thread is joined before it is read).

Two observations from the review are addressed in 4a08ef7 by documenting and pinning them rather than changing behaviour:

  • The retry budget is per chunk, so a card with many marginal sectors accrues attempts x chunks reads in the worst case. Deliberate recovery behaviour; now stated in data-safety.md, pinned by a new test that a shared per-file budget would fail.
  • A source modified mid-copy (or swapped between a retry's close and reopen) yields a copy that matches what was read, not any one version of the source. True of any single-pass copy tool and of this one before the chunk retry; now listed under "What is still not protected".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant