Retry transient read failures at the failing chunk - #3
Open
owenpkent wants to merge 3 commits into
Open
Conversation
A transient read no longer restarts the whole file. The read-ahead thread reopens the source, seeks back to the last chunk it delivered and reads on; the hashers only ever see chunks that were read successfully, so the running checksum needs no rewind and one marginal sector costs a re-read of 8 MiB rather than of the whole clip. The whole-file retry stays as the outer fallback for everything the chunk retry cannot reach: opening a target, a write to a blipping network destination, or a chunk that never reads good within its attempts. A chunk-level recovery is still reported as a job warning, aggregated per file so a card with many marginal sectors cannot flood the report.
The CLI help and the README flag table still described --retries as attempts per file, and the robocopy comparison in docs/performance.md did not mention that the adopted retry now resumes at the failing chunk.
The security review of this branch confirmed the integrity guarantees hold across the chunk retry, and surfaced two things worth recording rather than fixing: - The retry budget is per chunk, so a card with many marginal sectors accrues attempts x chunks reads in the worst case, not attempts x attempts. That is deliberate recovery behaviour and now stated in data-safety.md and the changelog, pinned by a test that a shared per-file budget would fail, and the exhaustion test no longer overstates its bound. - A source modified mid-copy, or swapped between a retry's close and reopen, was never protected against by any single-pass copy tool, including this one before the chunk retry. It is now listed under 'What is still not protected'.
Owner
Author
|
Security review of this branch found no vulnerabilities. The integrity guarantees hold across the chunk retry: the resume offset counts only bytes actually delivered, the hashers and destination writes consume one stream, a shrunken or swapped-shorter source is caught by the size check, and the retry tally has no concurrent access (the reader thread is joined before it is read). Two observations from the review are addressed in 4a08ef7 by documenting and pinning them rather than changing behaviour:
|
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the roadmap item "Retry the source, not just the read": a transient read failure no longer restarts the whole file.
What changed
_copy_fanoutnow retries a failing chunk in place: it reopens the source (after an I/O error the handle's buffered state cannot be trusted), seeks back to the last chunk boundary it delivered, and reads on. One marginal sector costs a re-read of 8 MiB, not of a 79 GB clip.run()stays as the outer fallback for everything the chunk retry cannot reach: opening a target, a write to a blipping network destination, or a chunk that never reads good within its attempts.retry.pyis unchanged, as the roadmap specified.Tests
Suite is 412 tests (was 409), all passing, ruff clean. New tests assert:
Existing test fakes grew
seek/closebecause the reader now manages handles directly instead of awithblock.Docs
docs/data-safety.mddocumented the old restart-the-file behaviour and is rewritten; ROADMAP moves the item to Done; CHANGELOG entry added; README flag table and CLI help no longer say "per file";docs/performance.md's robocopy comparison notes the retry now resumes at the failing chunk.