fix: retry transient source-read and segment-copy failures in UploadPartCopy#133
Merged
Conversation
…artCopy A single truncated frame GET (backend drops a long-lived connection mid-body: ClientPayloadError 'received 8331264 of 8388636 bytes') or a 200-with-embedded-InternalError UploadPartCopy failed the whole client part. botocore's retry layer only covers a request up to the response headers, so body-read failures had no retry at all. During the 2026-07-16 Scylla backup this failed 12/13 hosts' upload jobs. - read_source_bytes(): full-body ranged GET with retry on transient transport/backend errors, used by frame reads and single-object reads - _stream_raw_source_with_resume(): raw copy-source streaming resumes with a ranged GET at the exact byte offset already yielded - passthrough copy_segment: retry upload_part_copy on retryable errors (same part number + range, safe to re-issue) - new env knobs: S3PROXY_SOURCE_READ_ATTEMPTS (4), S3PROXY_SOURCE_READ_BACKOFF (0.5s)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
During the 2026-07-16 Scylla backup, 12/13 hosts' upload jobs failed. Root cause chain:
ClientPayloadError: received 8331264 of 8388636 bytes(deterministic offset, ~4 min in), or an UploadPartCopy streams a 200 then embedsInternalError: The server did not respond in time.Fix
read_source_bytes()inhandlers/base.py: ranged GET + full body read with retry (4 attempts, exponential backoff) on transient transport/backend errors. Used by_iter_multipart_plaintextframe reads and_download_encrypted_single._stream_raw_source_with_resume()inmultipart/copy.py: the raw copy-source stream resumes with a ranged GET at the exact byte offset already yielded — bytes are never re-fetched or duplicated.copy_segment: retriesupload_part_copyon retryable errors (same part number + range → idempotent).is_retryable_source_error) covers aiohttp payload/connection errors, botocore timeout/streaming errors, and S3 error codes InternalError/SlowDown/RequestTimeout/ServiceUnavailable/BadGateway. NoSuchKey/AccessDenied and other client errors are never retried.S3PROXY_SOURCE_READ_ATTEMPTS(default 4),S3PROXY_SOURCE_READ_BACKOFF(default 0.5s).Tests
tests/unit/test_source_read_retry.py(8 tests): truncated-body retry & give-up, non-retryable passthrough, error classification, stream resume at exact offset (whole-object and ranged), non-retryable propagation, and an end-to-end passthrough copy that survives one 500'd segment copy + one truncated frame GET and round-trips the plaintext. Full unit suite: 585 passed.🤖 Generated with Claude Code