Conversation
A restore aborts entirely when the server cannot acquire a datastore
write connection within its acquisition timeout, surfacing as:
error finalizing write of 10 batches: rpc error: code = ResourceExhausted
desc = error acquiring connection from pool: failed to acquire in time
The condition is transient, and zed already treats ResourceExhausted as
retryable for unary calls in DialOptsFromFlags. ImportBulkRelationships is
excluded from the stream retry interceptor because the interceptor cannot
transparently replay batches already sent on the stream, so the restore
performs its own retries -- but isRetryableError never classified this
code, sending it to the unrecoverable branch and ending the run.
Classify ResourceExhausted as retryable so the batch is retried through
writeBatchesWithRetry. This is safe: the acquisition failure happens before
BeginTx, so the failed attempt committed no rows. Callback errors are
rolled back, and the transaction does not partially commit.
gRPC reports oversized messages with the same code, which retrying cannot
resolve, so those remain unretryable. The exclusion covers the message-size
templates gRPC emits on the send side, the receive side, and after
decompression -- isRetryableError also drives the backup export loops,
where a permanent send-side size failure must not be retried.
Note this does not make restores unconditionally succeed. The manual loop
still stops after defaultMaxRetries retries, and --max-retries does not
configure that outer ceiling.
Signed-off-by: ivanauth <ivan@authzed.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
zed backup restoreaborts the entire run when the server briefly cannot acquire a datastore write connection: