Repository navigation
Read audit: bound the batch commit and discard a stalled connection - #448
Merged
Merged
Conversation
added 3 commits
October 25, 2026 21:19
A batch commit now has a 10 s bound (below the 20 s read wait). When it does not finish, the batch is failed as the existing retryable unavailable_retryable (503) class, the drain continues with the next batch, and the connection the commit held is closed and its pool slot freed instead of being released to the pool (sqlx pings a released connection, and the ping waits for the abandoned statement's reply). RecoveringConnection::execute_idempotent, whose only caller is the read-audit batch, acquires the PostgreSQL pool connection itself and detaches it if its future is dropped before the result arrives. Retry on a lost connection is unchanged. The timeout is logged as a warning with error_class read_audit_batch_timeout (batch size, elapsed and timeout in ms) and counted in tinycloud_read_audit_batch_timeout_total. Details: TC-2035
…itly Revert RecoveringConnection::execute_idempotent to its original form (the explicit acquire bypassed sea-orm's metric callback). The held pool slot is fixed at its root in the vendored sqlx-core: the release ping is bounded by a named timeout (5 s, per-pool option) and the connection is closed hard, releasing its permit, when it does not answer. Details: TC-2035
…meout # Conflicts: # CHANGELOG.md
samgbafa
marked this pull request as ready for review
October 26, 2026 10:19
Contributor
Author
|
APPROVE: see TC-2035 (reviewed exact head 52bbee1) |
Contributor
Author
This was referenced Oct 26, 2026
This was referenced Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A read-audit batch commit now has a client-side bound of 10 s (below the 20 s read wait). A commit that does not finish in that time is failed as the existing retryable
unavailable_retryable(503) class, the drain continues with the next batch, and, through the bounded release ping below, the connection the commit held is closed and its pool slot freed instead of being held.What changes
read_audit.rs:commit_batchwraps the batch commit (connection acquire included) intokio::time::timeout; the failed-batch path is shared with other batch failures. A timed-out batch is not requeued; its reads already got the retryable error. No read's data is returned without a committed record. Abandoned records queued behind the batch are skipped when the next batch forms, as before.vendor/sqlx-core(documented inPATCH.md): the ping that tests a released pool connection is bounded (named const, 5 s, per-poolrelease_ping_timeoutoption). sqlx returns a dropped statement's connection to the pool through that ping, and the ping waits for the abandoned statement's reply, so a stalled connection, its release task and its pool slot were held with no bound. Past the bound the connection is closed hard (itself bounded) and its permit released; the pool opens a replacement. This is pool-wide: any released connection whose ping does not answer in 5 s is closed instead of held.db_pool.rsandCargo.lockare unchanged.error_class = "read_audit_batch_timeout"(batch size, elapsed ms, timeout ms; counts and durations only) and counted intinycloud_read_audit_batch_timeout_total. The aggregateRead audit batch failedwarning also carries this class. The timeout warning is logged with its own error_class; log-shipper classification follows.Idempotence: the batch is one
INSERT ... ON CONFLICT DO NOTHINGstatement on keys fixed by the batch, so running it again after it committed writes nothing. If a timed-out commit did land on the server, its rows are extra audit rows and a client retry of the same record leaves no duplicates and no gaps.Tests (new only)
postgres_read_audit_timed_out_commit_is_idempotent_on_retryon the PostgreSQL lane).Details: TC-2035