feat(ai): close four session-reliability gaps for retained provider sessions - #12
Merged
Conversation
Work item 1: conversation bootstrap no longer fails structurally while an assistant is streaming. The snapshot retry predicate compared row_version and stream_head, which a coalesced live delta advances at roughly the coalescer rate, so the bounded snapshot returned Conflict for exactly the sessions a user is most likely to open. The predicate now covers only fields the bootstrap actually returns, and the watermark is documented as a resume floor: nothing at or below it is missing, run and tool-call rows may lead it, and the message window never does. Work item 2: session-event streams now carry a typed terminal close envelope for every server-side end, a bounded per-session jittered grace window for an unavailable authorization dependency (authoritative denials still fail fast), and a periodic bounded durable head read so single-replica delivery no longer depends solely on the process-local wakeup channel. Work item 3.2: run_failed and run_recovery_required events now carry a bounded failure record with a stable code, a retryable flag, and the admission reason. The reader accepts the previous v1 payload and the new v2 payload and fails closed on anything else. Retry admission is computed from committed rows only; RecoveryRequired is never retryable, an absent failure code is never retryable, and a run that already produced a durable assistant message is refused. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Work item 3.3. A terminal run never resumes, so retry means authoring a new run over the same already-persisted user message under current policy, and acknowledge means durably dismissing a failure. Both are recorded in a new disposition entity rather than by mutating or deleting the source run, so the run row, its immutable attempt outcomes, and its session and inbox events all survive. At most one disposition wins per run, and replaying a client request id returns the original decision instead of authoring a second run. Retry admission is re-decided from committed rows inside the same transaction that authors the new run, and the new run carries a fresh principal reference so it never resurrects the source run's captured authority, lease, attempt, checkpoint, approval, or provider session. The terminal event now classifies from the error code that lands on the run row rather than the completion's outcome code. Those are equal on every path this crate writes, but a host-authored completion may supply an outcome code with no error code, and classifying from the outcome code would have let the event advertise a retry that the mutation then refused. Bumps AI_SCHEMA_MODULE_VERSION to 0.61.0 for the new entity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Work item 4.2. Invalidating a retained provider session silently costs the model its entire context while the durable transcript still renders as continuous, so the next message starts a fresh thread with no visible cause. Every invalidation funnel now appends a durable session event carrying only the server-owned reason class, and an explicit rebind appends its own. The payload uses the existing content-free tagged envelope, so it discloses no cursor, prompt, provider payload, tool argument, or authorization detail and needs no scope content key. The events participate in the ordinary sequenced stream, so they replay, retain, and authorize like every other session event rather than forming a second channel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Work item 4.1, durable-proof slice. Interruption previously returned nothing, so a caller could not distinguish "no live resource" from "requested" from "the retained thread is safe to keep". AiRunInterruptSettlement now carries that distinction and fails closed: retains_thread() is true only for proven settlement, which no adapter reports. The Codex app-server turn/interrupt response is an empty object, TurnStatus has a first-class interrupted value, and resumed threads page prior turns back, so an acknowledgement cannot distinguish a discarded partial turn from a retained one. Treating it as settlement would let the model carry content the durable transcript never recorded. In-flight interruption already invalidates the retained thread through the executor's own ambiguous-turn cleanup, and work item 4.2 now discloses that, so a mid-generation Stop is visible rather than silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Work item 4.3. The durable queue was already there: a deferred turn is scheduled as a retry, and claim_next reclaims queued and retry-scheduled runs, so delivery survives a host restart and never depended on the Deferred outcome reaching the executor. What was missing was convergence at the end of that allowance. Exhausting the bounded retry allowance while cleanup stayed pending made schedule_retry conflict, propagated the error, and left the run running until its lease expired into RecoveryRequired. That is both misclassified and stuck until an operator looks at it: nothing was executed on the attempt, so there is no uncertain effect to review. The run now closes as a clean visible failure with a stable code that the work-item-3 classifier admits for retry, because no provider call, tool, or persisted output happened. A stale fence still fails the terminal write, so ordinary expired-lease reconciliation keeps owning that case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records the exact new and changed public APIs, the bootstrap watermark resume-floor semantics, the retained-thread disclosure events, the retry admission rules, and the behaviours that changed with no API change. Also states what multi-replica delivery would additionally require, which this release deliberately does not supply. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test drove a 10ms durable head check and a 5ms poll loop against one in-memory SQLite database. Running alone that was fine; running inside the full provider lane it starved the concurrent send_message of the write lock and the commit failed. Slow the head check and the poll loop to rates that still observe several reauthorization attempts promptly. The production default head-check interval is 10s and was never implicated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
graphql-orm-macros is a proc-macro crate, so it exposes no library API surface
for cargo-semver-checks to analyze. The step used to pass by checking nothing;
cargo-semver-checks 0.50.0 now fails the job outright:
error: no crates with library targets selected, nothing to semver-check
note: skipped the following crates since they have no library target:
graphql-orm-macros
This blocked every pull request regardless of content. scripts/check-semver.sh
has always excluded the crate for exactly this reason, with the rationale in a
comment; CI simply never matched it. Macro compatibility stays covered by the
aligned package-version gate and the compile/trybuild fixture matrix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
Author
|
Added
That blocked every pull request regardless of content — this branch touches zero files under |
The fixture pinned its FixedClock to the real clock, but send_message stamps next_attempt_at from the real clock while claim_next compares against the fixed one. Both landing in the same second made the queued run eligible; a second boundary between them made claim_next return None and the test panic with "queued run should exist". Lead the fixed clock by two seconds, matching the existing run-cancellation fixture, which already does this for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The companions job relinks every graphql-orm-ai test binary once per provider feature combination. With full debug info that filled the runner disk and the linker died mid-link: collect2: fatal error: ld terminated with signal 7 [Bus error], core dumped reported as "could not compile" for three test binaries. It is the same root cause that failed the workspace release twice with an explicit "No space left on device"; adding one more test binary to the crate was enough to reach it here. Use line-tables-only rather than disabling debug info outright, so failure backtraces keep file and line numbers, which is what CI diagnosis needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes four session-reliability gaps reported by a downstream consumer of the
retained Codex app-server deployment. Two of the four reported premises turned
out to be wrong at the pinned revision; this describes what was actually
broken.
Conversation bootstrap failed structurally during streaming. The snapshot
retry predicate compared
row_versionandstream_head, both of which everycoalesced live delta advances at roughly the coalescer rate. The bounded
snapshot therefore returned
Conflictfor exactly the sessions a user is mostlikely to open, which a client cannot distinguish from a disconnection. The
predicate now covers only fields the bootstrap returns; a live delta appends a
session event and changes nothing in the payload. The watermark is documented
as a resume floor rather than an equality point.
Session-event streams ended as silence.
RecvError::Closedreturned withno terminal envelope, so "stream over, resubscribe" looked identical to network
silence. A typed
AiSessionStreamClosenow accompanies every server-side end,yielded before the existing error so current clients keep working.
Reauthorization distinguishes an unavailable dependency (bounded, per-session
jittered grace) from an authoritative denial (immediate), so one authorization
restart no longer drops every open stream simultaneously. A periodic bounded
durable head check makes single-replica delivery independent of the
process-local wakeup channel.
Failed runs carried no classification. The
run_failedandrun_recovery_requiredevents already existed and were already emitted in theterminal write's transaction — the report that they were missing was stale. What
they lacked was any failure record. The payload advances to a v2 tagged shape
carrying a stable code, a retryable flag, and an admission reason, with the
reader accepting v1 and v2 and failing closed otherwise.
retryAiRunauthors anew run over the same durable user message under current policy where
re-execution is provably safe;
acknowledgeAiRunFailuredismisses a failurewithout deleting audit history.
Retained provider threads were dropped silently. Ordinary user behaviour —
pressing Stop, a changed rule fingerprint, an exceeded budget — invalidated the
retained thread, and the next message started a fresh one while the durable
transcript still rendered as continuous. Every invalidation funnel and every
explicit rebind now appends a durable session event carrying only the
server-owned reason class.
Messages accepted during cleanup already converged across a restart — the
deferred turn is scheduled as a durable retry and
claim_nextreclaimsretry-scheduled runs, so delivery never depended on the
Deferredoutcomereaching the executor. The gap was at the end of that allowance: exhausting it
propagated a conflict and left the run to expire into
RecoveryRequired, whichis misclassified because nothing had executed, and stuck until an operator
intervened. It now closes as a clean, retryable, visible failure.
Interrupt handling reports what it proved via
AiRunInterruptSettlement, whichfails closed. No adapter reports
Settled: the Codex app-serverturn/interruptresponse is an empty object,TurnStatushas a first-classinterruptedvalue, and a resumed thread pages prior turns back throughthread/turns/list, so an acknowledgement cannot distinguish a discardedpartial turn from a retained one. Treating it as settlement would let the model
carry content the durable transcript never recorded.
Verification
Local command output, run on this branch:
cargo fmt --all -- --checkcargo test -p graphql-orm,-p graphql-orm-storage,-p graphql-orm-backup --features orm-sqlite,-p graphql-orm-operation-catalog -p graphql-orm-ai-tool-profilesscripts/check-ai-provider-lanes.sh testcargo test -p graphql-orm-ai --features graphql-case-pascal --test graphql_namingcargo clippy -p graphql-orm-ai --all-targetsacross all five providerfeatures plus
local-harness, warnings deniedcargo check -p graphql-orm-aiforpostgres,mssql, and the mixedgraphql-orm-ai/sqlite,graphql-orm/mssqllanegraphql-orm-ai,graphql-orm-operation-catalog,graphql-orm-ai-tool-profilespython3 scripts/check-documentation.py,python3 scripts/generate-workspace-inventory.py --check,python3 scripts/check-release-state.py,scripts/check-workspace-dependencies.sh,scripts/check-package-release-policy.sh <merge-base>,scripts/check-semver.sh <merge-base>,scripts/check-release-manifest.shNew regression coverage:
original
Conflictagainst the previous predicate and to pass against thenew one, plus a unit test pinning exactly which fields admit or reject a
snapshot;
fast-fail on denial;
Databasehandles over one SQLite file so their wakeup channels are genuinelyindependent;
RecoveryRequiredand for an unclassifiedfailure, acknowledgement after a refused retry, and revoked-principal denial;
from an earlier watermark;
to a clean visible failure once the allowance is exhausted.
Documentation impact
Check exactly one option and explain the result.
Explanation:
graphql-orm-ai0.82.0 with schema module 0.61.0. Addscrates/graphql-orm-ai/docs/session-reliability-adoption.mdas the adoptioncontract — exact new and changed public APIs, watermark resume-floor semantics,
retry admission rules, and the behaviours that changed with no API change —
linked from the component docs index. Updates the component CHANGELOG,
MIGRATION (schema module step, source-breaking changes, additive GraphQL
surface, new event types, v1-to-v2 payload), README, the conversation-bootstrap
section of
capability-discovery-and-execution.md, and the generated workspaceinventory. Also states explicitly what multi-replica delivery would additionally
require, which this change deliberately does not supply.