Skip to content

feat(ai): close four session-reliability gaps for retained provider sessions - #12

Merged
Dastari merged 10 commits into
mainfrom
agent/session-reliability-gema
Aug 20, 2026
Merged

feat(ai): close four session-reliability gaps for retained provider sessions#12
Dastari merged 10 commits into
mainfrom
agent/session-reliability-gema

Conversation

@Dastari

@Dastari Dastari commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Summary

Closes four session-reliability gaps reported by a downstream consumer of the
retained Codex app-server deployment. Two of the four reported premises turned
out to be wrong at the pinned revision; this describes what was actually
broken.

Conversation bootstrap failed structurally during streaming. The snapshot
retry predicate compared row_version and stream_head, both of which every
coalesced live delta advances at roughly the coalescer rate. The bounded
snapshot therefore returned Conflict for exactly the sessions a user is most
likely to open, which a client cannot distinguish from a disconnection. The
predicate now covers only fields the bootstrap returns; a live delta appends a
session event and changes nothing in the payload. The watermark is documented
as a resume floor rather than an equality point.

Session-event streams ended as silence. RecvError::Closed returned with
no terminal envelope, so "stream over, resubscribe" looked identical to network
silence. A typed AiSessionStreamClose now accompanies every server-side end,
yielded before the existing error so current clients keep working.
Reauthorization distinguishes an unavailable dependency (bounded, per-session
jittered grace) from an authoritative denial (immediate), so one authorization
restart no longer drops every open stream simultaneously. A periodic bounded
durable head check makes single-replica delivery independent of the
process-local wakeup channel.

Failed runs carried no classification. The run_failed and
run_recovery_required events already existed and were already emitted in the
terminal write's transaction — the report that they were missing was stale. What
they lacked was any failure record. The payload advances to a v2 tagged shape
carrying a stable code, a retryable flag, and an admission reason, with the
reader accepting v1 and v2 and failing closed otherwise. retryAiRun authors a
new run over the same durable user message under current policy where
re-execution is provably safe; acknowledgeAiRunFailure dismisses a failure
without deleting audit history.

Retained provider threads were dropped silently. Ordinary user behaviour —
pressing Stop, a changed rule fingerprint, an exceeded budget — invalidated the
retained thread, and the next message started a fresh one while the durable
transcript still rendered as continuous. Every invalidation funnel and every
explicit rebind now appends a durable session event carrying only the
server-owned reason class.

Messages accepted during cleanup already converged across a restart — the
deferred turn is scheduled as a durable retry and claim_next reclaims
retry-scheduled runs, so delivery never depended on the Deferred outcome
reaching the executor. The gap was at the end of that allowance: exhausting it
propagated a conflict and left the run to expire into RecoveryRequired, which
is misclassified because nothing had executed, and stuck until an operator
intervened. It now closes as a clean, retryable, visible failure.

Interrupt handling reports what it proved via AiRunInterruptSettlement, which
fails closed. No adapter reports Settled: the Codex app-server
turn/interrupt response is an empty object, TurnStatus has a first-class
interrupted value, and a resumed thread pages prior turns back through
thread/turns/list, so an acknowledgement cannot distinguish a discarded
partial turn from a retained one. Treating it as settlement would let the model
carry content the durable transcript never recorded.

Verification

Local command output, run on this branch:

  • cargo fmt --all -- --check
  • cargo test -p graphql-orm, -p graphql-orm-storage,
    -p graphql-orm-backup --features orm-sqlite,
    -p graphql-orm-operation-catalog -p graphql-orm-ai-tool-profiles
  • scripts/check-ai-provider-lanes.sh test
  • cargo test -p graphql-orm-ai --features graphql-case-pascal --test graphql_naming
  • cargo clippy -p graphql-orm-ai --all-targets across all five provider
    features plus local-harness, warnings denied
  • cargo check -p graphql-orm-ai for postgres, mssql, and the mixed
    graphql-orm-ai/sqlite,graphql-orm/mssql lane
  • warnings-denied, missing-docs-denied rustdoc for graphql-orm-ai,
    graphql-orm-operation-catalog, graphql-orm-ai-tool-profiles
  • python3 scripts/check-documentation.py,
    python3 scripts/generate-workspace-inventory.py --check,
    python3 scripts/check-release-state.py,
    scripts/check-workspace-dependencies.sh,
    scripts/check-package-release-policy.sh <merge-base>,
    scripts/check-semver.sh <merge-base>,
    scripts/check-release-manifest.sh

New regression coverage:

  • bootstrap under concurrent session-event churn, verified to reproduce the
    original Conflict against the previous predicate and to pass against the
    new one, plus a unit test pinning exactly which fields admit or reject a
    snapshot;
  • stream close envelope on authoritative denial;
  • reauthorization blip survival with retries proven to have occurred, then
    fast-fail on denial;
  • durable replay fallback delivering with no in-process wakeup, using two
    Database handles over one SQLite file so their wakeup channels are genuinely
    independent;
  • retry idempotency, refusal for RecoveryRequired and for an unclassified
    failure, acknowledgement after a refused retry, and revoked-principal denial;
  • failed-run event and failure record replaying after a process restart;
  • retained-thread invalidation disclosed on the sequenced stream and replayed
    from an earlier watermark;
  • cleanup-pending deferral converging through a durable retry, and converging
    to a clean visible failure once the allowance is exhausted.

Documentation impact

Check exactly one option and explain the result.

  • Documentation updated
  • No documentation impact

Explanation: graphql-orm-ai 0.82.0 with schema module 0.61.0. Adds
crates/graphql-orm-ai/docs/session-reliability-adoption.md as the adoption
contract — exact new and changed public APIs, watermark resume-floor semantics,
retry admission rules, and the behaviours that changed with no API change —
linked from the component docs index. Updates the component CHANGELOG,
MIGRATION (schema module step, source-breaking changes, additive GraphQL
surface, new event types, v1-to-v2 payload), README, the conversation-bootstrap
section of capability-discovery-and-execution.md, and the generated workspace
inventory. Also states explicitly what multi-replica delivery would additionally
require, which this change deliberately does not supply.

Toby Martin and others added 8 commits August 20, 2026 02:23
Work item 1: conversation bootstrap no longer fails structurally while an
assistant is streaming. The snapshot retry predicate compared row_version and
stream_head, which a coalesced live delta advances at roughly the coalescer
rate, so the bounded snapshot returned Conflict for exactly the sessions a user
is most likely to open. The predicate now covers only fields the bootstrap
actually returns, and the watermark is documented as a resume floor: nothing at
or below it is missing, run and tool-call rows may lead it, and the message
window never does.

Work item 2: session-event streams now carry a typed terminal close envelope
for every server-side end, a bounded per-session jittered grace window for an
unavailable authorization dependency (authoritative denials still fail fast),
and a periodic bounded durable head read so single-replica delivery no longer
depends solely on the process-local wakeup channel.

Work item 3.2: run_failed and run_recovery_required events now carry a bounded
failure record with a stable code, a retryable flag, and the admission reason.
The reader accepts the previous v1 payload and the new v2 payload and fails
closed on anything else. Retry admission is computed from committed rows only;
RecoveryRequired is never retryable, an absent failure code is never retryable,
and a run that already produced a durable assistant message is refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Work item 3.3. A terminal run never resumes, so retry means authoring a new
run over the same already-persisted user message under current policy, and
acknowledge means durably dismissing a failure. Both are recorded in a new
disposition entity rather than by mutating or deleting the source run, so the
run row, its immutable attempt outcomes, and its session and inbox events all
survive. At most one disposition wins per run, and replaying a client request
id returns the original decision instead of authoring a second run.

Retry admission is re-decided from committed rows inside the same transaction
that authors the new run, and the new run carries a fresh principal reference
so it never resurrects the source run's captured authority, lease, attempt,
checkpoint, approval, or provider session.

The terminal event now classifies from the error code that lands on the run
row rather than the completion's outcome code. Those are equal on every path
this crate writes, but a host-authored completion may supply an outcome code
with no error code, and classifying from the outcome code would have let the
event advertise a retry that the mutation then refused.

Bumps AI_SCHEMA_MODULE_VERSION to 0.61.0 for the new entity.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Work item 4.2. Invalidating a retained provider session silently costs the
model its entire context while the durable transcript still renders as
continuous, so the next message starts a fresh thread with no visible cause.

Every invalidation funnel now appends a durable session event carrying only
the server-owned reason class, and an explicit rebind appends its own. The
payload uses the existing content-free tagged envelope, so it discloses no
cursor, prompt, provider payload, tool argument, or authorization detail and
needs no scope content key. The events participate in the ordinary sequenced
stream, so they replay, retain, and authorize like every other session event
rather than forming a second channel.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Work item 4.1, durable-proof slice. Interruption previously returned nothing,
so a caller could not distinguish "no live resource" from "requested" from
"the retained thread is safe to keep".

AiRunInterruptSettlement now carries that distinction and fails closed:
retains_thread() is true only for proven settlement, which no adapter reports.
The Codex app-server turn/interrupt response is an empty object, TurnStatus has
a first-class interrupted value, and resumed threads page prior turns back, so
an acknowledgement cannot distinguish a discarded partial turn from a retained
one. Treating it as settlement would let the model carry content the durable
transcript never recorded.

In-flight interruption already invalidates the retained thread through the
executor's own ambiguous-turn cleanup, and work item 4.2 now discloses that,
so a mid-generation Stop is visible rather than silent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Work item 4.3. The durable queue was already there: a deferred turn is
scheduled as a retry, and claim_next reclaims queued and retry-scheduled runs,
so delivery survives a host restart and never depended on the Deferred outcome
reaching the executor. What was missing was convergence at the end of that
allowance.

Exhausting the bounded retry allowance while cleanup stayed pending made
schedule_retry conflict, propagated the error, and left the run running until
its lease expired into RecoveryRequired. That is both misclassified and stuck
until an operator looks at it: nothing was executed on the attempt, so there is
no uncertain effect to review.

The run now closes as a clean visible failure with a stable code that the
work-item-3 classifier admits for retry, because no provider call, tool, or
persisted output happened. A stale fence still fails the terminal write, so
ordinary expired-lease reconciliation keeps owning that case.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records the exact new and changed public APIs, the bootstrap watermark
resume-floor semantics, the retained-thread disclosure events, the retry
admission rules, and the behaviours that changed with no API change. Also
states what multi-replica delivery would additionally require, which this
release deliberately does not supply.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test drove a 10ms durable head check and a 5ms poll loop against one
in-memory SQLite database. Running alone that was fine; running inside the full
provider lane it starved the concurrent send_message of the write lock and the
commit failed.

Slow the head check and the poll loop to rates that still observe several
reauthorization attempts promptly. The production default head-check interval
is 10s and was never implicated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
graphql-orm-macros is a proc-macro crate, so it exposes no library API surface
for cargo-semver-checks to analyze. The step used to pass by checking nothing;
cargo-semver-checks 0.50.0 now fails the job outright:

  error: no crates with library targets selected, nothing to semver-check
  note: skipped the following crates since they have no library target:
        graphql-orm-macros

This blocked every pull request regardless of content. scripts/check-semver.sh
has always excluded the crate for exactly this reason, with the rationale in a
comment; CI simply never matched it. Macro compatibility stays covered by the
aligned package-version gate and the compile/trybuild fixture matrix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Dastari

Dastari commented Aug 20, 2026

Copy link
Copy Markdown
Owner Author

Added 4efa91d, a CI fix unrelated to the feature work but blocking this PR.

cargo-semver-checks 0.50.0 now fails outright when the only selected crate is a proc-macro crate, instead of silently checking nothing:

error: no crates with library targets selected, nothing to semver-check
note: skipped the following crates since they have no library target: graphql-orm-macros

That blocked every pull request regardless of content — this branch touches zero files under crates/graphql-orm-macros. scripts/check-semver.sh has always excluded the crate for the same reason, with the rationale in a comment; the CI job simply never matched it. Macro compatibility remains covered by the aligned package-version gate and the compile/trybuild fixture matrix.

Toby Martin and others added 2 commits August 20, 2026 06:33
The fixture pinned its FixedClock to the real clock, but send_message stamps
next_attempt_at from the real clock while claim_next compares against the
fixed one. Both landing in the same second made the queued run eligible; a
second boundary between them made claim_next return None and the test panic
with "queued run should exist".

Lead the fixed clock by two seconds, matching the existing run-cancellation
fixture, which already does this for the same reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The companions job relinks every graphql-orm-ai test binary once per provider
feature combination. With full debug info that filled the runner disk and the
linker died mid-link:

  collect2: fatal error: ld terminated with signal 7 [Bus error], core dumped

reported as "could not compile" for three test binaries. It is the same root
cause that failed the workspace release twice with an explicit "No space left
on device"; adding one more test binary to the crate was enough to reach it
here.

Use line-tables-only rather than disabling debug info outright, so failure
backtraces keep file and line numbers, which is what CI diagnosis needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Dastari
Dastari merged commit ce14eb0 into main Aug 20, 2026
10 checks passed
@Dastari
Dastari deleted the agent/session-reliability-gema branch August 20, 2026 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant