Skip to content

fix(realtime): ack an already-handled repeat below the seq anchor; engine conformance scenarios - #10

Closed
sanctrl wants to merge 2 commits into
mainfrom
fix/engine-cursor-and-gaps
Closed

sanctrl wants to merge 2 commits into
mainfrom
fix/engine-cursor-and-gaps

Conversation

@sanctrl

@sanctrl sanctrl commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What

Fix: an already-handled repeat below the seq anchor is acked again (default mode). A live message.new below the conversation's anchor (seq < next_expected_seq) was dropped with no ack even when this session had already dispatched that message id. The delivery-ack protocol acks every dedup hit, because the earlier dispatch is the proof, and the TypeScript SDK already does this. The frame is now acked again and is still not re-dispatched. A below-anchor frame that no handler has seen is unchanged: it is dropped unacked and the server re-offers it.

Added: conformance/v1/engine-scenarios.json with a harness (tests/test_conformance_engine.py). This is the same file, byte for byte, as in agentchat-typescript (agentchatme/agentchat-typescript#10). Each scenario runs in both modes. The scenarios cover:

  • a drain page with a gap and a socket drop during an earlier row's handler
  • the same with the drop during a later row's handler ([g1, g3, h1], h1 held: the cursor stops at g1). This one catches the bug in an SDK that routes rows one at a time, like Python
  • the same page with a drop inside the gap window
  • a handler that raises during a drain
  • an already-handled repeat below the live anchor, both as a frame and as a drain row

The invariants checked on every scenario are: nothing acknowledged that a handler did not finish, no double dispatch, and seq order per conversation. tests/test_conformance.py now lists six files. tray.sample.json and gate-prompt.expected.txt are untouched.

The drain-cursor bug fixed in the TypeScript PR does not exist here, because _drain_row_ackable already stops the cursor. This drain awaits each handler while it routes rows, so here the exposure is a close during a later row's handler. The drain-gap-close-during-later-handler scenario and test_drain_cursor_covers_only_handled_rows_when_the_socket_drops_mid_handler both pin the rule. Both fail if _drain_row_ackable is changed to stop only while a row is still buffered (the old TS rule). The first version of the unit test did not catch that.

Tests that fail on old code

Run against origin/main's src/:

  • test_realtime_engine.py::test_default_mode_acks_a_below_anchor_repeat_it_already_handled
  • test_conformance_engine.py::test_engine_scenario[below-anchor-duplicate-frame-default]. Acks were [1, 2, 3]; expected [1, 1, 2, 3].

All other new cases pass on old code too. They pin existing behaviour.

The CHANGELOG now spells out where the SDKs still differ. For a below-anchor frame no handler has seen, Python drops it unacked until the next reconnect re-offers it, while TS dispatches it late and acks it.

Not in this PR: keeping the seq anchor after an in-order frame

This is the same reason as in the TypeScript PR. The server never pushes a sender its own message, so a kept anchor would treat every peer message that follows the agent's own reply as a gap. Each of those messages would be held 2 s and trigger a REST gap-fill, and the agent's own reply would be handed to message.new handlers. A correct design needs a server signal. That is a decision for the core.

Checks

  • pytest on 3.9, 3.11 and 3.13: 334 passed, 1 skipped (live smoke)
  • 20 repeats of the realtime and conformance files on 3.9
  • ruff check src tests and mypy are clean

Proposed version: 1.2.1 (a fix). The version number is not edited here.

…gine conformance scenarios

Default mode dropped a live message.new below the conversation's seq anchor
without an ack, even when this session had already dispatched that message
id. The delivery-ack protocol acknowledges every dedup hit (the earlier
dispatch is the proof), and the TypeScript SDK does. It is now acked again,
still without re-dispatch. A below-anchor frame no handler has seen is
unchanged: dropped unacked, the server re-offers it.

Adds conformance/v1/engine-scenarios.json, run in both modes by a new
harness (identical bytes ship in the TypeScript SDK): drain gap + close
during an earlier handler or inside the gap window, a raising handler during
a drain, and an already-handled repeat below the live anchor as a frame and
as a drain row. Also pins the drain-cursor rule for a socket drop during an
earlier handler with a regression test.
…handler

Shared scenario (identical bytes in agentchat-typescript): page [g1, g3, h1],
h1's handler held, the socket drops, h1 released; the default-mode cursor
stops at g1. This drain awaits each handler while routing, so the drop lands
during a LATER row's handler and clears the parked g3 before the cursor is
computed. Copying a cursor that stops only while a row is still buffered
into _drain_row_ackable now fails this scenario and the reworked unit test
(the earlier-handler shape did not catch it).

CHANGELOG: say where the SDKs still differ on a below-anchor frame no
handler has seen.
@sanctrl

sanctrl commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Folded into #12 (same reviewed head, merged conflict-free with the other two) so main gets one merge instead of three. Closing in favour of #12.

@sanctrl sanctrl closed this Nov 8, 2026
sanctrl added a commit that referenced this pull request Nov 8, 2026
…erved gate question fixtures (folds #9, #10, #11) (#12)

* conformance(v1): the served gate question records reply after the send

tray.sample.json carries the server's gate question as of
agentchat-platform/agentchat#152, and gate-prompt.expected.txt is its
rendering. Two lines changed on the server: defer states the server-owned
ladder (15 min, 24 h, stop) instead of "say when to look again", and the
verdict line records reply after the answer is sent instead of "the send
is the verdict" (a send marks nothing read).

The tool-less triage preamble is unchanged: it already tells the model not
to call agentchat_decide and that the host records the verdicts.

Byte-identical to agentchat-typescript's conformance/v1.

* fix(realtime): ack an already-handled repeat below the seq anchor; engine conformance scenarios

Default mode dropped a live message.new below the conversation's seq anchor
without an ack, even when this session had already dispatched that message
id. The delivery-ack protocol acknowledges every dedup hit (the earlier
dispatch is the proof), and the TypeScript SDK does. It is now acked again,
still without re-dispatch. A below-anchor frame no handler has seen is
unchanged: dropped unacked, the server re-offers it.

Adds conformance/v1/engine-scenarios.json, run in both modes by a new
harness (identical bytes ship in the TypeScript SDK): drain gap + close
during an earlier handler or inside the gap window, a raising handler during
a drain, and an already-handled repeat below the live anchor as a frame and
as a drain row. Also pins the drain-cursor rule for a socket drop during an
earlier handler with a regression test.

* test(conformance): drain gap with a socket drop during a later row's handler

Shared scenario (identical bytes in agentchat-typescript): page [g1, g3, h1],
h1's handler held, the socket drops, h1 released; the default-mode cursor
stops at g1. This drain awaits each handler while routing, so the drop lands
during a LATER row's handler and clears the parked g3 before the cursor is
computed. Copying a cursor that stops only while a row is still buffered
into _drain_row_ackable now fails this scenario and the reworked unit test
(the earlier-handler shape did not catch it).

CHANGELOG: say where the SDKs still differ on a below-anchor frame no
handler has seen.

* fix(realtime): never hand a message to the handlers twice while its first copy is running

After a reconnect the server pushes its stored backlog on the socket and the
REST drain brings the same rows. The dedup cache is filled only once a
handler succeeds, so with a slow handler the second copy was dispatched
again (e.g. an in-order copy whose cursor advances only after the first
copy's handler returned). _deliver_message_new now keeps a future per
message id in flight; a copy that finds one is not dispatched and takes the
outcome (shielded; a cancelled first copy counts as not handled). A copy
waits only where it waited on its own handler call before; a live repeat
below the anchor, which used to be dropped, never holds up the receive loop
and is acked once the first copy finished. Off with dedup_cache_size=0.

Also closes two races between concurrent copies: the per-conversation
cursor never moves backwards (_advance_past), and _cleanup_if_idle removes
only the state it was handed, never a newer connection's.

Shared scenarios repeat-while-handler-runs-in-order / -below-anchor
(identical bytes in agentchat-typescript).

* fix(realtime): a frame that parks while a gap is resolved is handed over, never cleared unseen

_resolve_gap and _flush_parked_frames snapshotted the buffer, awaited each
handler, then cleared it. The receive loop keeps running during those
awaits, so a frame that parked meanwhile was cleared without dispatch:
under ack_on_receipt (acked at receipt) it was never handled; in the
default mode it waited unacked for the next reconnect. A concurrent
in-order arrival could also pop entries the snapshot still expected
(KeyError in the gap task) or hand them over twice.

Both now pop each frame before handing it over, lowest seq first, and keep
going until nothing is parked (the flush across every conversation, since
disconnect() flushes before it closes the socket). The cursor advances past
each frame after its handler, never backwards.

Shared scenario frame-arrives-while-a-lapsed-gap-is-handled, with a new op
gap_window_lapses (identical bytes in agentchat-typescript).

* fix(realtime): progress beats dedup: nothing waits on another copy, gaps resolve on a snapshot

Review of #11. An in-flight repeat no longer waits for the first copy
anywhere: _adopt_in_flight records the outcome and returns. A live repeat
is acked when the first copy finished cleanly, never if it failed; a drain
row's cursor reads the outcome when computed and stops if the first copy is
not done (the next drain covers it as a dedup hit), so a slow or hung live
handler no longer holds up later drain rows, and a handler that awaits
disconnect() while a repeat of its own message arrives no longer
deadlocks. Adoption is per connection generation: a re-offer on a later
connection is dispatched afresh even if the first copy still runs.

_resolve_gap and the shutdown flush work on the frames parked when they
start; each frame leaves the buffer and the cursor moves past it before its
handler is awaited, so in-order arrivals go to the receive loop, a frame
behind a new hole gets its own gap window and report, and nothing is
cleared unseen (a state being resolved is never dropped as idle).
disconnect() stops taking frames in before it flushes, so it returns
promptly while frames keep arriving; a frame acked at receipt is never
dropped by the flush.

* fix(realtime): a gap fill acks nothing once disconnect() began; an adopted in-order repeat moves the cursor on

A gap fill whose fetch returned while disconnect() was flushing acked its
rows at receipt (ack_on_receipt), and the shutdown then cancelled the gap
task, possibly mid-handover: acked rows never handled. Once disconnect() has
begun, the fill now returns before acking (a gap fill marks nothing on the
server, which keeps the rows).

A live repeat that adopted a drain row still running its handler left the
cursor where it was, so the next live frame parked and opened a spurious gap
(a REST gap fill and a report 2 s later, though nothing was missing). The
in-order branch now moves the cursor on for an adopted copy exactly as for
a dispatch.

Docs: handlers in one conversation can overlap (they start in seq order),
as in the TypeScript SDK.
@sanctrl
sanctrl deleted the fix/engine-cursor-and-gaps branch November 11, 2026 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant