Skip to content

release: 1.5.0 — conversation runs and could-not-process in the kit - #32

Merged
sanctrl merged 8 commits into
mainfrom
feat/could-not-process
Nov 19, 2026
Merged

sanctrl merged 8 commits into
mainfrom
feat/could-not-process

Conversation

@sanctrl

@sanctrl sanctrl commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What changes

The Python half of "could not process" (contract v1, additive). Sometimes an agent's own AI cannot handle one conversation: a safety refusal, a crash on an attachment, a run past its time limit. That conversation no longer stops the others. The machine also reports the fact, so the server can hide the span from this agent and, if the conversation keeps failing, pause it.

Client (sync and async)

  • get_messages(conversation_id, *, limit=50, before_seq=None, after_seq=None, after_message_id=None, wait_ms=None, opts=None):
    • a new cursor after a message id;
    • the scoped wait: wait_ms from 0 to 50000, which needs after_seq or after_message_id; the per-call timeout is stretched to cover it.
    • Calls without the new arguments send the same query as before.
  • client.conversations.report_processing_failure(conversation_id, *, through_message_id, outcome, evidence, opts=None) -> ProcessingFailureAction, for POST /v1/conversations/{id}/processing-failures:
    • it sends an Idempotency-Key;
    • a plain 404 from a server that predates the route returns action="none", never an exception;
    • only the route's own CONVERSATION_NOT_FOUND and MESSAGE_NOT_FOUND raise NotFoundError.
  • Types:
    • Message.could_not_process (MessageCouldNotProcess: at, original_type);
    • Message.client_msg_id is now optional: the server leaves it out of a message hidden from its recipient, so parsing one never raises. Nothing in the SDK reads it from a received message; sends still carry one.
    • ProcessingFailureAction.hidden_count (optional) beside hidden_message_ids, which the server caps at 200;
    • InboxCardPreview.could_not_process;
    • Texts.conversation_run_template and Texts.could_not_process_note;
    • ConversationListItem.last_message_could_not_process (like preview.could_not_process, sent only when true);
    • ProcessingFailureAction, with the aliases ProcessingFailureOutcome, ProcessingFailureEvidence and ProcessingFailureActionName;
    • ErrorCode.COULD_NOT_PROCESS.

Kit (agentchatme.integration.ReceiveLoop)

  • The gate split (gate-call mode). A failed gate chunk of several cards is gated again, one card per call, in the same cycle and before turns start. Muted cards are skipped.
    • Each such call emits a gate_card event; split_calls_ok and split_calls_failed count them.
    • If two cards in a row fail and nothing in the cycle has been decided, the split goes on only when check_runtime passes. Otherwise it stops (split_stopped).
  • What gets reported. A single-card gate call, reply turn or conversation run that failed or timed out, while its card is still owed, is reported only with evidence:
    • other_run_completed: work of the same kind for another conversation completed after the failure started. A gate call counts for a gate call, a turn for a turn, a run for a run.
    • runtime_check: the new optional check_runtime hook passed, and no other conversation of that kind has failed since that kind last completed. The hook must be a real, minimal model call through the same path as the work, never a login or version check. With one active conversation it is the only evidence there is.
    • A run that timed out (timed_out=True, or a gate call past its deadline) is reported only on other_run_completed, never on the runtime check alone. A timeout is ambiguous (a brownout, a hung hook), and the check does not go through those paths.
    • Without evidence, nothing is reported.
    • A turn whose AgentChat calls are known, and none of them worked, is never reported.
    • Right before each report, the card is read on a fresh tray. If it is gone, muted, answered or decided since, there is no report.
    • through_message_id is the anchor of the card the failed work started from.
  • Following the answer.
    • retry or hidden: that conversation runs again after the others, from a fresh tray. A failed turn's conversation is held out of the gate from its end until it starts again.
    • paused or none: stop for that conversation.
    • At most MAX_FAILURE_REPORTS (3) reports per conversation in one cycle.
  • Conversation runs (single-run mode). When conversation_run_template is served and conversation_runs=True (the default):
    • each owed card gets its own run, carrying RunRequest.conversation_id, card and anchor, with the prompt from conversation_run_prompt;
    • pending invites get one invite run;
    • runs go max_concurrent_turns at a time;
    • silent runs are paced (D5).
    • Texts without the template keep one run for the whole tray.
  • New fields. TurnEnd.timed_out and RunEnd.timed_out are optional and default to False.
  • Status and events.
    • New counters: processing_failures, failures_reported, failure_actions, failure_report_errors, failures_waiting, reruns, runtime_checks_ok, runtime_checks_failed, gate_splits, split_calls_ok, split_calls_failed, splits_stopped.
    • New CycleSummary fields: split_calls_ok, split_calls_failed, split_stopped, failures_reported, reruns, runs_started, runs_completed, runs_failed, runs_not_started.
    • New events: processing_failure, runtime_check and gate_card. The run events of conversation runs carry conversation_id.

Conformance: conformance/v1/texts.sample.json and manifest.sha256 are copied byte for byte from the published contract, which now carries conversation_run_template and could_not_process_note. The sha256 was checked against the published manifest, and scripts/check-contract-fixtures.sh passes.

There is no version bump: the number is set at release.

Choices where the contract left room

  • Evidence order. Work is ordered by a counter, never by a clock.
    • Runs and gate calls go one at a time, so "completed after the failing one started" usually means a later run. A failure therefore waits in memory until the others have run: one per conversation, the newest.
    • The runtime check runs only then, once.
    • Evidence must arrive within EVIDENCE_WINDOW_MS (10 min, one run's full budget) of the failure's end. A runtime that was broken and is fixed later cannot turn its old failures into reports.
  • Evidence must be the same kind of work, for another conversation.
    • A gate call (light, no tools) proves nothing about the turn path: its launcher, the tools inside it, or the turn's own model.
    • A conversation's own later end clears its waiting failures; it does not prove them.
  • The runtime check counts only for an isolated, plain failure. If several conversations are failing together, that is the signature of a broken runtime. It does not need the path to have completed first: an agent with one active conversation must be able to report right after a restart. It never counts for a timeout. This is the same rule as the TypeScript kit.
  • Fresh card before each report.
    • A turn that answered and then failed is never reported. On the server's rule, an empty span with the card still on the tray means pause.
    • A tray read that fails reports nothing.
  • Split guard. I chose to ask check_runtime before stopping, instead of stopping outright. The risk this avoids: two flagged cards first in a chunk, with nothing else decided, would otherwise starve the innocent cards behind them. Without a runtime check, the split stops after two failures in a row. That bounds a dead or hung model to 2 extra calls per cycle, not one per card.
  • Unreadable tray in conversation-runs mode. It starts nothing (tray_failed; the server re-wakes), because the runs need the cards.
  • The cap of 3 counts reports in one conversation's chain within a cycle: the failed run plus the runs started again. That matches the server's retry, hidden, paused.
  • A gate call is a failure only when the model was asked. If the prompt could not be built, that is not the model failing.

Verification

  • Tests: ruff, mypy (strict) and pytest ran locally on Python 3.9.25, 3.11.18 and 3.13.18.

    • Result: 1069 passed and 2 skipped (live smoke, Windows proactor) on each version. The contract-fixture check passes.
    • The new kit tests were repeated 8 times each on 3.9 and 3.13 at the head commit with no failures.
    • New: tests/test_could_not_process.py, 83 tests, and one conformance test that reads both new texts from the published sample. They cover:
      • the client: URL, body, timeout, idempotency, an old server's 404;
      • the types, with the server's optional fields present and absent, including a hidden message without client_msg_id and hidden_count;
      • the split and the split guard;
      • evidence (same kind only, isolation, one active conversation right after a start, the window) and no-evidence cases;
      • runtime check pass and fail;
      • retry, then hidden, then the note gated;
      • paused, none and unknown actions stopping;
      • the cap of 3;
      • timeouts (gate call, reply turn, conversation run): never on the runtime check alone, reported on another run's completion;
      • a failed reply turn reported after another turn completes, then run again;
      • turns that answered and then failed, or never looked, never reported;
      • the hold from a failed turn's end through its runtime check and report;
      • stop during a runtime check;
      • conversation runs, including the published template starting them, retry, hidden, paused, pacing, concurrency, a stale waiting failure, and stop.
    • Updated: 9 existing tests in tests/test_receive_loop_chunks.py. They asserted that a failed chunk stays undecided, which the contract changes.
    • The single-run tests keep texts without the template, as an older server serves them, so they still pin one run for the whole tray.
  • Review: an independent review agent read the diff against the contract. Its first pass found six problems, all fixed in commit 2 with a test for each:

    • a runtime check accepted while every conversation failed;
    • work of another kind accepted as evidence;
    • a stale waiting failure reported after the agent answered;
    • the split's cost on a dead model;
    • a rerun racing a cycle;
    • stop() waiting on a runtime check.

    Its second pass confirmed those fixes and raised two smaller points. Commit 3 fixed the earlier hold. It also added a "proven path" rule for the runtime check, which commit 4 removed again: both kits share one rule, isolation only, so that an agent with one active conversation can report. Commit 5 aligns timeouts with the TypeScript kit (other-run evidence only). Commit 6 makes client_msg_id optional and adds hidden_count. Commit 7 copies the published fixtures. Its three reproductions of the first findings now fail on this branch.

    One thing stays open, by design: if the model hangs while check_runtime passes, the split still costs one deadline per card. That needs a check through the same path, which the docstring asks for.

  • Live check: the server half is now live, and the published texts carry the template and the note (checked by the fixture copy). The live-fire with this SDK has not run yet: a stand-in runtime that fails one chosen conversation on purpose. It is pending.

Risk and rollback

  • Cost of the split. When a chunk fails, the split costs extra gate calls: one per owed card in that chunk, and at most 2 when nothing has been decided and there is no passing runtime check. Each call is bounded by gate_call_timeout_ms.
  • Additive surface. Hosts that read CycleSummary fields or event types only see additions; gate_chunk events are unchanged.
  • Rollback. Revert this PR. Servers and clients without the new fields keep the old paths: no template means a single run, and no report route means none.

…it and conversation runs

The Python half of the could-not-process contract (v1, additive).

Client: get_messages takes after_message_id and wait_ms (the scoped wait,
0-50000, with an after cursor). conversations.report_processing_failure
posts a failed run with its evidence and returns the server's action
(retry, hidden, paused, none); a server that predates the route is none.
Types: Message.could_not_process, InboxCardPreview.could_not_process,
Texts.conversation_run_template, ProcessingFailureAction.

Kit: a failed gate chunk is gated again card by card in the same cycle.
A failed single-card gate call, reply turn or conversation run is
reported only with evidence (another conversation's work completed after
it started, or check_runtime passed), and the loop follows the answer:
retry or hidden runs that conversation again after the others, at most
three reports per conversation per cycle. Single-run mode starts one run
per owed card when the texts carry conversation_run_template.
… before each report

From an independent review of the first commit:

- Evidence is work of the same kind: a gate call for a gate call, a turn
  for a turn, a run for a run. A gate call proves nothing about a broken
  turn path.
- A runtime check speaks only for a failure that is isolated (no other
  conversation of that kind failing since that kind last completed):
  several failing together is how a broken runtime looks.
- A turn whose AgentChat calls are known and none worked is not reported.
- A later end of a conversation's work clears its waiting failures, and
  each report first reads the card on a fresh tray: gone, muted, answered
  or decided since means no report (an old failure reported after the
  answer would find an empty span and pause the chat).
- The split stops after two cards in a row fail with nothing decided,
  unless check_runtime passes (split_stopped).
- A conversation whose failed turn is being reported and started again is
  held out of the gate.
- A runtime check ends with stop().
…d a failed turn from its end

From the second review pass:

- check_runtime counts as evidence only once work of that kind has
  completed at least once in this loop: a fresh install whose turns cannot
  run, with one conversation and a light check that passes, reports
  nothing. The docstring says the check must go through the same path as
  the work.
- A failed turn's conversation is held out of the gate from the turn's end,
  through the runtime check, until it is settled; the tray is read again
  after a check that ran.
…nute window

- check_runtime counts as evidence whenever the failing conversation is
  the only one of its kind failing since that kind last completed; the
  condition that work of that kind first completed in this loop is gone,
  so an agent with one active conversation can report after a restart.
  The docstrings say the check must be a real, minimal model call through
  the same path as the work, never a login or version check.
- EVIDENCE_WINDOW_MS is 10 minutes: one run may take its whole budget.
- Types: Texts.could_not_process_note and
  ConversationListItem.last_message_could_not_process (sent only when
  true).
…un's completion

A timeout is ambiguous (a brownout, a hung hook), and a runtime check does
not go through those paths: a timed_out failure (TurnEnd/RunEnd.timed_out,
or a gate call past its deadline) now takes only other_run_completed as
evidence. Plain failures keep both kinds. Same rule as the TypeScript kit.
…idden_count

The server leaves the sender-written client_msg_id out of a message hidden
from its recipient: Message.client_msg_id is now optional, so parsing a
hidden message never raises (nothing in the SDK reads it from a received
message; sends still carry one). ProcessingFailureAction gains the
optional hidden_count beside hidden_message_ids, which the server caps at
200; the processing_failure event carries it.
…uld_not_process_note)

The published texts.sample.json (and manifest.sha256) gained the two
could-not-process texts; copied byte for byte (sha256 checked against the
published manifest). A conformance test reads both from the sample, and a
kit test shows the published template starts conversation runs. The
single-run tests keep texts without the template (as an older server
serves them), so they still pin one run for the whole tray.
@sanctrl sanctrl changed the title Could not process: the scoped wait, the failure report, the chunk split and conversation runs release: 1.5.0 — conversation runs and could-not-process in the kit Nov 19, 2026
@sanctrl
sanctrl enabled auto-merge (squash) November 19, 2026 23:27
@sanctrl
sanctrl merged commit dbfab0b into main Nov 19, 2026
12 of 13 checks passed
@sanctrl
sanctrl deleted the feat/could-not-process branch November 19, 2026 23:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant