Skip to content

fix(runtime): preserve pending tool state and conversation order - #1198

Merged
yaozheng-fang merged 6 commits into
volcengine:mainfrom
evanlowe:fix/codex-pending-interrupt-once
Nov 3, 2026
Merged

yaozheng-fang merged 6 commits into
volcengine:mainfrom
evanlowe:fix/codex-pending-interrupt-once

Conversation

@evanlowe

@evanlowe evanlowe commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Submitting one result from a batch of external tools could restart the model while another tool was still waiting for an answer, producing a second question or tool call. Codex direct transport could also hang on a second interrupt, and PiAgent lost native call/result relationships when restoring history. Final answers could appear before reasoning after a database reload because they retained the turn-start timestamp.

This PR fixes those generic runtime contracts:

  • Interrupt a pending Codex turn only once, while still propagating queued stream errors.
  • Restore Pi history as native user, assistant, and tool-result messages, preserving IDs, aliases, parallel calls, progress updates, callback changes, and system instructions. Private temporary context files are removed by the invocation resource stack.
  • Wait for every outstanding external call belonging to the current Agent and branch before requesting another model response. Progress responses with will_continue=True stay pending; default Runner result submissions are associated with their original invocation. Explicit invocation ownership and response branches take precedence when IDs are reused. New user turns and other branches remain runnable.
  • On a persistent Codex thread, include every consecutive external result received since the paused tool batch. Earlier partial results have not reached Codex yet. Preserve callback edits and stop at user/model/reasoning boundaries so repeated call IDs in later batches stay separate.
  • Persist ADK completion state for finished alternative-runtime agents so a parallel sibling is not executed again when another branch resumes.
  • Timestamp finalized responses when they are emitted, preserving the event ID, actions, branch, and partial flag. Database ordering then agrees with streaming order.

This is not tool-name/argument deduplication: a later user request can legitimately execute the same tool again. There are no public Agent/HTTP signature, runtime dependency, or sandbox-policy changes.

Validation for head e25c60ca (2026-11-03):

  • PASS: full repository pre-commit, including Ruff, formatting, secrets, and dependency policy, after fetching/rebasing origin/main.
  • PASS: current-head Codex, external continuation, model callback, and runtime conformance suite — 565 passed, 39 skipped, 2 xfailed, no failures.
  • PASS: independent review and 37 external-continuation tests. Includes callback-edited result values, callback text appended to the first, middle, last, first+last, or every result, new user turns, reused call IDs, and ordinary user, model/reasoning, tool-call, and empty-content boundaries.
  • PASS: deterministic CLI reproduction with a real Codex binary and loopback model. The previous translator fails all eight partial-batch cases; the fixed translator passes default/resumable Runner, warm/cold persistent threads, and either result order. Partial/progress results do not request the model, complete results resume the same thread, and tools execute once. Together with existing shim/direct single-tool tests: 10 passed, 0 skipped.
  • PASS: eight real CLI partial-batch cases also pass when before_model_callback appends text to every external-result content.
  • PASS: final candidate wheel installed outside the source checkout; all seven affected modules import from site-packages and match wheel bytes. Complete runtime suite: 785 passed, 39 skipped, 4 xfailed. Separate real Codex/Pi CLI tests: 15 passed, 0 skipped.
  • PASS: all six GitHub workflows for this head. Python 3.10: 8175 passed, 53 skipped, 4 xfailed. Python 3.12: 8178 passed, 50 skipped, 4 xfailed. Linux real Codex smoke: 34 passed; CI verified the tests actually ran. Secret scanning, license, pre-commit, dependency policy and Studio/sidecar gates passed.
  • PASS: v9 cloud Harness with this wheel and frozen BFF request/result handling — fixed Codex suite 13/13; both formerly failing same-batch Codex cases now retain both real results and complete without replay. ADK same-batch cases also pass. Total 17 passed, 0 failed, 2 uncovered; all nine probe sessions were deleted and GET returned 404.
  • UNCOVERED: Pi emitted sequential calls rather than the required same-batch calls in this cloud run, so its two batch cases are not counted as passes and were not retried to replace the result. Final-wheel deterministic real Pi coverage is reported separately above.
  • PASS: v9 runtime-switch regression — two concurrent sessions with 12 consecutive turns each, plus planning on ADK/Pi/Codex, for 27/27 checks. Each runtime accounts for nine checks. All five probe sessions were deleted and subsequent GET returned 404.
  • PASS: authenticated Studio browser on v9. Codex and Pi both completed same-batch time-plus-question with the original tool result; both accepted two same-batch questions answered in reverse order, including refresh while one original question remained pending. Switching Codex to ADK retained the conversation marker and city/time preferences, then executed the time tool. This is separate browser coverage for the Pi batches that the independent cloud probe did not trigger. Final browser refresh and cleanup remain in progress.

Before the final Codex translation fix, head 8942e869 passed the full local repository suite (8152 passed, 52 skipped, 4 xfailed) and isolated-wheel runtime suite (767 passed, 39 skipped, 4 xfailed), but the subsequent live mixed-wait probe found the missing result. Those counts apply only to that earlier revision; the added CLI cases cover the previously missed path.

An earlier runtime-only regression encountered an existing DeepSeek cleanup-test failure: its process-wide atexit.register mock captured Python's first weakref.finalize registration. The pre-change wheel reproduces it in the same dependency environment; no DeepSeek implementation or tests were changed. The final full local and isolated-wheel runtime suites passed, including that test; the earlier failure and baseline evidence remain recorded.

The candidate wheel is veadk_python-1.1.24.dev6+ge25c60ca5-py3-none-any.whl, SHA256 fdefec3d4df7b36be002c7ea384db565a8ae7914755b9d5a7ad885c7a3bb66f6. Its development version comes from existing SCM versioning; no version metadata was edited and no release was published. Consumers pinned to 1.1.23 need a subsequent SDK release.

Existing conformance xfails remain, including cancellation of an actively running Pi SDK bridged tool. BFF coroutine cancellation or Harness model cancellation does not establish coverage of that separate limitation. This follows the pending-tool support in #1197.

@evanlowe evanlowe changed the title fix(runtime): interrupt pending Codex turns only once fix(runtime): preserve tool continuation across Codex and PiAgent Nov 3, 2026
@evanlowe evanlowe changed the title fix(runtime): preserve tool continuation across Codex and PiAgent fix(runtime): preserve pending tool state and conversation order Nov 3, 2026

@yaozheng-fang yaozheng-fang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the runtime continuation changes and regression evidence for e25c60c. All CI workflows passed; the installed wheel and cloud runtime passed the previously failing Codex split-result cases, multi-session runtime switching, and authenticated Studio tool pause/resume with reverse answers and refresh. Existing skipped/xfail cases and the independent Pi probe cases that did not produce a simultaneous batch remain documented.

@yaozheng-fang
yaozheng-fang merged commit 8b1b72d into volcengine:main Nov 3, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants