Skip to content

Retry transient model failures without ending the session - #379

Merged
Haz3-jolt merged 3 commits into
Observal:mainfrom
ShaanNarendran:fix/transient-model-retries-37
Sep 5, 2026
Merged

Haz3-jolt merged 3 commits into
Observal:mainfrom
ShaanNarendran:fix/transient-model-retries-37

Conversation

@ShaanNarendran

@ShaanNarendran ShaanNarendran commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Purpose

Transient provider and stream failures currently end a model turn immediately. This change adds bounded retry handling while preserving one daemon-owned operation and keeping the session usable after exhaustion.

Fixes

Fixes #37
Related to #38

Approach

  • Classify OpenAI Responses HTTP, fetch, provider, and stream failures with provider-neutral categories and request phases.
  • Keep provider-specific classification in packages/ai, but let the kernel session own retry policy, backoff, cancellation, operation ownership, and canonical retry events, following Pi’s session-level architecture.
  • Use three total attempts, abortable exponential backoff, bounded jitter, and Retry-After guidance capped at 60 seconds.
  • Fail closed when dispatch state is unknown, a response stream is interrupted, or text, thinking, or tool output has been exposed.
  • Do not persist raw HTTP error bodies in canonical model errors.
  • Record model.retry_scheduled canonically and render sanitized retry status in the TUI.
  • Advance the exact-match wire protocol to version 9 because older clients cannot validate the new canonical event.
  • Verify through the daemon and SDK that an unrecoverable model error releases operation ownership, reaches every attached client, and permits another prompt in the same session.

No production dependency is added. Provider-specific classification remains in packages/ai; the kernel owns retry behavior over the existing ModelPort contract.

This addresses the session-usability part of #38. An explicit command to replay a failed turn is not included, so this PR no longer claims to close #38.

How was this tested?

Passed:

  • pnpm --filter @axl/ai test — 81 passed
  • pnpm --filter @axl/kernel test — 82 passed
  • pnpm --filter @axl/sdk test — 27 passed
  • pnpm --filter @axl/daemon test — 56 passed
  • pnpm --filter @axl/tui test — 191 passed
  • pnpm check — final run passed: 558 passed, 8 environment-dependent integration tests skipped
  • pnpm audit --audit-level high — no known vulnerabilities

One earlier pnpm check run hit the pre-existing intermittent descendant-process cancellation assertion. The focused test passed immediately afterward, and the complete pnpm check rerun passed.

reuse lint could not be run because the reuse executable is not installed in this environment. No new files or dependencies are introduced. A live provider smoke test was not run because no external provider call was authorized for this review.

Learning

Pi 0.85.0 keeps retry ownership at the agent-session layer, outside provider adapters. Axl follows that ownership model independently while retaining typed provider classification, canonical retry evidence, and the issue’s fail-closed redispatch rules.

Checklist

  • I reviewed the complete diff.
  • I added or updated the smallest relevant test for behavior changes.
  • I ran the relevant formatting, lint, type-check, test, boundary, and license checks.
  • Every new file has SPDX metadata, directly or through REUSE.toml.
  • Every commit has a matching DCO Signed-off-by trailer.
  • UI changes include screenshots attached to the pull request, not committed to the repository.

AI assistance

  • Generative AI materially assisted this change. Tool and model/version: pi (gpt-5.6-sol for the original change; the follow-up model was not exposed by the harness).
  • I manually reviewed, understood, and tested the generated work.

ShaanNarendran and others added 2 commits September 5, 2026 23:17
Signed-off-by: Shaan Narendran <shaannaren06@gmail.com>
Signed-off-by: Hari Srinivasan <harisrini21@gmail.com>
@Haz3-jolt
Haz3-jolt force-pushed the fix/transient-model-retries-37 branch from 4cf439f to 7e50096 Compare September 5, 2026 17:58
Signed-off-by: Hari Srinivasan <harisrini21@gmail.com>
@Haz3-jolt
Haz3-jolt merged commit e98f686 into Observal:main Sep 5, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Keep sessions usable after unrecoverable turn errors Add bounded retries for transient model-turn failures

2 participants