fix(bridge): keep threads alive after ZCode-side failures

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## Unreleased
+
+- Classify the observed transient Z.ai account failures as typed internal
+  provider errors so bb's Provider retry plugin can retry the failed turn up to
+  four times with backoff. Retry remains owned by bb; other authentication and
+  configuration failures stay terminal.
+- Self-heal threads whose ZCode process exits unexpectedly: the bridge now asks
+  bb (`provider/recovery` with `restartRecommended`) to restart it and resume
+  the thread from its native checkpoint, and the residual `No ZCode session`
+  turn error carries the same recovery hint. Previously every later submit on
+  such a thread failed permanently until the thread was manually stopped.
+
 ## 0.14.1 — 2026-10-28
 
 - Drop the `[Experimental]` label from the plugin's store name and overview

docs/configuration.md

@@ -206,6 +206,26 @@ stale-turn recovery when no turn remains active. `/compact` uses the native
 compaction operation. Normal context reporting uses native estimates; billing
 cost is not fabricated.
 
+Recognized transient Z.ai account failures (an inactive API-key status or an
+account-service request/response failure) are reported to bb as retryable
+internal provider errors. With bb's built-in Provider retry plugin enabled, a
+failed turn is retried up to four times with backoff; bb decides whether to
+continue accepted input or resend unaccepted input. Other authentication and
+configuration failures remain terminal. `bb provider-retry explain` shows the
+last scheduling or skip decision.
+
+When the ZCode process exits unexpectedly mid-thread, the bridge fails the
+running turn, ends the session, and asks bb to restart the bridge and resume
+the thread from its native checkpoint (`provider/recovery`,
+`restartRecommended`). bb defers that restart while sibling threads on the same
+process are mid-turn, and the restart plus resume take a moment once begun — a
+submit that races it fails with `No ZCode session; start or resume the thread
+first.` and carries the same recovery hint. A repeated error means the
+restart has not completed yet (the host-daemon log shows `Restarting the
+"zcode" bridge` once it starts): wait for the recovery to finish, then
+resubmit. Threads that never started a turn have no checkpoint to restore and
+start fresh.
+
 ## MCP and native runtime preferences
 
 Set `memoryEnabled` or `nativeSearchEnhancementsEnabled` through bb's plugin

knowledge/architecture/session-invariants.md

@@ -18,8 +18,14 @@ sources:
     resource: https://github.com/dpolishuk/bb-zcode/blob/3dc639de8e7926b2460059dcf20156950fc7f2d3/src/tool-server.ts
   - title: Current accepted model selection and turn settlement
     resource: ../../src/bridge.ts
+  - title: Crash recovery notification and no-session recovery hint
+    resource: ../../src/bridge.ts
   - title: Native follow-up, resume, rejection, and model-switch regressions
     resource: ../../tests/core-features.test.ts
+  - title: Crash-recovery regressions (in-flight and settled turns)
+    resource: ../../tests/core-features.test.ts
+  - title: bb daemon crash-recovery contract, bb 0.46 daemon bundle
+    resource: ../contracts/protocol-contracts.md#crash-recovery-contract
   - title: Real CLI fixture for reporting budgets and background work
     resource: ../../scripts/smoke-live.ts
 generated:
@@ -30,6 +36,8 @@ verified:
     at: 2026-10-20T11:15:00Z
   - by: Codex
     at: 2026-10-22T11:22:00Z
+  - by: zcode-agent/GLM-5.3
+    at: 2026-11-01T16:35:00Z
 ---
 
 # Session invariants
@@ -57,6 +65,12 @@ matching tests and documentation updates (see
   invented, native Bash file edits are reported as ignored rather than claimed
   restorable, and the fallback "ZCode models unavailable" entry does not claim
   authentication.
+- Losing the ZCode process is reported, not papered over: the running turn
+  fails, the session ends, and — when the native checkpoint can restore it —
+  the bridge asks bb to restart and resume (`provider/recovery`
+  `restartRecommended`). The bridge never rebuilds a dead session in place or
+  answers as if nothing happened; resume still goes through the strict
+  checkpoint path and fails loudly when the session is not restorable.
 
 ## Context selection lifetime
 
@@ -79,6 +93,14 @@ loopback fixture (native background completion and cold resume). Other pinned
 bounds retain the verification dates stated below; see
 [validation](../../docs/validation.md) for the evidence boundary.
 
+The crash-recovery invariant was re-verified on 2026-11-01: synthetic
+regressions cover both a crash while the turn is still in flight and a crash
+after it settled, and the bb-side behavior (restart the bridge worker, resume
+the thread and idle siblings, defer while a sibling is mid-turn) was verified
+statically against the bb 0.46 daemon bundle — see
+[Protocol contracts](../contracts/protocol-contracts.md#crash-recovery-contract).
+Live bb-smoke on a disposable instance remains the unexecuted step.
+
 ## Credential containment
 
 - Short-lived Coding Plan project tokens exist only in memory; desktop

knowledge/contracts/protocol-contracts.md

@@ -132,6 +132,27 @@ snapshots.
 
 - These are agent approval policies, not an OS sandbox; the CLI runs with the
   host account's permissions.
+- **Transient failure classification**: the two observed temporary Z.ai account
+  failures (inactive API-key status and account-service request/response
+  failure) produce a `provider.error` delta with `errorInfo.category:
+"internal"` before `turn.boundary(status: "failed")`. This lets bb's Provider
+  retry plugin apply its own bounded durable retry policy and accepted-input
+  continuation semantics. The bridge never replays the prompt itself and leaves
+  other authentication/configuration failures unclassified.
+- **Crash recovery contract**: bb keeps the stored provider-thread claim after
+  the `session.ended` delta and only clears it through `thread/stop` semantics,
+  so a bridge that silently drops a session strands the thread (every later
+  `turn/start` fails with `No ZCode session`). The protocol escape hatch —
+  verified against the bb 0.46 daemon (`daemon-bundle.mjs`) — is the
+  unsolicited `provider/recovery` notification with
+  `kind: "restartRecommended"` plus the failing thread id: the daemon shuts the
+  bridge worker down, resumes the named thread via `thread/resume`, then
+  resumes idle sibling threads, deferring while a sibling is mid-turn. The
+  bridge sends it from the ZCode child-exit handler when the session is
+  restorable, and attaches the same hint as `data.recovery` on the residual
+  `No ZCode session` `-32602` error so a racing submit also triggers the
+  restart. The daemon auto-resumes on its own only when its own session map
+  was reset (daemon restart); an in-worker child exit is invisible to it.
 
 ## Marketplace catalog contract
 

knowledge/log.md

@@ -1,5 +1,30 @@
 # Knowledge log
 
+## 2026-11-01 (crash self-healing)
+
+- **Fixed** threads stranded by an unexpected ZCode process exit. `session.ended`
+  does not clear bb's stored provider-thread claim, so every later `turn.submit`
+  failed forever with `No ZCode session; start or resume the thread first.`
+  (observed on `thr_8htvmki9xm` and `thr_d6pdiigxzx`; the bb 0.46 daemon keeps
+  its own session map and cannot see an in-worker child exit). The bridge now
+  sends an unsolicited `provider/recovery` `restartRecommended` notification
+  when a restorable session crashes — bb restarts the bridge worker and resumes
+  the thread and idle siblings from native checkpoints — and attaches the same
+  recovery hint to the residual no-session turn error for submits that race the
+  restart. Non-restorable sessions (no turn ever started) still just end.
+  Recorded the daemon-side contract in
+  [protocol contracts](contracts/protocol-contracts.md); three focused
+  regressions cover both paths; 254 tests green.
+
+## 2026-10-29 (transient provider retry)
+
+- **Fixed** ZCode turns ending permanently on the two observed temporary Z.ai
+  account failures. The bridge now emits typed `provider.error` before the
+  failed boundary, allowing bb's Provider retry plugin to schedule its normal
+  four-attempt backoff rather than adding a plugin-side replay loop. Regression
+  coverage includes both transient messages, delta ordering, and a terminal
+  reasoning-configuration error; full `npm run check` passes with 251 tests.
+
 ## 2026-10-27 (v0.13.0, feat/workflow-visualization branch)
 
 - **Added** timeline rendering for dynamic workflows (roadmap stage 9 v1):

knowledge/runbooks/diagnostics.md

@@ -36,18 +36,19 @@ runbook adds the code-level view for contributors.
 
 Error messages are stable search keys; each is produced by exactly one module:
 
-| Message / behavior                                                             | Produced by          | Meaning                                                                                |
-| ------------------------------------------------------------------------------ | -------------------- | -------------------------------------------------------------------------------------- |
-| `ZCode was not found. Install ZCode or set the plugin executable setting.`     | `src/launch.ts`      | Resolution order (setting → `BB_ZCODE_EXECUTABLE` → PATH → macOS bundle) found nothing |
-| `Cannot launch ZCode: …` / `ZCode exited (signal/code)`                        | `src/client.ts`      | Child spawn failure or early exit; check executable and provider configuration         |
-| `ZCode stdin closed.`                                                          | `src/client.ts`      | Child closed stdin while requests were pending                                         |
-| `ZCode Coding Plan: cannot read the desktop configuration…`                    | `src/coding-plan.ts` | Desktop sign-in missing; open ZCode and sign in again                                  |
-| `The ZCode follow-up queue is full.`                                           | `src/bridge.ts`      | More than 32 pending follow-ups (bounded queue, by design)                             |
-| `This ZCode CLI does not expose compatible v4 history controls. Update ZCode.` | `src/lifecycle.ts`   | v4 row query failed schema check; CLI too old or protocol changed                      |
-| `ZCode returned inconsistent history rows.`                                    | `src/lifecycle.ts`   | Rows not strictly descending / outside the requested range                             |
-| `Invalid bb tool catalog: too many or duplicate tools.`                        | `src/tool-server.ts` | >256 tools or duplicate names                                                          |
-| `bb skill roots must be absolute paths (maximum 128).`                         | `src/skills.ts`      | bb sent relative skill paths or too many roots                                         |
-| Tool endpoint 401 / 404 / 405 / 413                                            | `src/tool-server.ts` | Bad bearer token or browser `Origin` / wrong path or inactive / non-POST / body >1 MiB |
+| Message / behavior                                                             | Produced by          | Meaning                                                                                                                                                                                        |
+| ------------------------------------------------------------------------------ | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ZCode was not found. Install ZCode or set the plugin executable setting.`     | `src/launch.ts`      | Resolution order (setting → `BB_ZCODE_EXECUTABLE` → PATH → macOS bundle) found nothing                                                                                                         |
+| `Cannot launch ZCode: …` / `ZCode exited (signal/code)`                        | `src/client.ts`      | Child spawn failure or early exit; check executable and provider configuration                                                                                                                 |
+| `ZCode stdin closed.`                                                          | `src/client.ts`      | Child closed stdin while requests were pending                                                                                                                                                 |
+| `ZCode Coding Plan: cannot read the desktop configuration…`                    | `src/coding-plan.ts` | Desktop sign-in missing; open ZCode and sign in again                                                                                                                                          |
+| `The ZCode follow-up queue is full.`                                           | `src/bridge.ts`      | More than 32 pending follow-ups (bounded queue, by design)                                                                                                                                     |
+| `No ZCode session; start or resume the thread first.`                          | `src/bridge.ts`      | `turn/start`/`turn/steer` arrived after the ZCode process exited or the bridge worker restarted; carries `data.recovery` `restartRecommended` so bb restarts the bridge and resumes the thread |
+| `This ZCode CLI does not expose compatible v4 history controls. Update ZCode.` | `src/lifecycle.ts`   | v4 row query failed schema check; CLI too old or protocol changed                                                                                                                              |
+| `ZCode returned inconsistent history rows.`                                    | `src/lifecycle.ts`   | Rows not strictly descending / outside the requested range                                                                                                                                     |
+| `Invalid bb tool catalog: too many or duplicate tools.`                        | `src/tool-server.ts` | >256 tools or duplicate names                                                                                                                                                                  |
+| `bb skill roots must be absolute paths (maximum 128).`                         | `src/skills.ts`      | bb sent relative skill paths or too many roots                                                                                                                                                 |
+| Tool endpoint 401 / 404 / 405 / 413                                            | `src/tool-server.ts` | Bad bearer token or browser `Origin` / wrong path or inactive / non-POST / body >1 MiB                                                                                                         |
 
 Bridge-level errors are thrown as `ProtocolError` (default code `-32000`);
 native requests time out after 30 s by default (`src/client.ts`).
@@ -92,6 +93,24 @@ From the provider skill's diagnostics section (kept authoritative there):
   [Change verification](change-verification.md)).
 - Follow-up failures after queue overflow are expected bounded behavior, not a
   bug; report them to the user instead of retrying unbounded.
+- The two observed transient Z.ai account failures (`The API key status is not
+active.` and `account request failed or returned an invalid response.`) emit
+  `provider.error` with `errorInfo.category: "internal"` before the failed turn
+  boundary. bb's Provider retry plugin owns the bounded retry and accepted-input
+  continuation; the bridge does not replay input itself. Inspect the recorded
+  decision with `bb provider-retry explain <thread-id>`. Other native failures,
+  including reasoning/configuration errors, stay unclassified and terminal.
+- When the ZCode process exits unexpectedly, the bridge emits the
+  `session.ended` delta and then an unsolicited `provider/recovery`
+  (`restartRecommended`, `retryable: true`) notification for restorable sessions.
+  bb reacts by restarting the bridge worker and resuming the thread (and idle
+  sibling threads) from native checkpoints — look for `Restarting the "zcode"
+bridge` in the host-daemon log to confirm it ran. This matters because
+  `session.ended` alone does not clear bb's stored provider-thread claim, which
+  is why submits used to fail forever with `No ZCode session` (observed
+  2026-11-01: threads `thr_8htvmki9xm`, `thr_d6pdiigxzx`). The notification is
+  deliberately skipped when no turn ever started (`restorable: false`) — there is
+  no checkpoint to restore.
 
 ## Related knowledge
 

skills/zcode-provider/SKILL.md

@@ -61,6 +61,16 @@ Use this skill for setup and diagnostics of the `zcode` provider in bb.
 - A model catalog or healthy protocol probe is not proof of authenticated model
   connectivity. `npm run test:live` uses a real CLI and synthetic loopback model
   with isolated data. Keep that evidence separate from a real provider run.
+- Recognized transient Z.ai account failures emit typed `provider.error` deltas
+  so bb's Provider retry plugin can retry the turn up to four times with
+  backoff. Check `bb provider-retry explain <thread-id>` for the decision;
+  authentication and configuration errors otherwise remain terminal.
+- When the ZCode process exits unexpectedly, the bridge asks bb to restart it
+  and resume the thread from its native checkpoint. A submit racing that
+  restart fails with `No ZCode session; start or resume the thread first.`; if
+  the error repeats, the restart is still deferred or in progress (bb defers
+  it while sibling threads are mid-turn) — wait for `Restarting the "zcode"
+bridge` in `~/.bb/logs/host-daemon.*.log`, then resubmit.
 - Diagnose a stopped session by its native error. Never silently replace an
   unresumable session, replay a prompt after an uncertain timeout, or copy
   credentials out of ZCode to work around authentication.