Repository navigation
Dialog turns fail when GPU0 NInfer crashes: GPU device faults since the 2026-11-05 farm boot #294
Description
Problem
A dialog-workflow task on the dialog lane queue reached terminal all_attempts_exhausted surfaced as provider_unavailable with an empty route object, the eighth occurrence of this signature. The only attempt visible for this occurrence is the self-hosted GPU route failing provider_unavailable, a connection/backend-unavailable classification in our LLM client. An earlier scoped capture of the same signature showed four retry rounds of exactly three started attempts: both weighted models on the external primary route failing provider_overloaded, then the self-hosted route failing a protocol-class reason, with the designed fallback tail (local shared-pool reasoner, then terminal cloud fallback) never attempted despite migration-set five-hop/180s dialog budgets. The provider-side failures are real as far as our own telemetry shows, but the three-attempt truncation is unexplained: a concrete code-visible silent-reset path exists (the admin workflow-update handler defaults omitted retry budget fields to three hops/sixty seconds and the startup convergence that restores five hops runs only once), and tail ineligibility is an alternative. No live workflow row, full event stream, or admin audit trail is available in scoped evidence, so no code defect is confirmed and no fix is justified yet.
External cause: confirmed. Code defect: possible.
Observations
- Terminal event: a dialog-workflow task on the dialog lane queue reached all_attempts_exhausted surfaced as provider_unavailable with an empty route object; task state failed about 126 seconds after creation.
- This is the eighth occurrence of the same signature, first seen months ago; a prior scoped capture of the same signature recorded twelve attempt_failed rows across four retry rounds roughly fifteen seconds apart: both weighted models on the external primary route failing provider_overloaded, then the self-hosted GPU route failing a protocol-class reason (provider_unavailable on the final round).
- The scoped attempt stream for this occurrence exposes a single attempt_failed row: the self-hosted GPU route failing provider_unavailable; no attempt against the designed fallback tail appears.
- In the LLM client, provider_unavailable classifies transport failures and unreachable-backend codes plus message rules (crates/openplotva-llm/src/retry.rs), i.e. a provider-connection response class, not an internal validation path.
- Host and runtime stayed healthy: running, restart_count 0, oom_killed false, container memory about 0.9GiB of 23.5GiB, ample disk, database pool with zero waiting locks; the incident-time host sample shows a 1-minute load spike of about 44 on 8 CPUs versus about 10 now.
- The deployed runtime revision equals the repository base commit, so the running code is exactly main with no undeployed routing change.
- Dialog lane counters: about 35.6k completed versus 168 failed (about 0.5 percent) and 1 processing, so dialog routing is broadly healthy; the memory lane currently shows about 16.9k completed versus 32 failed.
- Code at base: the routed attempt walker ends a pass when started attempts reach the workflow retry.max_hops (crates/openplotva-app/src/routed_attempts.rs).
- Code at base: migration 179 sets the dialog workflow to five hops / 180000ms wall clock and orders the fallback tail (local reasoner fallback order 0, cloud fallback order 99); a startup convergence re-applies the same shape only while a once-only app-setting marker is absent and a readiness check passes (crates/openplotva-app/src/model_routing.rs, migrations/179_memory_routing_cascades.up.sql).
- Code at base: the admin update_workflow handler defaults an omitted retry_max_hops to 3 and retry_wall_ms to 60000, and the storage UPDATE writes those columns unconditionally, so a partial admin workflow update can silently revert the five-hop design with no convergence to restore it (crates/openplotva-app/src/http/routing.rs, SQL_UPDATE_WORKFLOW in crates/openplotva-storage/src/llm_routing.rs).
- PR Route memory workflows through independent cascades #98's own description states the previous global three-hop workflow budget could stop dialog before its fallbacks - the exact truncation shape observed in the prior capture of this signature.
Hypotheses
- External (supported as far as our telemetry goes): the external primary route was genuinely and continuously overloaded (both weighted models provider_overloaded across four rounds) and the self-hosted route was unreachable or erroring; exhaustion is then the designed terminal outcome per pass and no code defect is required to explain the failing attempts themselves.
- Configuration drift: the live dialog workflow row carries the reduced three-hop / sixty-second budget, for example after an admin workflow update that omitted the budget fields and hit the omit-defaults of three hops / sixty seconds, with the once-only convergence marker preventing re-normalization; this yields exactly three started attempts per round before the tail.
- Tail ineligibility: the fallback tail was not executable at incident time - assignments disabled or absent, capability-filtered (for example a vision requirement), breaker-open, or its single-slot shared capacity pool held - so the tail was skipped without consuming hops.
- Local resource pressure on the self-hosted backend: the incident-time host load spike (about 44 on 8 CPUs) degraded the self-hosted GPU backend into transport-class failures; this would relocate part of the cause from remote provider to local infrastructure.
Supporting
- provider_overloaded hit both weighted primary models across four distinct rounds spread over about one hundred seconds in the prior capture, a sustained provider-side signal rather than a single transient request failure.
- The observed per-round composition - three started attempts (two external primaries plus the self-hosted route) with the tail absent - equals exactly what a three-hop budget produces; a five-hop budget with a live, eligible tail would start tail attempts before exhausting.
- The deployed migration writes the five-hop / 180s dialog budget, so only a later workflow-row edit (for example the omit-default update path) explains a live three-hop value; the once-only convergence marker blocks automatic restoration.
- PR Route memory workflows through independent cascades #98's description independently names the three-hop-stops-before-fallbacks mode, and open issue Dialog turns terminate as provider_protocol_error after our own validators reject the answer: two session-engine exits bypass the model-output split and requeue to exhaustion #277 confirms the same terminal event and empty-route signature recurs on the dialog workflow.
- provider_unavailable is a provider-connection classification in our client, and open issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279 documents the same exhaustion class from provider_unavailable bursts on another workflow.
- The dialog lane completes about 99.5 percent of tasks, so episodic provider bursts plus a budget or eligibility question fit better than broad routing breakage.
Contradicting
- If the live dialog budget were five hops with enabled, selectable, capability-eligible tail assignments, the walker would have started tail attempts before exhausting; it did not, so either the loaded budget or the tail eligibility differs from the documented design, and the scoped evidence cannot distinguish which.
- The three-attempt round shape comes from the earlier capture of this signature; the scoped attempt stream for this occurrence shows only one row, so this occurrence's round structure is unverified.
- Task lifetime of about 126 seconds cannot discriminate a 60s versus 180s per-pass wall budget because job-level retry rounds requeue separately from the per-pass budget.
- Healthy host, container and database samples and the very low dialog failure ratio weaken local-resource and broad-code-defect explanations, though sampled metrics cannot exclude transient causes (the load spike remains unexplained).
- The external-overload signal comes only from our own attempt classifications; no independent provider-side confirmation of the overload or unavailability windows is available in the scoped evidence.
Related changes
- PR Route memory workflows through independent cascades #98 (merged, deployed): set the dialog five-hop / 180s budget, ordered the fallback tail, and added the once-only startup convergence whose marker is central to the drift hypothesis; its description documents the prior three-hop truncation problem.
- PR Replace GPU2 reasoner with Ternary Bonsai #93 (merged): replaced the local reasoner model and placed a dialog fallback-tail model on a single-slot capacity pool shared with memory workflows, making pool contention a live tail-skip candidate.
- Migration 179 (deployed): writes the designed 5-hop / 180000ms dialog budget and fallback ordering at deploy time.
- Issue Dialog turns terminate as provider_protocol_error after our own validators reject the answer: two session-engine exits bypass the model-output split and requeue to exhaustion #277 (open): same terminal all_attempts_exhausted event, workflow and empty route object with a different surfaced reason (provider_protocol_error) and a separately verified requeue-to-exhaustion funnel in the session engine.
- Issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279 (open): same exhaustion event class from provider_unavailable bursts on a single provider in the draw workflow, previously assessed external-possible with no code defect observed.
- PR Keep memory runs and alerts alive when an LLM backend is unreachable #289 (merged): resilience for memory runs and alerts when an LLM backend is unreachable; same backend-unavailability class on memory workflows, not the dialog path.
Missing
- The live dialog workflow row at incident time - retry_max_hops, retry_wall_ms, enabled actually loaded at runtime - plus any admin update_workflow audit trail near the incident window; the scoped evidence tools cannot query SQL.
- The enabled/role/fallback_order/capability/pool/breaker rows of the dialog fallback-tail assignments at incident time, needed to test the tail-ineligibility hypothesis.
- The full routing event stream for this occurrence beyond the single attempt_failed row - the all_attempts_exhausted detail fields and any capacity_pool_wait, capacity_wait_timeout, circuit_open_exhaustion or no_candidates events - to show why each pass stopped before the tail.
- Raw provider responses or HTTP status codes behind the self-hosted route's provider_unavailable, needed to separate transport failures from HTTP error classes and from any decoder-side contribution.
- Whether the failing dialog turns carried media, which would activate the vision capability filter and legitimately narrow the candidate set.
- Whether the self-hosted GPU backend shares the incident host, needed to weigh the load-spike hypothesis.
- Why the self-hosted route's attempt rows record no resolved model attribution despite a previously merged attribution fix.
- Independent provider-side or third-party confirmation of the external overload and unavailability windows.
Acceptance
- Establish the effective loaded hop budget and the fallback-tail assignments' eligibility for the dialog workflow at incident time from runtime state or an authoritative snapshot.
- If omit-defaults drift is confirmed: change the workflow-update path so omitted budget fields preserve the stored values (or are required), and add a regression test proving a partial update cannot silently change retry_max_hops or retry_wall_ms.
- If the truncation is a walker or selection defect: reproduce it with a synthetic five-candidate route fixture and verify all eligible fallback routes are attempted before terminal exhaustion within the configured budgets.
- Obtain independent confirmation (provider status or retained raw responses) of the external unavailability windows to close the external-cause assessment.
Impact
8 captured terminal events.
Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.
Activity
iamwavecut commentedon Nov 13, 2026
Problem
Ninth occurrence of the unchanged signature: a dialog-workflow task on the dialog lane queue reached terminal all_attempts_exhausted surfaced as provider_unavailable with an empty route object, and the scoped attempt stream again shows only the self-hosted GPU route failing provider_unavailable with no attempt against the designed fallback tail. Compared with the retained count-8 analysis there is no new causal fact - same deployed revision (runtime revision equals the repository base commit), same evidence shape, same open questions - so the prior assessment stands: external backend unavailability/overload is supported as far as our own telemetry goes, while the leading code-defect candidate is a partial admin workflow update silently reverting the migration-set five-hop/180s dialog budget to the handler's omit-defaults of three hops/60s, which reproduces the observed three-attempt rounds that stop before the tail. The decisive evidence (live workflow row, fallback-tail assignment eligibility, full routing event stream) remains unavailable to scoped tools, so no fix is justified yet.
External cause: confirmed. Code defect: possible.
Observations
Hypotheses
Supporting
Contradicting
Related changes
Missing
Acceptance
Impact
9 captured terminal events.
Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.
iamwavecut commentedon Nov 13, 2026
Triage 2026-11-13: the cause is GPU0 crashes on the farm, not the hop budget
Checked against production data (read-only), the code at
cb3f843(the running revision) and the farm logs.The configuration-drift hypothesis does not hold. The live
dialogworkflow row isretry_max_hops = 5,retry_wall_ms = 180000, with the GPU2 Qwen fallback (order 0) and the OpenRouter fallback (order 99) enabled. The "two weighted external primaries failingprovider_overloaded" capture predates 2026-09-19, when those vram.cloud models were disabled. The omit-defaults path inupdate_workflowis real but the admin UI always sends the stored values; it is fixed separately in a hardening PR.What happened on 2026-11-13 06:20 UTC (the newest occurrence):
fallback_reserve30 s of the 120 s turn budget).llm-ninfer-gemma) ended itself after its device-stall watchdog saw no device progress for 120 s. Requests in flight failed with HTTP 503 "Remote end closed connection", then "Connection refused".provider_unavailable.Root cause on the farm:
llm-ninfer-gemmarestarted 23 times from 2026-11-07 21:09 to 2026-11-13 16:09 UTC (17 immediate CUDA errors, 6 device stalls ended by the 120 s watchdog);ninfer-3090on GPU2 restarted 13 times. The kernel log shows Xid 13/31/32/41/43/69/70 on all three GPUs since the 2026-11-05 boot (driver 620.70.06), also from non-NInfer processes; the previous boot had none. At least 81 of the 238 terminal dialog failures since 2026-11-07 21:00 UTC fall within the GPU0 restart windows.Actions taken on 2026-11-13:
0c08dfe) to release 2026-11-12.4 (e2e41d4). The release carries fixes on the crash paths: NaN router scores ordered below finite scores (MoE routing), legacy-stream waits incopy_and_wait/fill_and_wait, one request ended on non-finite logits instead of the engine, and thread/device binding of the stall probe.NINFER_DEVICE_STALL_SECONDS=30on both engines, so a stalled GPU ends its requests after 30 s instead of 120 s and the dialog fallbacks get time inside the 120 s turn budget.Keeping this open until the restart rate under the new release is known. The driver rollback remains the owner's decision.
[-]Dialog turns exhaust with provider_unavailable before the designed fallback tail; retry rounds stop at three attempts (recurrence, count 8)[/-][+]Dialog turns fail when GPU0 NInfer crashes: GPU device faults since the 2026-11-05 farm boot[/+]iamwavecut commentedon Nov 13, 2026
Concur with the external-cause conclusion, and thanks — the live-row check settles both open code-side hypotheses from the earlier recurrence analysis. With the dialog workflow actually loaded at 5 hops / 180 s and both fallback assignments enabled, neither the omit-defaults budget drift nor tail ineligibility can produce the observed stops, so I'm withdrawing both. The real-but-latent defect on that path —
update_workflowsubstituting fixed defaults for omitted fields — is correctly tracked as hardening in PR #297, whose description already states it was not the cause here.The 2026-11-13 06:20 timeline also matches the walker as designed: a 120 s device-stall watchdog cannot fit inside a 120 s turn budget, so after the primary consumed the lane slice the GPU2 fallback only got the ~30 s reserve and the cloud tail was structurally unreachable. No dialog-walker change is needed; the 30 s stall setting restores room for both tail hops.
Nothing further is actionable from the code side right now: the open question is the restart rate under the new engine release, which only production time can answer, and the driver rollback is your call. One follow-up I'd suggest once the release has been running: re-check the terminal dialog failures that fall outside both engines' restart windows (only 81 of the 238 since 2026-11-07 were inside the GPU0 windows) to confirm the device faults account for the rest, or surface a second contributor. Want me to run that comparison when enough data has accumulated?