[configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed

Problem

The [configured resource]-draw workflow recorded all_attempts_exhausted with reason provider_unavailable again. The incident signature is unchanged since its first occurrence about two hours after the [configured resource]-draw re-route merged; the count rose from 11 to 13 over about 53.6 hours, so two new exhaustions occurred since the prior scoped analysis, under the same deployed revision. This occurrence failed both of the workflow's candidates at the same second on the same single draw provider with the identical reason. Our client classifies HTTP 429 or >=500 as provider_unavailable, so this is a provider-response class, not an internal error path. A re-route change merged minutes before this analysis replaces the disabled previous primary model's assignments with a new model family on the same provider, but it is not deployed yet and does not remove the single-provider dependency. No code defect is observed; the leading hypothesis remains provider-side 429/5xx bursts, with load concentration from the re-route and retry-cadence amplification as secondary hypotheses requiring provider-side and production-row evidence the scoped tools cannot reach.

External cause: possible. Code defect: not_observed.

Observations

Incident snapshot: event_type all_attempts_exhausted, reason provider_unavailable, workflow image_generation_boogu_turbo ([configured resource] draws); count 13, first_seen 2026-10-21T15:52:21Z (about 2 h after the [configured resource]-draw re-route merged), last_seen 2026-10-23T21:30:37Z; the count was 11 at the prior scoped analysis, so two new occurrences ran since, about 12 h apart.
Route-attempt evidence for this occurrence: two attempt_failed events at the same timestamp, one per candidate of the workflow (the previous primary image model and the newer turbo model inserted as fallback by the routing migration), both provider_unavailable on the same single draw provider; references attempt:1 and attempt:2.
provider_unavailable for this provider class is a provider-response classification: HTTP 429 or >=500 becomes provider_unavailable, other statuses become request_failed (crates/openplotva-app/src/image_jobs.rs, boogu_http_status_error, verified at repository HEAD).
Both candidates are endpoint models of the single draw provider; no draw workflow has a cross-provider fallback (migrations 210 and 212 both keep every draw slot on that provider), so exhaustion under a provider-wide 429/5xx burst is configured behavior.
Deployed revision is the merge before the repository base: the running build contains the [configured resource]-draw re-route (primary switched to the turbo model, previous model inserted as fallback with breaker threshold 5 and 30 s cooldown, migration 210) but not the newer re-route (migration 212).
The newer re-route (merged 2026-10-23T21:51:06Z, merge commit equals the repository base, awaiting deploy) re-points every assignment of the disabled-by-it previous primary model to the new model family on the same provider, including this workflow's fallback row, and adds a third parallel VIP slot on the same provider.
Runtime healthy at evidence time: running, restart_count 0, oom_killed false, moderate CPU, ample memory and disk; database available with 0 waiting locks; queues progressing (image-regular 4 failed vs 2154 completed in the windowed counters, image-vip 0 failed vs 114 completed).
Both the [configured resource] lane and the VIP lane use the same provider, service and generator code path; the windowed failure counters concentrate the failures in the highest-volume lane.
Existing unit tests cover provider_unavailable fallback events and all-slots-failed behavior for this queue in crates/openplotva-app/src/image_jobs.rs, and the observed event sequence matches those tested paths.
A repository-level guard treats a single-attempt exhaustion with a retryable reason as non-actionable, but this occurrence recorded 2 failed attempts, so it surfaced as an incident as designed (crates/openplotva-app/src/runtime_routing.rs).

Hypotheses

H1 (leading, external): the single draw backend intermittently returns 429/5xx for all endpoints in short bursts (capacity, overload or worker restarts); both distinct endpoint models fail identically at the same instants and exhaustion follows as designed; recurrence over 53.6 h at count 13 fits intermittent bursts rather than a hard outage.
H2 (needs provider-side evidence, related to our changes): the [configured resource]-draw re-route concentrated the highest-volume image lane onto the turbo endpoint, and parallel VIP slots added on the same provider raised concurrent slot demand enough to provoke 429 responses; deploy-to-incident timing and shared-endpoint topology support checking this but do not prove it.
H3 (design question, weaker): the job-retry cadence against the 30 s fallback breaker cooldown lets jobs re-exhaust quickly during a burst, amplifying the incident count per burst; configured behavior to confirm as intended rather than defective.
H4 (weakly contradicted): a client-side request defect specific to the [configured resource] path provoking 5xx responses; weakened because the VIP lane shares the provider, the service and the generator code and completes with ~0 failures in the window, but not excluded without raw provider responses.

Supporting

Two distinct endpoint models failed with the identical reason on the same provider at the same timestamp, consistent with a provider-level rather than model-level condition.
provider_unavailable is a provider-response classification (429/5xx) in our client code, not an internal error path.
Host, runtime, database and non-draw queues were healthy at evidence time, ruling out app-wide resource failure or crash-looping.
Failures remain low-volume and concentrated in the highest-volume image lane while sibling image queues on the same provider completed, consistent with intermittent bursts hitting the lane with the most traffic.
The incident persisted identically across the redeploy to the current running revision, which contains no draw-routing change beyond the [configured resource]-draw re-route, so a build-specific client fault is unlikely.
The fallback candidate appears in the chain at all, which the routing migration only inserts after switching the primary, so the deployed database rows reflect the re-route rather than the pre-change single-candidate chain.

Contradicting

No raw provider response bodies, status codes or transport errors are available in the scoped evidence, so a client-side misclassification or request defect (H4) cannot be fully excluded from observation alone.
If the provider were hard-down, the sibling VIP draw workflow on the same provider should fail in the same window; its windowed failure counter is 0 versus 114 completed, favoring bursts over a sustained outage, but windowed counters cannot align failures to the exact incident instants.
The attempt evidence lists the previous primary model as attempt:1 and the newer turbo model as attempt:2; with identical timestamps the ordering does not prove which assignment ran first, so the deployed primary/fallback order cannot be confirmed from the attempt list alone.
The [configured resource]-draw re-route about 2 h before first_seen remains a correlated change that no current evidence rules out as a contributing cause (H2).

Related changes

PR Free draws on Krea 2; VIP draws Krea 2 + MOSS-Image #274 (merged 2026-10-21T13:52:47Z, deployed before first_seen): switched this workflow's primary to the newer turbo model and inserted the previous model as fallback (migration 210, breaker 5 / 30 s); both candidates in every failed chain of this incident, including the current one, come from this change.
PR Route FLUX.3 work and a third VIP picture to Fibo 2.0 #278 (merged 2026-10-23T21:51:06Z, awaiting deploy; its merge commit is the repository base): replaces every assignment of the previous primary model with a new model family on the same provider (migration 212), disables the old model, and adds a third parallel VIP slot on the same provider; once deployed it changes this workflow's chain and adds endpoint load, but leaves the single-provider dependency intact.
PR Give VIP draws a third picture from MOSS-Image #266 (merged 2026-10-19): expanded VIP draws to additional parallel picture slots on the same single draw provider, increasing concurrent endpoint pressure that the load-aggravation hypothesis (H2) needs evaluated.
PR Fix image model routing for free generation and editor #92 (merged 2026-07-31): historical fix of image model routing for [configured resource] generation in the same workflow area; precedent that this workflow's routing has regressed before.

Missing

Raw provider responses for the failed attempts: exact HTTP status codes or transport errors, to separate 429 capacity from 5xx failure or connection-level faults and to exclude client-side misclassification.
Draw backend health and logs for each burst window (slot residency, load, restarts) to confirm or refute a provider-side condition; not accessible from the scoped evidence tools.
Deployed workflow_assignments rows for this workflow (primary/fallback roles, weights, breaker settings) to verify production routing matches migration 210's intent, since the attempt ordering in the evidence is ambiguous.
Per-window failure records for sibling workflows on the same provider aligned to the exact burst instants, to confirm provider-wide correlation rather than [configured resource]-lane-specific failures.
Timestamps of all 13 exhaustion events to establish whether they cluster into few bursts or recur steadily across the 53.6 h span.
Post-deploy behavior of the pending re-route (PR Route FLUX.3 work and a third VIP picture to Fibo 2.0 #278): whether the changed chain and added VIP slot raise or lower the incidence, observable only after the controller deploys it.

Acceptance

Deep stage: obtain the classified status codes or transport errors behind the provider_unavailable attempts for a burst window.
Deep stage: align sibling-workflow failures on the same provider with the burst instants; provider-wide correlation confirms H1, [configured resource]-lane-specific failures point back at our request path (H4).
Deep stage: verify the workflow_assignments rows for the [configured resource]-draw workflow match migration 210's intent (turbo primary, previous model fallback with markers) in production, not just in the repository.
Deep stage: review whether the retry cadence versus the 30 s breaker cooldown should space [configured resource]-draw retries during provider bursts, as a deliberate resilience decision rather than a defect fix.
After the pending re-route deploys, re-check whether the incidence persists under the changed chain before proposing routing changes of our own.
Any code fix requires a reproduced, verifiable behavioral defect in classification, fallback selection or breaker handling; none is observed so far.

Impact

13 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

Problem

The [configured resource]-draw image workflow recorded all_attempts_exhausted with reason provider_unavailable. The signature is unchanged since 2026-10-25 (first seen) and its count rose from 63 to 67 within about seven minutes, i.e. this is an active or just-ended burst of the series already tracked by the closed operational issue #279, not a new failure mode. Scoped attempt evidence for this occurrence shows the primary turbo candidate failing provider_unavailable on the single draw backend at 07:50:25 UTC and the fallback dev candidate failing identically about 30 s later, matching the fingerprint of the earlier burst window (07:40:55-07:44:05 UTC) in the prior scoped analysis. provider_unavailable is the client's classification of provider HTTP 429 or >=500 responses, verified in the deployed tree at crates/openplotva-app/src/image_jobs.rs with unit tests covering the 503 fallback_reason and all-candidates-failed events for the image queues. The deployed runtime revision and build revision equal the repository base, the repository has no in-progress patch, and the two merges since the 2026-11-13 triage (dialog answer classification, partial workflow admin update) do not touch the draw request path. Runtime, database and queues are healthy, and user impact stays tiny (5 failed vs about 2157 completed [configured resource]-lane jobs). No code defect is observed; the leading hypothesis remains a backend-side condition on the single draw backend (restart or capacity 429/5xx under burst), with the specific trigger for this window unverified.

External cause: possible. Code defect: not_observed.

Observations

Incident snapshot: event all_attempts_exhausted, reason provider_unavailable, on the [configured resource]-draw image workflow; signature count 67 since first seen 2026-10-25, latest occurrence 2026-11-14 07:50:55 UTC, status diagnosing.
Scoped attempt evidence for this occurrence: the primary turbo candidate failed with provider_unavailable on the single draw backend at 07:50:25 UTC and the fallback dev candidate failed with the identical reason at 07:50:55 UTC - one job-level cycle exhausting the configured two-candidate chain, same fingerprint as the prior burst window (07:40:55-07:44:05 UTC).
provider_unavailable on this path is the client's classification of provider HTTP 429 or >=500 responses, verified in the deployed tree at crates/openplotva-app/src/image_jobs.rs (status 429 or >=500 maps to provider_unavailable), with existing unit tests covering the 503 fallback_reason and all-candidates-failed events for the image queues.
The running container's revision and build revision equal the repository base 90cd0a0 (merged 2026-11-13 18:22 UTC); the repository is clean with no in-progress patch.
The draw request path (crates/openplotva-app/src/image_jobs.rs) is unchanged since the provider-routing refactor already deployed before the 2026-11-13 triage; the two merges after it (dialog answer classification in dialog/validator files, partial workflow admin update in admin routing/storage files) do not touch it, confirmed via commit file lists.
Runtime healthy at evidence time: running, restart_count 0, not OOM-killed, about 21-28% CPU, 728.8MiB of 23.47GiB memory, ample disk; database connections and locks healthy with no waiters; queues available.
User impact remains tiny: the [configured resource] image lane shows 5 failed versus about 2157-2160 completed jobs (8 cancelled), consistent with the tracking issue's closure note that job-level retries recover most jobs.
The signature count rose from 63 (latest 07:44:05 UTC in the prior scoped analysis) to 67 (latest 07:50:55 UTC), i.e. four new terminal events in about seven minutes - far above the 20-day average of roughly three per day and consistent with one continuing burst window.
The re-route migration that created the chain's current candidate pair (turbo primary, dev fallback, both on the single draw backend) is present in the deployed tree, so the changed chain is live and this burst occurred under it.

Hypotheses

H1 (external, leading): the single draw backend returned 429/5xx responses for both candidates of the chain during an active burst window, matching the previously triaged burst series that the closed tracking issue's operational triage matched to recreations/restarts of the draw service container (a deploy and a farm outage); job-level retries recover most jobs and the lane counters stay near-flat.
H2 (external, secondary): a 429-class response from backend capacity or rate limiting under concurrent load, for which the parallel priority-lane slots added on the same backend are a load factor.
H3 (our code, unsupported): the draw request path itself produces the failure class or mis-selects the fallback - not supported: the draw path is unchanged since the triage, the classification is unit-tested, and the chain exhausted exactly as configured.

Supporting

Both candidates of the chain failed with the identical provider_unavailable reason on the same single draw backend within about 30 s of each other, and the same pair pattern repeated across the earlier burst window - near-simultaneous failure of independent candidates points to a backend-side condition.
provider_unavailable is a provider-response classification (HTTP 429 or >=500) in the deployed client code, not an internal error path.
The operational triage recorded on the closed tracking issue matched earlier bursts of this exact signature to specific recreations of the draw service container and to a farm outage, establishing a backend-side precedent for this signature.
Our own runtime and database are healthy at evidence time (no restarts, no OOM, no lock waiters), reducing the likelihood of a local infrastructure trigger.
The incident recurred identically under the deployed chain re-route and across the two post-triage merges, neither of which touches the draw request path.

Contradicting

No code-level trigger for H3: the recent merges do not touch the image job path, and existing tests cover the classification and exhaustion events observed.
No raw provider response bodies, status codes or transport errors are available in the scoped evidence, so a client-side request defect provoking 5xx responses is not formally excluded, though nothing in the evidence supports it.
Nothing in the scoped evidence contradicts H1, but the draw-backend health window for this specific burst is not accessible, so the restart link remains unconfirmed for this occurrence.

Related changes

PR Route FLUX.3 work and a third VIP picture to Fibo 2.0 #278 (merged, migration 212): re-route that created the chain's current candidate pair on the single draw backend and added a third parallel priority-lane slot; deployed in the base revision and live during this burst.
PR Free draws on Krea 2; VIP draws Krea 2 + MOSS-Image #274 (merged, migration 210): set the [configured resource]-draw chain to the current turbo-primary plus dev-fallback pattern whose exhaustion produces this signature.
PR Give VIP draws a third picture from MOSS-Image #266 (merged): added parallel priority-lane slots on the same single draw backend, a load factor to weigh when assessing 429-class capacity responses.
The two merges after the 2026-11-13 triage (dialog answer classification, partial workflow admin update) are deployed in the base revision; their commit file lists show no draw-path changes, so they are not plausibly causal.

Missing

The classified provider status codes or transport errors behind each provider_unavailable attempt in this burst window (429 versus >=500 distinction), not exposed by the scoped attempt rows.
Draw-backend health for the 07:40-07:51 UTC burst window (container recreation events, restarts, load, slot residency) to confirm or refute the restart-correlated burst mechanism for this occurrence; not accessible from the scoped evidence tools.
Timestamps of all 67 terminal events to establish whether they cluster into few bursts or recur steadily across the 20-day span.
Per-window failure records for sibling workflows (priority draw lane) on the same backend aligned to the burst instants, to confirm backend-wide correlation rather than [configured resource]-draw-lane-specific failures.
Production workflow assignment rows for the [configured resource]-draw workflow (primary/fallback roles, weights, breaker settings) to verify live routing matches the migration's intent rather than assuming it.

Acceptance

Deep stage: obtain the classified provider status codes or transport errors behind the provider_unavailable attempts for the burst window; a 429/5xx class confirms the provider-response hypothesis.
Deep stage: align draw-backend container recreation/restart events and sibling-workflow failures on the same backend with the burst instants; backend-wide correlation confirms the external hypothesis, lane-specific failures point back at our request path.
Deep stage: verify the production workflow assignment rows for the [configured resource]-draw workflow match the migration's intent (turbo primary, dev fallback) rather than assuming it.
Deep stage: review whether the job retry cadence versus the roughly 30 s fallback cooldown should space [configured resource]-draw retries during provider bursts, as a deliberate resilience decision rather than a defect fix.
Any code fix requires a reproduced, verifiable behavioral defect in classification, fallback selection, or breaker handling; none is observed so far.

Impact

67 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

Triage 2026-11-13: operational, closing

Checked against production data (read-only) and the farm.

The bursts are restarts of our own draw service on the farm, not an external provider. The 2026-11-11 19:44 UTC burst (7 exhaustions) matches the recreation of the draw-api container at 19:44:18 UTC (the Krea 2 SDA LoRA deploy). The 31 exhaustions on 2026-11-05 match the farm outage after that day's driver upgrade and reboot. The 2026-11-03/04 bursts cannot be matched any more: the container was recreated since then.
Both candidates of image_generation_boogu_turbo (now Krea 2 Turbo primary, Fibo 2.0 dev fallback) live on the same draw-api, so a restart exhausts the chain as configured.
User impact is small because the image job retries recover most turns: image-regular had 1, 3 and 1 failed jobs on 2026-11-11, 11-12 and 11-13 against 311, 985 and 663 completed; image-vip had none.
The classification (provider_unavailable for HTTP 429/5xx) and the fallback order behave as tested. No code defect.

Closing as operational. A deploy-aware suppression of these incidents would be a separate improvement.

Problem

The VIP text-to-image draw workflow (public key image_generation_flux) keeps reaching terminal all_attempts_exhausted with reason provider_unavailable: the signature first captured at 08:45:55Z has grown to count 7 by 09:26:06Z on the deployed revision, which equals the repository base containing the draw re-route. Scoped route attempts show the two candidates of the slot (the primary draw model and the re-routed fallback draw model) failing with the identical provider-response class on the single draw provider, ~90-100 s apart, in every captured occurrence. The image client maps HTTP 429 or >=500 to provider_unavailable, so these are provider-response classifications, not internal error paths; exhaustion follows as configured because every draw slot keeps both candidates on the single draw provider with no cross-provider fallback. This is the same signature and causal picture as the prior scoped analysis of this incident family; the count increase alone is not new causal evidence. Raw provider responses, sibling-slot window correlation and job-level occurrence mapping remain missing, so external burst (leading) versus load aggravation by the newly deployed third parallel VIP slot cannot yet be separated; no code defect is observed.

External cause: possible. Code defect: not_observed.

Observations

Incident snapshot: event_type all_attempts_exhausted, reason provider_unavailable, workflow image_generation_flux (VIP text-to-image draw slot), route model = the re-routed fallback draw model, provider = the single draw provider; count 7, first_seen 08:45:55Z, last_seen 09:26:06Z (~40 min).
Scoped route attempts: 7 attempt_failed events between 09:13:31Z and 09:22:56Z, alternating the primary draw model and the re-routed fallback draw model roughly every 90-100 s, all reason provider_unavailable on the same single draw provider; attempt reference numbers are inverted relative to timestamps and name a different internal incident id, so attempt-to-occurrence mapping is ambiguous.
Verified at the repository base (equal to deployed and build revision 90cd0a0): the image client maps HTTP 429 or >=500 to provider_unavailable and other statuses to request_failed (crates/openplotva-app/src/image_jobs.rs:4643-4651), so the observed reason is a provider-response classification, not an internal error path.
Verified in runtime routing: a repository-level guard suppresses single-attempt retryable all_attempts_exhausted as non-actionable, while multi-attempt exhaustion surfaces as an incident (crates/openplotva-app/src/runtime_routing.rs:362-375, covered by unit tests); each captured occurrence recorded 2 failed attempts, so it surfaced as designed.
Verified in migrations at the deployed revision: the observed candidate chain is the primary draw model plus the re-routed fallback draw model (migration 212 swapped the previous fallback model family onto every image-workflow fallback assignment; migration 210 set the [configured resource] lane's mirroring two-candidate shape), both candidates are endpoint models of the single draw provider sharing one draw capacity pool, and migration 212 also added a third parallel VIP draw workflow on that provider; no draw workflow has a cross-provider fallback, so exhaustion under a provider-wide 429/5xx burst is configured behavior.
Live lane counters at re-check: VIP draw lane 107 completed / 2 failed (the failed count rose by 1 since the prior scoped analysis); [configured resource] draw lane 2078 completed / 4 failed / 20 pending - both lanes broadly healthy with low absolute failures relative to completions, fitting intermittent bursts rather than a hard outage or a systematic path break.
Host, runtime and database healthy at evidence time: running, restart_count 0, no OOM kill, load ~3.4-3.8 on 8 CPUs, ample memory and disk, zero waiting locks; deployed revision equals build revision equals repository base, so the running code is exactly main including the re-route.
The signature is unchanged from the prior scoped analysis of this incident family (same first_seen timestamp); the count rose from 3 to 7 and lane failure counters moved only marginally, which is recurrence of the same condition, not a new causal fact.

Hypotheses

H1 (leading, external): the single draw backend returns 429/5xx for its endpoints in bursts (capacity, overload or worker restarts); both distinct endpoint models of the slot fail with the identical class and exhaustion follows as designed. The recurring VIP-lane bursts plus the earlier documented bursts on the [configured resource] lane of the same provider fit an intermittent provider-side condition; still unproven without raw responses.
H2 (needs evidence, related to our changes): the deployed re-route added a third parallel VIP draw workflow on the same provider and shared capacity pool, with slow (100-120 s) generations and a 240 s retry wall, raising concurrent endpoint pressure enough to provoke 429 responses. All observed occurrences are under the deployed re-route, but correlation over ~40 minutes proves nothing.
H3 (weak): a client-side request defect specific to this slot provoking 5xx. Weakened because the [configured resource] lane shares the provider, service and generator code path with only 4 failures against 2078 completions, the classification path is unit-tested, and the prior scoped analyses of the same class on this provider found no code defect; not excluded without raw responses.

Supporting

Both distinct endpoint models of the workflow failed with the identical provider-response class on the same single provider within ~90-100 s of each other in every captured occurrence, consistent with a provider-level rather than model-level condition.
provider_unavailable is a provider-response classification (HTTP 429/>=500) in the image client at the deployed revision, not an internal error path.
Host, runtime, database and queue evidence are healthy, ruling out app-wide resource failure or crash-looping.
The sibling [configured resource]-draw workflow on the same provider and shared capacity pool showed the same both-candidate exhaustion class in prior documented bursts, closed as external-possible with no code defect.
Both draw lanes remain broadly healthy (107 completed / 2 failed VIP; 2078 completed / 4 failed [configured resource]), fitting intermittent bursts rather than a sustained outage or a code-level break in the VIP path.

Contradicting

No raw provider response bodies, status codes or transport errors are in the scoped evidence, so H1 versus a client-side misclassification (H3) and 429 capacity versus 5xx failure cannot be separated from observation alone.
Attempt references are numbered inversely to their timestamps and reference a different internal incident id than the snapshot, so the deployed primary/fallback execution order and the mapping of attempts to occurrences cannot be confirmed from the attempt list alone.
Seven occurrences over ~40 minutes cannot distinguish a pre-existing provider burst pattern from load aggravated by the newly deployed third VIP slot (H2); the rising VIP-lane rate while the [configured resource] lane stays quiet is suggestive but not conclusive.

Related changes

PR Route FLUX.3 work and a third VIP picture to Fibo 2.0 #278 (merged, deployed in this revision): replaced every assignment of the previous disabled fallback model with the new draw model family on the same provider (migration 212), including this workflow's fallback row, and added a third parallel VIP draw workflow on the same provider and shared capacity pool; all observed occurrences are the first observed under that chain.
PR Free draws on Krea 2; VIP draws Krea 2 + MOSS-Image #274 (merged, deployed earlier): set the [configured resource] lane's two-candidate same-provider shape with the breaker settings (migration 210) explicitly mirroring this VIP slot's existing primary-plus-fallback shape, per the migration's own comment; the observed candidates come from that topology after the later re-route swapped the fallback model.
PR Give VIP draws a third picture from MOSS-Image #266 (merged, deployed): earlier expansion of VIP draws to more parallel picture slots on the same single draw provider, part of the concurrent endpoint pressure relevant to H2.
Issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279 (closed): the same exhaustion class from provider_unavailable bursts on the same provider's [configured resource]-draw lane, assessed external-possible with no code defect; its follow-up to re-check incidence after the re-route deploys is exactly the window these VIP-lane occurrences fall in.

Missing

Raw provider responses for the failed attempts across all occurrences: exact HTTP status codes or transport errors, to separate 429 capacity from 5xx failure and to exclude client-side misclassification.
Window-aligned failure records for the sibling draw slots and the [configured resource]-draw lane at each burst window, to confirm provider-wide correlation rather than VIP-slot-specific failure.
Deployed workflow_assignments rows for image_generation_flux (primary/fallback roles, weights, breaker settings) to verify production routing matches migration 212's intent, since the attempt reference ordering is inverted and names a different internal incident id.
Mapping of the 7 captured attempts to distinct queued jobs versus requeues of one job (job-level retry cadence against the 240 s retry wall, the 3-hop/60 s workflow budget and the 30 s breaker cooldown), to establish the true user-facing failure count behind the lane counters.
Draw backend health and capacity-pool telemetry for the burst windows (endpoint concurrency, slot residency, restarts), not accessible from the scoped evidence tools, needed to weigh H2.
Exhaustion-rate history of the [configured resource]-lane signature relative to the re-route deploy time, to establish whether provider bursts persisted across the deploy or the VIP-lane acceleration is new.

Acceptance

Deep stage obtains, or explicitly rules out, raw HTTP status codes or transport errors for the failed attempts, separating 429 capacity from 5xx failure and confirming or excluding the provider-response classification.
Window-aligned failure records across the sibling draw slots and [configured resource] lane for the burst windows either confirm provider-wide correlation (supporting external cause) or show VIP-slot-specific failure (requiring a code-level investigation).
Any proposed fix must state the functional defect it removes (e.g. a cross-provider fallback or retry-budget change is a routing design decision, not a defect repair) with a regression check; absent new evidence, no code change is justified by the current observations.

Impact

7 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

Problem

The second VIP text-to-image workflow (created as the MOSS-Image slot, whose fallback was later re-pointed to the dev draw model) recorded nine terminal all_attempts_exhausted events with reason provider_unavailable between 08:46 and 09:23 UTC. Scoped attempt rows show the slot's primary image model and its dev-model fallback both failing provider_unavailable on the single self-hosted draw provider, about 90 s apart within each job-retry cycle. The image client classifies HTTP 429 or >=500 as provider_unavailable (verified at the deployed revision, unit-tested), so this is a provider-response class, not an internal error path; the two-candidate chain is deliberately single-provider by migration, so exhaustion under a provider-wide burst is configured behavior. The same morning produced the identical exhaustion class on the sibling [configured resource]-draw lane (earlier burst) and on other VIP draw slots of the same provider, matching the burst series the owner triaged as operational on issue #279 (farm-side draw service restarts). Host, container, database and queue counters were healthy and the draw request path is unchanged since the 2026-11-08 routing refactor, so no code defect is observed. The provider-side cause for this specific window remains inferred: raw status codes, backend restart/GPU telemetry and the live assignment rows are not available in the scoped evidence, so the deep stage should confirm the window's backend correlation before any operational or code action.

External cause: possible. Code defect: not_observed.

Observations

Incident snapshot: terminal all_attempts_exhausted with reason provider_unavailable on the second VIP text-to-image workflow; route names the workflow's dev-model fallback on the single draw provider; count 9, first seen 08:46:25 UTC, last seen 09:23:26 UTC, status diagnosing.
Scoped route-attempt rows (newest first, 09:12:31-09:17:06 UTC): four attempt_failed events alternating the slot's two chain models, all provider_unavailable on the same single draw provider; within each job-retry cycle the primary fails first and the fallback about 90 s later, consistent with the slot's 120 s retry wall and a late-failing slow generation.
Verified in the deployed tree (running revision equals build revision equals repository base 90cd0a0): the image client maps HTTP 429 or >=500 to provider_unavailable and other statuses to request_failed (crates/openplotva-app/src/image_jobs.rs), and unit tests cover the provider_unavailable fallback events and all-candidates-failed exhaustion for the image queues.
Verified in migrations at the deployed revision: this workflow was created with an image-model primary plus a same-provider fallback sharing one capacity pool, 3 hops / 120000 ms retry wall, circuit breaker 5 failures / 30000 ms cooldown; the later re-route swapped the fallback family onto the dev draw model observed failing here, and no draw workflow has a cross-provider fallback.
The same morning produced the identical exhaustion signature on other draw workflows of the same single provider: the [configured resource]-draw lane (count 67 by 07:50:55 UTC, earlier burst window) and other VIP draw slots (recurrences by 09:13:31 UTC), i.e. provider-wide correlation across different request shapes.
Host and runtime healthy at evidence time: container running, restart_count 0, not OOM-killed, moderate CPU and memory, ample disk; database connections and locks healthy with zero waiting locks; queues available.
Lane counters: the VIP image lane shows 107 completed versus 2 failed, the regular image lane 2081 completed versus 4 failed - low absolute failures relative to completions, fitting intermittent bursts rather than a hard outage or a systematic path break.
A repository-level guard suppresses only single-attempt retryable exhaustion as non-actionable (crates/openplotva-app/src/runtime_routing.rs, unit-tested); these occurrences recorded 2 failed attempts, so surfacing as an incident is designed behavior.
The draw request path is unchanged since the 2026-11-08 provider-routing refactor; the two merges at base (dialog answer classification, partial workflow admin update) do not touch image job code, confirmed via their commit file lists.
No task payload is linked to this incident in the scoped evidence (task not linked), so the job-level context of the terminal events is unavailable.
The repository is clean at the base revision with no pending patch.

Hypotheses

H1 (leading, provider/infrastructure-side): the single self-hosted draw backend returned 429/5xx responses or refused work in bursts during this window, exhausting every same-provider candidate chain on it; matches the burst series the owner triaged as operational on issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279 (draw service restarts / farm outage) and would explain the simultaneous recurrences on the sibling lanes.
H2 (capacity, related to our changes, unproven): the deployed re-route added a third parallel VIP draw slot with slow (100-120 s) generations on the same provider and shared capacity pool, raising concurrent endpoint pressure enough to provoke 429-class responses; correlation across the morning windows is suggestive but not conclusive.
H3 (farm hardware): the farm-side GPU device faults documented on the open dialog-lane issue since the 2026-11-05 farm boot extend to the draw worker in this window, producing backend-unavailable responses; same-farm instability background, unconfirmed for the draw worker.
H4 (our code, weakly contradicted): a client-side request defect specific to this workflow's path provoking the failures - weakened because sibling workflows on the same provider and code path fail with the identical class in the same window, the classification is unit-tested, the chain exhausted exactly as configured, and prior scoped analyses of this signature found no code defect; not formally excluded without raw responses.

Supporting

Both distinct endpoint models of the slot failed with the identical provider-response class on the same single provider within about 90 s of each other in every captured cycle - near-simultaneous failure of independent candidates points to a backend-side condition.
provider_unavailable is a provider-response classification (HTTP 429 or >=500) in the deployed client code, not an internal error path.
Multiple sibling draw workflows on the same provider - with different request shapes - recorded the same both-candidate exhaustion class across this morning's windows, fitting a provider-wide condition rather than a slot-specific request defect.
The owner's operational triage on issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279 (2026-11-13) checked production data and the farm and matched earlier bursts of this exact signature to recreations of the draw service container and a farm outage, establishing a backend-side precedent.
Host, runtime, database and non-draw queues were healthy at evidence time, ruling out app-wide resource failure or crash-looping as the trigger.
The draw request path is unchanged since the routing refactor deployed before that triage, and the observed exhaustion sequence matches the unit-tested classification and fallback behavior.

Contradicting

No raw provider response bodies, status codes or transport errors are available in the scoped evidence, so 429 capacity versus 5xx backend faults versus refused connections cannot be separated, and a client-side defect provoking 5xx (H4) is not formally excluded.
This window is sustained (nine occurrences over about 37 minutes on this slot, plus sibling lanes) rather than the brief restart-like bursts previously documented, and the restart correlation for this specific window is unconfirmed.
Backend-side health for this window (draw service container recreations, worker restarts, GPU faults, slot residency) is not accessible from the scoped evidence tools, so the provider-side cause remains inferred from our own classifications.
The live workflow_assignments rows for this workflow cannot be queried by the scoped tools; the chain shape is verified from migrations, not from the production row.
Only the newest four attempt rows are visible; the composition of the other occurrences in the window is unverified.

Related changes

PR Route FLUX.3 work and a third VIP picture to Fibo 2.0 #278 (merged 2026-10-23, deployed in the running revision): re-pointed every text-to-image fallback assignment onto the dev draw model observed failing here and added a third parallel VIP slot on the same provider - the changed chain is live and is a load factor for H2.
PR Give VIP draws a third picture from MOSS-Image #266 (merged 2026-10-19, deployed): created this workflow with the same-provider two-candidate shape, 3 hops / 120 s retry wall and shared capacity pool.
PR Free draws on Krea 2; VIP draws Krea 2 + MOSS-Image #274 (merged 2026-10-21, deployed): set the [configured resource]-draw lane's same-provider chain whose burst opened this morning's earlier window on the same provider.
Issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279 (closed 2026-11-13 as operational): tracking issue for this exact exhaustion signature series on the same provider; its owner triage matched prior bursts to draw service restarts and a farm outage, and today's recurrences were appended there.
Issue Dialog turns fail when GPU0 NInfer crashes: GPU device faults since the 2026-11-05 farm boot #294 (open): farm-side GPU device faults since the 2026-11-05 farm boot producing provider_unavailable-class exhaustion on the dialog lane - same-farm instability background for H3.

Missing

The classified HTTP status codes or transport errors behind each provider_unavailable attempt in this window (429 versus >=500 versus refused connection), not exposed by the scoped attempt rows.
Draw-backend health for the 08:46-09:23 UTC window: service container recreation events, worker restarts, GPU fault records, endpoint concurrency and pool occupancy, to confirm or refute the restart/GPU correlation for this occurrence.
The deployed workflow_assignments rows for this workflow (roles, weights, breaker settings) to verify the production chain matches the migration intent; the scoped evidence tools cannot query SQL.
The full attempt history beyond the newest-first four-row window, including the primary attempts of the other five-plus occurrences and their job-level retry cadence.
Whether the farm GPU faults documented on the open dialog-lane issue affected the draw worker during this window.

Acceptance

Deep stage: obtain the status or transport detail behind the provider_unavailable attempts; 429 versus 5xx versus refused connections separates capacity pressure from backend faults from network issues and settles H1 versus H2.
Deep stage: align backend restart/GPU-fault events and sibling draw-workflow exhaustion instants with this slot's occurrence times; provider-wide correlation with a backend event confirms the operational classification for this window.
Deep stage: verify the live workflow assignment rows for this workflow match the migration chain's intent.
Any code fix requires a reproduced, verifiable behavioral defect in classification, fallback selection or breaker handling; none is observed so far, and the single-provider chain is configured behavior, so a routing redesign would be an owner decision rather than an incident fix.

Impact

9 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

Problem

The third VIP text-to-image workflow recorded three terminal all_attempts_exhausted events (provider_unavailable) within six minutes. Both of its candidates on the single draw backend failed with the identical provider-response class about 90 s apart. Verified in the deployed code, provider_unavailable on this path means a 5xx-class status or transport/connection failure, not rate limiting, and no draw workflow has a cross-provider fallback, so exhaustion is the configured outcome of a backend-side burst. Two other draw workflows on the same backend hit the same signature earlier the same morning, and the closed tracking issue's owner triage already matched such bursts to draw-service container recreations. Host, runtime, database and the draw request path are healthy and unchanged; the VIP lane completed 107 draws with 0 failures in the windowed counters. No behavioral defect is observed.

External cause: possible. Code defect: not_observed.

Observations

Incident snapshot: event all_attempts_exhausted, reason provider_unavailable, workflow is the third VIP text-to-image slot; count 3, first_seen 2026-11-14T08:50:00Z, last_seen 2026-11-14T08:56:15Z, status diagnosing.
Scoped attempt rows for the captured occurrence: two attempt_failed rows on the single draw backend, the workflow's primary draw model at 08:48:30Z and its same-provider fallback draw model at 08:50:00Z, both provider_unavailable; the reference numbering is newest-first by the evidence query, so execution order is primary then fallback.
Verified in the deployed tree (equal to repository base and build revision): the image client builds an error message for HTTP 429 or >=500 responses (crates/openplotva-app/src/image_jobs.rs, boogu_http_status_error) and the routing reason is derived from that message by the shared rule table (crates/openplotva-llm/src/retry.rs via image_generation_retryable_reason); there the rate-limit phrases including '429' are matched before the unavailability phrases, so an observed provider_unavailable implies a 5xx-class status or a transport/connection failure, not rate limiting.
Verified in migrations at HEAD: this workflow was created with a primary draw model plus a same-provider fallback sharing one capacity pool (3 hops / 240 s retry wall), and no draw workflow has a cross-provider fallback, so both routed attempts hit the same backend and exhaustion under a backend-wide burst is configured behavior.
The same morning window produced the identical signature on other draw workflows of the same backend: the [configured resource]-draw lane at 07:50Z and the first VIP slot at 08:44-08:46Z, plus an earlier occurrence pair on this same workflow at 08:46:25Z and 08:49:30Z, all triaged in comments on the closed tracking issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279.
The 2026-11-13 owner triage on issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279 checked production data and the farm and matched earlier bursts of this exact signature to recreations of the draw service container (a deploy and a farm outage), closing the issue as operational with no code defect.
Host and runtime healthy at capture time: running, restart_count 0, no OOM kill, moderate CPU, ample memory and disk, database connections and locks healthy with zero waiters; deployed revision equals build revision equals repository base (merged 2026-11-13).
The draw request path (crates/openplotva-app/src/image_jobs.rs) is unchanged since the provider-routing refactor deployed before the 2026-11-13 triage; the two merges after it touch dialog and admin workflow-update files only.
Lane counters at capture: the VIP image lane shows 107 completed, 0 failed, 1 processing; the regular image lane 2109 completed, 4 failed, 48 cancelled, 18 pending, so job-level retries recover most turns and failures concentrate in short bursts.
A repository-level guard suppresses single-attempt retryable exhaustion as non-actionable; this occurrence recorded two failed attempts, so it surfaced as an incident as designed (crates/openplotva-app/src/runtime_routing.rs, covered by unit tests).
Existing unit tests cover the provider_unavailable fallback events and the all-candidates-failed exhaustion events for the image queues, and the observed two-attempt sequence matches those tested paths.

Hypotheses

H1 (leading, backend-side): the single self-hosted draw backend returned 5xx-class responses or refused connections during a short burst (service restart/recreation or outage window), failing both candidates of the configured chain about 90 s apart; three workflows on that backend showing the same signature within one morning window and the previously verified restart correlation support this.
H2 (weakened, capacity): concurrent load from three parallel VIP slots, including the slow (100-120 s) third-slot generations, provokes capacity responses on the backend; weakened because rate-limit/429 responses classify as provider_overloaded on this path and both attempts classified provider_unavailable.
H3 (client-side defect specific to the new workflow, weakly contradicted): the new workflow's request path provokes the failures; weakened because two other draw workflows that do not use the new prompt path failed identically in the same window and the VIP lane otherwise completed with 0 failures.

Supporting

Both distinct endpoint models of the chain failed with the identical provider-response class on the same backend within 90 s of each other, pointing to a backend-level rather than model-level condition.
The observed class excludes rate limiting on this code path, which matches the restart/outage burst mechanism previously verified for the sibling draw lanes rather than a capacity response.
Two other draw workflows on the same backend failed with the same signature earlier the same morning, so the condition is provider-wide across lanes, not specific to the new workflow.
Host, container, database and queue counters were healthy at capture, ruling out app-wide resource failure or crash-looping on our side.
The draw request path is unchanged since the refactor deployed before the 2026-11-13 triage, and the merges after it do not touch it.
The classification and exhaustion event paths are unit-tested at HEAD and the observed sequence matches the tested behavior.

Contradicting

No raw provider response bodies, status codes or transport errors are available in the scoped evidence, so a client-side misclassification or request defect cannot be formally excluded from observation alone, though nothing in the evidence supports it.
Backend-side health for this specific burst window (container recreation events, restarts, load) is not accessible from the scoped evidence tools, so the restart correlation established on 2026-11-13 remains unconfirmed for this window.
Windowed lane counters cannot be aligned to the exact burst instants, so sibling-lane health during the window is inferred rather than measured.
The attempt rows cover only the captured occurrence; the composition of the other two exhaustions in the six-minute window is unverified.

Related changes

PR Route FLUX.3 work and a third VIP picture to Fibo 2.0 #278 (merged 2026-10-23, deployed in the running revision): created this third VIP workflow with a primary draw model and same-provider fallback, and re-pointed draw routing; both candidates in the failing chain come from this change, and this burst is the first observed on the workflow under the deployed chain.
PR Give VIP draws a third picture from MOSS-Image #266 (merged 2026-10-19): added parallel VIP draw slots on the same single backend, the load context for the capacity hypothesis H2.
PR Free draws on Krea 2; VIP draws Krea 2 + MOSS-Image #274 (merged 2026-10-21): set the [configured resource]-draw chain whose burst opened this morning's window on the same backend.
PRs Classify validator-rejected dialog answers as model output #296 and Keep stored workflow settings on a partial admin update #297 (merged 2026-11-13, deployed): dialog answer classification and partial workflow admin update; their file lists show no draw-path changes, so they are not plausibly causal.
Issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279 (closed operational): tracking issue for this exact exhaustion signature series; today's three prior triage comments and the 2026-11-13 restart-correlation closure live there.

Missing

Backend-side health for the 08:44-08:56Z burst window (draw service container recreation/restart events, load, slot residency) to confirm or refute the restart correlation for this occurrence; not accessible from the scoped evidence tools.
The classified HTTP status codes or transport errors behind the two provider_unavailable attempts (5xx versus connection-refused distinction), not exposed by the attempt rows.
Attempt rows for the other two exhaustions in the 08:50-08:56Z window, and timestamps of all occurrences of the signature, to establish burst clustering.
Deployed workflow assignment rows for this workflow to confirm production routing matches the migration's intent; the scoped evidence tools cannot query SQL.
Per-window failure records for sibling draw lanes aligned to the exact burst instants, to measure provider-wide correlation rather than infer it from windowed counters.

Acceptance

Deep stage or owner: correlate the 08:44-08:56Z burst window with backend operational state (service recreation/restart events); a match closes the causal chain the same way the 2026-11-13 triage did.
Deep stage: obtain the classified status codes or transport errors behind the provider_unavailable attempts for the window, to separate 5xx failures from connection-level faults.
Any code fix requires a reproduced, verifiable behavioral defect in classification, fallback selection or breaker handling; none is observed, and those paths are unit-tested at HEAD.
Resilience options (deploy-aware suppression of these terminal events, or a cross-provider draw fallback) are deliberate design changes to propose separately, not defects to fix in this incident.

Impact

3 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

Problem

Terminal all_attempts_exhausted (provider_unavailable) on the third VIP text-to-image workflow, count 3 within ~6 minutes, continuing a burst that already hit two sibling draw workflows on the same single draw backend the same morning. Both configured candidates (primary and re-routed fallback) fail with the identical provider-response class 60-90 s apart; the image client derives that class from HTTP 429/>=500 responses, no cross-provider draw fallback exists, and exhaustion follows as configured. Host, runtime and database are healthy; the draw request path is unchanged by recent merges and its classification/exhaustion paths are unit-tested. The closed tracking issue's owner triage already matched this signature family to draw-service container recreations and closed it operational with no code defect. This occurrence adds no new causal fact; the open gap is backend-side telemetry and raw status detail for this window, so external cause remains possible rather than confirmed and no code defect is observed.

External cause: possible. Code defect: not_observed.

Observations

Incident snapshot: terminal all_attempts_exhausted with reason provider_unavailable on the third VIP text-to-image workflow (image lane), count 3, first_seen 2026-11-14T08:52:05Z, last_seen 2026-11-14T08:58:25Z, status diagnosing.
Verified in the deployed tree (repository HEAD equals the running build revision): the image client maps HTTP 429 or >=500 to a provider_unavailable message prefix and other statuses to request_failed, carrying the status code in the message (crates/openplotva-app/src/image_jobs.rs, HTTP-status error builder).
Verified at HEAD: the shared message rule table is applied first-match-wins and lists the overloaded/rate-limit phrases (including 429 and rate limit) before the unavailability phrases (502/503/504, bad gateway, connection refused/reset, send failures), so an observed provider_unavailable implies a 5xx-class status or transport/connection failure rather than rate limiting (crates/openplotva-llm/src/retry.rs).
Verified at HEAD: the admin-actionable guard suppresses only single-attempt retryable exhaustion; a two-attempt exhaustion like this one is reported as designed, and unit tests cover both cases (crates/openplotva-app/src/runtime_routing.rs).
Verified in routing migrations: the chain re-route replaced this slot's previous fallback model family with the new dev model on the same single draw provider, with both candidates sharing one capacity pool; the earlier migration built the two-candidate shape with breaker settings; no draw workflow gained a cross-provider fallback.
Scoped route-attempt evidence: six attempt_failed rows on the single draw backend spanning 08:44:55Z-08:52:05Z; the workflow's two candidates (its primary draw model and the re-routed fallback draw model) both fail with the identical provider-response class each cycle, roughly 60-90 s apart.
Deployed revision equals build revision equals repository base (merged 2026-11-13); repository clean with no in-progress patch, so the running code is exactly main.
Host/runtime/database healthy at evidence time: running, restart_count 0, not OOM-killed, moderate CPU, ample memory/disk, zero waiting locks; VIP image lane windowed counters 107 completed / 0 failed.
This window is a continuation of the same morning burst already triaged on the closed tracking issue: the sibling regular draw lane at 07:50Z, the first VIP slot at 08:44-08:46Z, and this same workflow at 08:46:25Z/08:49:30Z and 08:50:00-08:56:15Z.
Prior scoped analyses of this exact signature family (four comments on the closed tracking issue) each concluded external-possible with no code defect; the 2026-11-13 owner triage matched earlier bursts to draw-service container recreations and a farm outage and closed the issue as operational.
Host evidence re-checked at triage time: runtime running, restart_count 0, not OOM-killed, moderate CPU and load, ample memory and disk; deployed revision and build revision equal the repository base.

Hypotheses

H1 (leading, external): the single self-hosted draw backend returned 5xx-class responses or refused connections during a continuing burst window (service restart/recreation or outage), failing both candidates of the configured chain; three draw workflows on that backend showed the same signature within one morning window.
H2 (weakened, capacity): concurrent load from three parallel VIP draw slots on the same backend and capacity pool provokes throttling; weakened because on this code path rate-limit responses classify as a different failure class than the observed one.
H3 (weak): a client-side request defect on this workflow's path provokes the failures; weakened because sibling draw lanes sharing provider, service and generator code failed identically in the same window and the lane otherwise completes with zero failures.

Supporting

Both distinct endpoint models of the chain fail with the identical provider-response class on the same backend within 60-90 s of each other, pointing to a backend-level rather than model-level condition.
The observed class maps to provider HTTP 429/>=500 responses in the image client (crates/openplotva-app/src/image_jobs.rs), a provider-response classification rather than an internal error path; verified at HEAD, the shared rule table matches rate-limit phrases to a different class first, so the observed class implies 5xx-class statuses or transport/connection failures.
No draw workflow has a cross-provider fallback (routing migrations keep every draw slot on the single draw provider), so exhaustion under a backend-wide burst is configured behavior.
Host, container, database and queue evidence healthy at capture, ruling out app-wide resource failure or crash-looping on our side.
Two other draw workflows on the same backend hit the same signature earlier the same morning, so the condition is backend-wide across lanes, not specific to this workflow.

Contradicting

No raw provider response bodies, HTTP status codes or transport errors are available in the scoped evidence, so a client-side misclassification or request defect cannot be formally excluded from observation alone.
Backend-side health for this specific burst window (draw-service container recreation/restart events, load) is not accessible from the scoped evidence tools, so the restart correlation established on 2026-11-13 is unconfirmed for this window.
Windowed lane counters cannot be aligned to the exact burst instants, so sibling-lane health during the window is inferred rather than measured.

Related changes

PR Route FLUX.3 work and a third VIP picture to Fibo 2.0 #278 (merged, deployed in the running revision): created this third VIP workflow with a primary draw model plus same-provider fallback and re-pointed draw routing; both candidates in the failing chain come from this change.
PR Give VIP draws a third picture from MOSS-Image #266 (merged): added parallel VIP draw slots on the same single backend, load context for the capacity hypothesis.
PR Free draws on Krea 2; VIP draws Krea 2 + MOSS-Image #274 (merged, deployed): built the same-provider two-candidate draw shape whose exhaustion produces this signature family.
PRs Classify validator-rejected dialog answers as model output #296 and Keep stored workflow settings on a partial admin update #297 (merged 2026-11-13, deployed): dialog answer classification and partial workflow admin update; their file lists show no draw-path changes.
Issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279 (closed operational): tracking issue for this exhaustion-signature series; its owner triage matched bursts to draw-service container recreations.

Missing

Backend operational state for the 08:44-08:59Z burst window (draw-service container recreation/restart events, load, slot residency) to confirm the restart correlation for this occurrence; not accessible from the scoped evidence tools.
The classified HTTP status codes or transport errors behind each provider_unavailable attempt (5xx versus connection-refused distinction), not exposed by the attempt rows.
Attempt rows for all three exhaustions in this window and full timestamps of the signature's occurrences to establish burst clustering.
Deployed workflow assignment rows for this workflow (primary/fallback roles, weights, breaker settings) to verify production routing matches the migrations' intent; scoped tools cannot query SQL.
Window-aligned failure records for sibling draw lanes at the exact burst instants, to measure provider-wide correlation rather than infer it.

Acceptance

Deep stage or owner: correlate the 08:44-08:59Z burst window with backend operational state (service recreation/restart events); a match closes the causal chain the same way the 2026-11-13 triage did.
Deep stage: obtain the classified status codes or transport errors behind the provider_unavailable attempts for the window.
Any code fix requires a reproduced, verifiable behavioral defect in classification, fallback selection or breaker handling; none is observed.
Resilience options (deploy-aware suppression of these terminal events, or a cross-provider draw fallback) are deliberate design changes to propose separately, not defects to fix in this incident.

Impact

3 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

Problem

The terminal event all_attempts_exhausted with reason provider_unavailable recurred 5 times in about 12.5 minutes on the third VIP draw picture slot (created by the draw re-route now deployed; running revision equals the repository base). Seven attempt rows, all provider_unavailable on the single draw provider, alternate between the chain's primary dev draw model and its turbo fallback at a uniform ~90-100 s cadence across ~9.5 minutes, i.e. a sustained provider-level condition rather than a single episodic burst. The reason is a provider-response/connection classification in our client (HTTP 429 or >=500, transport failures, unreachable backend), and the workflow chain is deliberately single-provider with both candidates sharing one GPU media pool, so exhaustion under a provider-wide condition is configured behavior. No verifiable code defect is observed; raw provider responses and provider-side health data are missing, so a client-side misclassification cannot be fully excluded.

External cause: possible. Code defect: not_observed.

Observations

Incident snapshot: event_type all_attempts_exhausted, reason provider_unavailable, route is the primary dev draw model on the single draw provider, workflow is the third VIP text-to-image slot; 5 occurrences between 2026-11-14T09:14:31Z and 09:27:06Z, status diagnosing.
Route-attempt evidence: 7 attempt_failed rows all reason provider_unavailable, all on the same single draw provider, alternating the chain's primary dev draw model and the turbo fallback at a uniform ~90-100 s cadence from 09:17:41Z to 09:27:06Z; every row references this incident (attempt:1..7), so linkage is unambiguous.
The uniform spacing with no recovery gap across ~9.5 minutes fits a sustained provider-level condition during that window rather than isolated per-request faults.
The deployed runtime revision equals the repository base commit (running, restart_count 0, oom_killed false, live host evidence re-checked), so the running code is exactly main including the draw re-route that created this workflow.
Directly verified at base this session: image_jobs.rs defines the third VIP slot workflow key; the image client maps HTTP 429 or >=500 to provider_unavailable and other statuses to request_failed; retry.rs defines provider_unavailable as a provider-response FailureReason; migration 212 creates this workflow with 3 hops and a 240000 ms retry wall, Fibo 2.0 Dev primary and Krea 2 fallback, both sharing the GPU1 media pool with the other [configured resource] models.
The migration comment sizes the retry wall so the fallback can still start after an attempt that waited for the lane and failed late (a primary draw takes 100-120 s), which matches the observed uniform ~90-100 s spacing between alternating candidate attempts.
Verified earlier at base (predecessor scoping, issue 279): the draw client classifies HTTP 429 or >=500 as provider_unavailable, other statuses as request_failed, and transport/unreachable-backend failures as provider_unavailable in the shared retry classifier, so the reason is a provider-response/connection class, not an internal validation path.
The workflow is created by migration with retry budget 3 hops and a 240 s retry wall sized for a 100-120 s primary draw; primary is the dev draw model with the turbo model as fallback, and both models are documented to share the same GPU media pool on the single draw provider, so there is no cross-provider fallback.
Host, database and runtime healthy at evidence time: running, no restarts or OOM kills, moderate CPU, ample memory and disk, zero waiting locks, queues progressing.
Sibling draw lanes remained broadly healthy in windowed counters: VIP draw lane 2 failed vs 107 completed, regular draw lane 4 failed vs 2078 completed, so failures stay low relative to completions.
A repository-level guard suppresses single-attempt exhaustions with retryable reasons; this incident has multiple failed attempts per evidence and surfaced as designed.
No task payload is linked to this incident (task evidence not_linked), so job-level context of the occurrences is unavailable.
An open issue on the dialog lane documents provider_unavailable-class exhaustion together with GPU device faults on the shared farm since the 2026-11-05 boot, in the same overall window, though on a different client and workflow.

Hypotheses

H1 (leading, external): the single draw backend was unavailable or refusing/limiting (429/5xx or connection failures) for a sustained ~10-minute window; both distinct endpoint models failed the identical class throughout and exhaustion followed as configured; farm-side GPU fault reports since the 2026-11-05 boot fit this.
H2 (related to our changes, unproven): the re-route concentrated additional draw work (a third parallel VIP picture plus all former previous-model assignments) onto the same provider and GPU media pool, raising concurrent slot demand enough to provoke 429 capacity responses; supported by topology but not by provider-side load data.
H3 (design question, weaker): job-level retry cadence versus the workflow retry wall lets jobs re-exhaust quickly during a provider-down window (5 exhaustions in 12.5 minutes), amplifying occurrence count; configured behavior to review as a resilience decision rather than a defect.
H4 (weakly contradicted): a client-side request defect specific to this workflow's path provoking the failures; weakened because sibling draw lanes share the provider, service and generator code path and complete with near-zero failures, and the classification matches the previously verified provider-response path, but raw responses are needed to exclude it fully.

Supporting

Two distinct endpoint models on the same provider failed with the identical reason class throughout a ~9.5-minute window at uniform cadence, consistent with a provider-level condition rather than a model-specific one.
The alternating primary/fallback attempt pattern at ~90-100 s spacing matches the designed chain behavior (long primary attempt that waits on the lane, then fallback) during a provider-down window, so the attempt sequence needs no client malfunction to explain it.
provider_unavailable is a provider-response/connection classification in the client code, verified at the deployed revision during the predecessor scoping, not an internal error path.
The workflow's chain is deliberately single-provider by migration, so exhaustion under a provider-wide condition is configured behavior, exactly as scoped on the predecessor workflow (issue 279).
Host, runtime, database and non-draw queues were healthy, ruling out app-wide resource failure or crash-looping at evidence time.
Failure counts remain very low relative to completions on the draw lanes, fitting a time-bounded provider condition rather than broken routing or a bad deploy.
All 7 visible attempt rows reference this incident directly, resolving the attempt-linkage ambiguity noted in the predecessor analysis.

Contradicting

No raw provider responses (exact HTTP status codes or transport errors) exist in the scoped evidence, so a client-side misclassification or request defect (H4) cannot be excluded from observation alone, and 429 capacity cannot be separated from 5xx failure or connection faults.
Only 7 attempt rows are visible while the occurrence count is 5 (roughly 10 candidate attempts expected); the full per-occurrence attempt sets and execution order within each occurrence are unverified.
If the provider were hard-down for the full window, sibling draw lanes on the same provider should fail in the same window; their windowed counters are near zero, but windowed counters cannot be aligned to the exact instants.
The re-route merge predates first_seen by weeks, so a direct deploy-triggered regression is not supported; only the load-shape change it introduced remains a candidate contributor (H2).

Related changes

The draw re-route PR 278 (merged 2026-10-23, now deployed at the running revision): created this workflow with a third parallel VIP picture on the same provider, replaced every previous-model assignment with the new model family, and kept the chain single-provider on the shared GPU media pool.
The prior VIP expansion PR 266 (merged 2026-10-19): added parallel picture slots on the same single provider, part of the concurrent-demand growth that H2 needs evaluated.
Closed issue 279: the predecessor single-provider chain's bursts; its analysis verified the same classification code and concluded external possible / code defect not_observed, with an explicit acceptance item to re-check incidence after the re-route deployed. This incident is that post-deploy recurrence on the new chain.
Open issue 294: provider_unavailable-class exhaustion on the dialog lane together with GPU device faults on the shared farm since the 2026-11-05 boot; same failure class and shared farm, different workflow and client.

Missing

Raw provider responses for the failed attempts: exact HTTP status codes or transport errors per attempt, to separate 429 capacity from 5xx failure or connection-level faults and to exclude client-side misclassification (H4).
The complete attempt sets for all 5 occurrences with execution order (7 of ~10 expected attempts visible), to confirm both chain candidates of each occurrence actually ran and failed.
Draw backend health and logs for the window 09:14-09:28 UTC (slot residency, queue depth, GPU faults, worker restarts) to confirm the provider-side condition (H1) versus load provoked by our added slots (H2); not accessible from the scoped evidence tools.
Deployed workflow_assignments rows for this workflow (primary/fallback roles, weights, breaker settings) to verify production routing matches the migration's intent rather than assuming it.
Timestamps of exhaustion events on sibling draw workflows aligned to the burst instants, to distinguish provider-wide correlation from workflow-specific failures.
The linked task payloads (task evidence returns not_linked), so job-level retry cadence and queue behavior for the occurrences cannot be inspected.

Acceptance

Deep stage: obtain the classified status codes or transport errors behind the provider_unavailable attempts for the window; 429 capacity versus 5xx/transport failure changes both H1's mechanism and any mitigation.
Deep stage: recover the full attempt sets for all 5 occurrences with execution order, to confirm both chain candidates of each occurrence actually ran and failed.
Deep stage: verify the deployed workflow_assignments rows for this workflow match the migration's intent (dev-model primary, turbo fallback, 3 hops, 240 s wall) in production, not just in the repository.
Deep stage: align sibling draw-workflow failures on the same provider with the window instants; provider-wide correlation confirms H1, while failures confined to this workflow point back at our request path (H4).
Deep stage: review whether the retry cadence versus the workflow retry wall should space retries during provider-down windows, as a deliberate resilience decision rather than a defect fix.
Any code fix requires a reproduced, verifiable behavioral defect in classification, fallback selection or breaker handling; none is observed so far.

Impact

5 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

[configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279

Description

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

Activity

iamwavecut commented on Oct 25, 2026

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

iamwavecut commented on Nov 13, 2026

Triage 2026-11-13: operational, closing

iamwavecut commented on Nov 14, 2026

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

iamwavecut commented on Nov 14, 2026

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

iamwavecut commented on Nov 14, 2026

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

iamwavecut commented on Nov 14, 2026

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

iamwavecut commented on Nov 14, 2026

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

Metadata

Metadata

Assignees

Labels

Projects

Milestone