fix: harden pipeline and ops transient failure recovery

[i-xtsu-sixyou-ken-mei]

Intent

Reduce recurrence of the actionable production failures in the 2026-09-07 zap-pilot-ops snapshot, and make the remaining Sentry candidates explicit so they can be resolved only after deploy/observation rather than being silently hidden.

Context

The snapshot contained 5 unresolved podcast-pipeline Sentry issues, 1 zap-pilot-web issue, a repeatedly failing Ops Cost Sync, and a failing Cron Failure Alert.

Candidate Sentry issues from the handoff:

Candidate	Current disposition
7697446832 / PODCAST-PIPELINE-6 — [step:generateScript] OpenRouter request timed out after 600000ms	Fixed in this PR. Script generation keeps its 600s long-form budget, but timeout is now a normal retryable transport failure and advances through LLM_FALLBACK_MODELS instead of terminating on the primary model.
7713815594 / PODCAST-PIPELINE-16 — TypeError: fetch failed from Supabase	Safely mitigated in this PR. Supabase GET/HEAD requests retry transient network errors, 408/429, and 5xx with bounded backoff. Mutations are deliberately never blindly replayed. The available Sentry sample does not identify whether the failed operation was a read or mutation, so recurrence after deploy remains the verification gate.
7714955636 / PODCAST-PIPELINE-19 — vipspng: libpng read error	Already fixed on current main before this PR. Visual fingerprint/decode failure is caught as a candidate rejection, the bad image is removed, and planning continues with the next candidate. No duplicate code change added here.
UNKNOWN_4 — low-volume podcast-pipeline candidate not returned by the top-3 MCP sample	Not safely actionable. No title/stack/event sample was available; this PR does not guess a remediation.
UNKNOWN_5 — low-volume podcast-pipeline candidate not returned by the top-3 MCP sample	Not safely actionable. No title/stack/event sample was available; this PR does not guess a remediation.
7716381551 / ZAP-PILOT-WEB-1 — NetworkError: Failed to fetch analytics-engine	No speculative code change. The shared web HTTP layer already retries network/timeout/5xx once and the affected analytics calls already use a 60s request budget. The single observed event does not identify a missing client recovery mechanism; verify by post-deploy observation before resolving.

Related non-Sentry signals addressed here:

• ops-cost-sync.yml repeated Brave provider failure: Brave quota probe now retries transient network/408/429/5xx failures with bounded backoff and retains the concrete final error instead of collapsing to Provider request failed.
• cost-ledger:provider/brave unknown: same collector fix restores a durable Brave snapshot when the probe succeeds after a transient failure.
• cron-failure-alert.yml failure: the alert no longer depends on separately synchronized Telegram GitHub secrets. It authenticates the existing Infisical machine identity and reads the Telegram bot token / allowed-user list from the production environment rail.

Operational signals intentionally not guessed at from this snapshot:

• social-daemon ... stale: local-Mac process/supervision state, not enough evidence of an application defect in this incident packet.
• social-queue:waiting-media/episodes 18 lanes: a symptom without the blocking stage/error sample; no safe generic retry/change inferred here.

Scope

• unify script timeout behavior with the shared OpenRouter model fallback chain
• add bounded, read-only Supabase transport retry
• harden Brave cost collection and preserve diagnostics
• move cron failure Telegram notification onto the canonical production env rail
• add/update regression tests and schedule docs

Out of scope

• blindly retrying Supabase mutations
• changing already-present zap-pilot-web HTTP retry budgets without evidence
• inventing fixes for the two Sentry candidates whose event samples are unavailable
• restarting operator-Mac processes or force-releasing waiting-media jobs from CI
• resolving Sentry issues automatically; operator will resolve after deployment/observation

Product contract / invariants

• This PR does not change a documented product/architecture invariant.
• This PR intentionally changes an invariant with explicit product approval.

Affected invariant:
OpenRouter transport fallback policy in apps/podcast-pipeline/src/services/llm.ts and its regression tests. Script generation previously opted out of timeout model failover.

Why:
A 600s primary-model timeout produced Sentry 7697446832 and failed the episode even when LLM_FALLBACK_MODELS was configured. Retryable transport failures should have one consistent model-fallback policy across workloads.

Acceptance criteria

• A script primary model that times out can continue on the next configured LLM_FALLBACK_MODELS candidate with a fresh 600s deadline.
• Exhausting the script model chain does not trigger a second full timeout replay through endpoint rerouting.
• Supabase reads retry only bounded transient transport/server failures; mutations remain single-attempt.
• Brave quota collection retries bounded transient failures and surfaces useful terminal diagnostics.
• Cron failure alerting reads Telegram credentials from the production Infisical rail instead of duplicate Telegram repository secrets.
• Every candidate Sentry issue from the handoff is recorded above with its actual remediation/verification state.

Implementation

• Removed the script-only fallbackOnTimeout: false escape hatch from the shared OpenRouter transport policy.
• Kept SCRIPT_OPENROUTER_TIMEOUT_MS = 600_000; each configured model candidate gets its own deadline.
• Installed a custom Supabase global.fetch wrapper that retries GET/HEAD only, up to 3 attempts with 250ms/500ms backoff for network failures, 408/429, and 5xx.
• Added 3-attempt Brave quota-probe retry with 250ms/500ms backoff; preserved Brave Search ... diagnostics through Control Center.
• Added scripts/notify-cron-failure.mjs; the workflow authenticates Infisical and invokes it through scripts/env/run.mjs --environment prod.
• Updated docs/schedules.md to remove the old manual Telegram GitHub-secret synchronization procedure.

Contract alignment

• Implementation matches the approved fallback invariant.
• Regression / contract tests cover the changed retry behavior.
• Scoped AGENTS.md remains accurate; no scoped instruction change was required by this patch.
• README / runbook remains accurate (docs/schedules.md updated for alert credentials).

Validation

• Repository diff review against latest main (c9b0dc292ab2d63853ba986f4748ea3dd192fcbd) — clean: branch is 1 commit ahead / 0 behind and changes only the 10 intended files.
• Regression tests added/updated for OpenRouter script timeout failover, Supabase safe read retry, and Brave retry behavior.

Validation gaps

• This connector environment cannot execute the repository test suite locally. GitHub PR CI is the executable validation gate.
• Production verification still requires deployment plus observation for Sentry recurrence, especially 7713815594, 7714955636, and 7716381551.
• UNKNOWN_4 / UNKNOWN_5 require their concrete Sentry event samples before claiming any fix.

Known unrelated failures

None intentionally introduced. Existing production signals listed above that lack sufficient diagnostics are recorded rather than hidden.

Reviewer notes

Focus review on retry safety boundaries:

1. OpenRouter timeout failover is model-level only; an exhausted timeout chain is not replayed again through the script endpoint-reroute path.
2. Supabase mutation methods are specifically excluded from transport replay.
3. Cron alerting now has one credential source of truth: the prod Infisical rail.