Skip to content

fix(video): give live muxerFailed pump exits a recovery arm - #341

Merged
superuser404notfound merged 1 commit into
superuser404notfound:mainfrom
tschuegy:fix/live-muxerfailed-recovery
Aug 9, 2026
Merged

fix(video): give live muxerFailed pump exits a recovery arm#341
superuser404notfound merged 1 commit into
superuser404notfound:mainfrom
tschuegy:fix/live-muxerfailed-recovery

Conversation

@tschuegy

@tschuegy tschuegy commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Problem

A live pump death with muxerFailed has no recovery arm: handlePumpFinished hits a bare return, shouldHaltLiveProduction returns false for it (so the provider is not even halted), and no host signal fires. The provider keeps serving a frozen playlist, AVPlayer parks on it waiting for buffer that never comes, and the session zombifies until the viewer zaps away.

Field trace (immediately after the rotation death that #340 fixes at its root):

[HLSSegmentProducer] pump finished: reason=muxerFailed packetsRead=39549 packetsWritten=24320 lastError=0
[NativeAVPlayerHost] #7 errorLog ... -12888 - Playlist File unchanged for longer than 1.5 * target duration  (repeats forever)
[LagDiag] clk=483.66 dclk=0.00 tcs=waiting rate=1.00 wait=AVPlayerWaitingToMinimizeStallsReason fwd=2.0 ... (frozen until app exit)

The shouldHaltLiveProduction comment assumed "AE#222 rebuilds into the same provider" — that was doubly false for live:

  1. muxerFailed never reaches the AE#222 arm at all (bridged sessions are excluded from the exit-scan, so the reason is never converted).
  2. Even the AE#222 arm itself was a silent no-op for live: its live paths returned without doing anything, and the admitted rebuild goes through requestRestart, which bails on the empty live segmentPlan (performRestart's idx < segmentPlan.count guard) — a second zombie path, hit by non-bridged E-AC-3 rotations.

Fix

New handleLiveMuxerFailure: rebuilds the producer in place on the same demuxer and provider at liveContinuationPoint()finishLiveReopen minus the fresh connection. A reopen is deliberately not used: the socket is healthy (the muxer died, not the connection), and performLiveReopen opens its new connection before closing the old demuxer, which would double-connect against max_connections=1 portals.

  • Bounded by progress, not per session: new segments since the last death reset the budget (an hours-long channel legitimately crosses several encoder restarts); three consecutive barren deaths exhaust it.
  • On exhaustion (or a producer-build throw): markLiveProductionHalted() — releasing held blocking-reload waiters — and onLiveSourceReset so the host can retune. Same last-resort surface as Rerouted live-ingest session death relands on the known-bad native path: #168 verdict not remembered, no in-engine revive #199's reopen exhaustion.
  • Live .needsAudioSampleEntryPrime is routed through the same in-place rebuild (the captured prime is already in sessionAudioMoovPrimeFrame), closing zombie path 2.
  • shouldHaltLiveProduction still returns false for both reasons — an eager halt would 503 the very provider the rebuild serves — but the comment now names the owning arm, and both arms halt themselves on budget exhaustion.

Tests

  • liveMuxerRebuildDecision extracted as a pure decision and pinned in LiveProductionHaltTests: first death always rebuilds; progress resets the budget; exactly the cap's worth of barren rebuilds, then halt + host retune.
  • Existing halt-classification assertions updated with rationale strings naming the new arm.

Root-cause companion: #340 prevents this particular muxer death entirely; this arm is what keeps any future live muxer death from zombifying the session.

🤖 Generated with Claude Code

A live pump death with muxerFailed hit a bare return in
handlePumpFinished: no rebuild, no halt (shouldHaltLiveProduction
assumed the AE#222 arm covers it), no host signal. The provider kept
serving a frozen playlist, AVPlayer parked on it waiting for buffer
that never came, and the session zombified until the viewer zapped
away. Field trace: a reconnect join with a new encoder session forced
a same-PID rotation whose first cut deferred awaiting an audio sample
entry (bridged E-AC-3), and the resulting muxerFailed exit was
swallowed.

The AE#222 assumption was doubly false for live: its live paths
returned without doing anything, and even the admitted rebuild went
through requestRestart, which bails on the empty live segmentPlan —
a second silent zombie path, now also routed into the new arm.

handleLiveMuxerFailure rebuilds the producer IN PLACE on the same
demuxer and provider at the live continuation point (finishLiveReopen
minus the fresh connection — the socket is healthy, and a reopen
would double-connect against single-connection portals because it
opens the new connection before closing the old demuxer). Bounded by
progress, not per session: new segments since the last death reset
the budget, three consecutive barren deaths halt production
(releasing blocking-reload waiters) and publish onLiveSourceReset so
the host can retune.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@superuser404notfound
superuser404notfound merged commit 1778a5c into superuser404notfound:main Aug 9, 2026
3 checks passed
superuser404notfound added a commit that referenced this pull request Aug 9, 2026
Follow-up to #341. The new in-place muxer rebuild arm ships a pure
progress-budget decision and pins it with three tests. The reopen ladder right
above it has always made the exact same decision from an inline copy, untested:
compare the provider's continuation point against the last one, count barren
attempts, give up at 3. Two copies of one rule, one of them the only tested one,
is how the two drift.

Both arms now call `liveRecoveryBudgetDecision` (renamed off the muxer arm,
since it is no longer only about muxer rebuilds). Behaviour is unchanged in both
arms: the old `barrenNow >= cap` and the new `cycles < cap` are the same
predicate over the same counters, and the reopen arm keeps its own counter pair.

Also written down at the decision, because it is a property of the rule and not
of either arm: a source that dies after every handful of segments never goes
barren and recovers forever. That hole is deliberate, bounding it needs a rate
rather than a counter, and an eager cap would kill exactly the long-running
channels the progress budget exists for.

Plus a test that both arms ship the same cap, a CHANGELOG entry, and the
house-style em-dashes removed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HX5zV3Fcf7Nzq4DYGdXvQE
@superuser404notfound

Copy link
Copy Markdown
Owner

Merged. Verified onto current main (the branch was based on 6.15.2): 1689 swift-testing plus 468 XCTest green, tvOS Simulator build clean, and the cap test is load-bearing (flipping cycles < cap to <= fails it on 4 vs 3).

The two diagnoses check out against the source, both of them:

  • performRestart really does bail on live at idx < segmentPlan.count with an empty plan, so the admitted AE#222 rebuild consumed its gate and rebuilt nothing. Silent, because the guard returns before it touches anything.
  • The in-place rebuild is the right tool rather than a reopen, and liveReopenOutputEndSeconds is what makes it safe: it sets videoTarget = Int64.min and stamps the first tfdt at the continuation point, so wiring onVideoShiftKnown to the rebase handler is required, not incidental. stop() nils the producer under restartLock before it joins the pump, so the producer === failed guard covers the teardown race and there is no lock inversion from the pump thread.

One thing pulled in on top in 513fd83: the reopen ladder directly above handlePumpFinished's new arm has always made the identical decision from an inline copy (continuation point vs the last one, count barren attempts, give up at 3), untested. Both arms now call the shared liveRecoveryBudgetDecision, renamed off the muxer arm since it is no longer only about muxer rebuilds. Same predicate over the same counters, each arm keeps its own counter pair.

Also written down at the decision rather than in either arm: a source that dies after every handful of segments never goes barren and recovers forever. Deliberate, since bounding it needs a rate rather than a counter and an eager cap would kill the long-running channels the progress budget exists for, but worth naming where the next reader will look.

Ships in the next release, together with #340.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants