Skip to content

fix(playback): make the stall watchdog level-triggered with a host escalation rung - #342

Open
tschuegy wants to merge 1 commit into
superuser404notfound:mainfrom
tschuegy:fix/stall-watchdog-level-rearm
Open

fix(playback): make the stall watchdog level-triggered with a host escalation rung#342
tschuegy wants to merge 1 commit into
superuser404notfound:mainfrom
tschuegy:fix/stall-watchdog-level-rearm

Conversation

@tschuegy

@tschuegy tschuegy commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Problem

The #65 stall re-engage watchdog is one-shot and edge-triggered: armed per playbackStalled notification, then a single instantaneous check 6 s later. Any media fetch inside that grace window disarms it permanently — only a new playbackStalled re-arms.

Field trace: a live playlist froze at its last segment (pump death; recovery arm added in #341). AVPlayer stalled, drained the remaining tail segments (fetches inside the grace window → watchdog dead), then parked in waitingToMinimizeStalls with ~2 s buffered:

[NativeAVPlayerHost] #7 playbackStalled
[LagDiag] clk=483.66 dclk=0.00 tcs=waiting rate=1.00 wait=AVPlayerWaitingToMinimizeStallsReason fwd=2.0 keepUp=n empty=n ... (forever)

In that parked state nothing can ever fire again: playbackStalled needs a buffer-exhaustion transition (buffer stays non-empty), a waiting player never posts failedToPlayToEndTime, and the producer-side wedge detector died with the pump. The freeze was terminal from the couch.

Fix

Three changes; together they close every exit from the trap:

  1. Level re-watch. Fetch activity during the grace window re-baselines and keeps watching (bounded by maxStallWatchPasses = 10 ≈ 60 s) instead of returning permanently. Only a fully silent window escalates into the unchanged nudge → stage-2-reload ladder; a recovered/paused/failed player still disarms (those states have other owners). Verdict extracted as a pure function.

  2. Persistent reload budget (storm shape). On a frozen playlist each stage-2 reload replays the tail and re-stalls within seconds, and the fresh stall supersedes the ladder task before any post-reload check can run — a per-task rung alone can never fire. A gate spanning stall events (same shape and epsilon as the Loopback-HLS: a single backward seek on heavy 4K can wedge the segment producer (video stalls, audio continues -> A/V desync) #93 item-death gate) admits two reloads at the same frozen position; after that the ladder publishes liveSourceReset instead of reloading again. Real progress restores the budget.

  3. Post-reload rung (silent-park shape). When no further stall supersedes, a rendered clock that has not moved a whole window (12 s) after the stage-2 reload while the player still waits publishes liveSourceReset: the local session is unrecoverable consumer-side and only the host can retune. Live-gated; decision extracted as a pure function.

liveSourceReset is the existing host contract ("negotiate a fresh URL and call load"); this only adds the two missing publishers for consumer-side dead ends.

Tests

StallWatchdogLevelRearmTests pins the incident case (fetch-during-grace must re-watch, not disarm), escalation on silence, disarm on recovery/failure/cap, the frozen-clock rung (live-only), and the storm-shape budget (two reloads at a frozen position, then escalate; progress restores).

🤖 Generated with Claude Code

…calation rung

The superuser404notfound#65 stall re-engage watchdog was one-shot and edge-triggered:
armed per playbackStalled notification, then a single instantaneous
check 6 s later. Any media fetch inside that grace window disarmed it
PERMANENTLY. Field trace: a live playlist froze at its last segment,
AVPlayer drained the remaining tail (fetches during the grace window
killed the watchdog), then parked in waitingToMinimizeStalls with ~2 s
buffered — a state in which playbackStalled never re-fires, a waiting
player never posts failedToPlayToEndTime, and the producer-side wedge
detector had died with the pump. No recovery layer ever re-examined
the session; the freeze was terminal from the couch.

Three changes, smallest that closes every exit from that trap:

- Level re-watch: fetch activity during the grace window re-baselines
  and keeps watching (bounded by maxStallWatchPasses) instead of
  returning permanently; only a fully silent window escalates into the
  unchanged nudge/reload ladder.
- Persistent reload budget (storm shape): on a frozen playlist each
  stage-2 reload replays the tail and re-stalls within seconds, and
  the fresh stall supersedes the ladder task before any post-reload
  check can run — so a per-task rung alone can never fire. A gate
  spanning stall events (same shape as the superuser404notfound#93 item-death gate) admits
  two reloads at the same frozen position, then the ladder publishes
  liveSourceReset instead of reloading again.
- Post-reload rung (silent-park shape): when no further stall
  supersedes, a rendered clock that has not moved a whole window after
  the stage-2 reload while the player still waits publishes
  liveSourceReset — the local session is unrecoverable consumer-side
  and only the host can retune.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant