fix(playback): make the stall watchdog level-triggered with a host escalation rung - #342
Open
tschuegy wants to merge 1 commit into
Open
Conversation
…calation rung The superuser404notfound#65 stall re-engage watchdog was one-shot and edge-triggered: armed per playbackStalled notification, then a single instantaneous check 6 s later. Any media fetch inside that grace window disarmed it PERMANENTLY. Field trace: a live playlist froze at its last segment, AVPlayer drained the remaining tail (fetches during the grace window killed the watchdog), then parked in waitingToMinimizeStalls with ~2 s buffered — a state in which playbackStalled never re-fires, a waiting player never posts failedToPlayToEndTime, and the producer-side wedge detector had died with the pump. No recovery layer ever re-examined the session; the freeze was terminal from the couch. Three changes, smallest that closes every exit from that trap: - Level re-watch: fetch activity during the grace window re-baselines and keeps watching (bounded by maxStallWatchPasses) instead of returning permanently; only a fully silent window escalates into the unchanged nudge/reload ladder. - Persistent reload budget (storm shape): on a frozen playlist each stage-2 reload replays the tail and re-stalls within seconds, and the fresh stall supersedes the ladder task before any post-reload check can run — so a per-task rung alone can never fire. A gate spanning stall events (same shape as the superuser404notfound#93 item-death gate) admits two reloads at the same frozen position, then the ladder publishes liveSourceReset instead of reloading again. - Post-reload rung (silent-park shape): when no further stall supersedes, a rendered clock that has not moved a whole window after the stage-2 reload while the player still waits publishes liveSourceReset — the local session is unrecoverable consumer-side and only the host can retune. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The #65 stall re-engage watchdog is one-shot and edge-triggered: armed per
playbackStallednotification, then a single instantaneous check 6 s later. Any media fetch inside that grace window disarms it permanently — only a newplaybackStalledre-arms.Field trace: a live playlist froze at its last segment (pump death; recovery arm added in #341). AVPlayer stalled, drained the remaining tail segments (fetches inside the grace window → watchdog dead), then parked in
waitingToMinimizeStallswith ~2 s buffered:In that parked state nothing can ever fire again:
playbackStalledneeds a buffer-exhaustion transition (buffer stays non-empty), a waiting player never postsfailedToPlayToEndTime, and the producer-side wedge detector died with the pump. The freeze was terminal from the couch.Fix
Three changes; together they close every exit from the trap:
Level re-watch. Fetch activity during the grace window re-baselines and keeps watching (bounded by
maxStallWatchPasses = 10≈ 60 s) instead of returning permanently. Only a fully silent window escalates into the unchanged nudge → stage-2-reload ladder; a recovered/paused/failed player still disarms (those states have other owners). Verdict extracted as a pure function.Persistent reload budget (storm shape). On a frozen playlist each stage-2 reload replays the tail and re-stalls within seconds, and the fresh stall supersedes the ladder task before any post-reload check can run — a per-task rung alone can never fire. A gate spanning stall events (same shape and epsilon as the Loopback-HLS: a single backward seek on heavy 4K can wedge the segment producer (video stalls, audio continues -> A/V desync) #93 item-death gate) admits two reloads at the same frozen position; after that the ladder publishes
liveSourceResetinstead of reloading again. Real progress restores the budget.Post-reload rung (silent-park shape). When no further stall supersedes, a rendered clock that has not moved a whole window (12 s) after the stage-2 reload while the player still waits publishes
liveSourceReset: the local session is unrecoverable consumer-side and only the host can retune. Live-gated; decision extracted as a pure function.liveSourceResetis the existing host contract ("negotiate a fresh URL and call load"); this only adds the two missing publishers for consumer-side dead ends.Tests
StallWatchdogLevelRearmTestspins the incident case (fetch-during-grace must re-watch, not disarm), escalation on silence, disarm on recovery/failure/cap, the frozen-clock rung (live-only), and the storm-shape budget (two reloads at a frozen position, then escalate; progress restores).🤖 Generated with Claude Code