fix: reconcile the participant roster after a signal resume - #1198
Conversation
A resume, unlike a full reconnect, never rebuilds the roster from a JoinResponse, and the DISCONNECTED update for anyone who left during the outage went to a socket we no longer had. Those participants stayed in `room.remoteParticipants` forever. The server answers a resume with the ReconnectResponse followed immediately by a full roster snapshot on the same socket, so the snapshot is authoritative: any participant we still hold that is absent from it left while we were away and its disconnect is synthesized. Mirrors `reconcile_absent_participants` in rust-sdks. The reconciliation is armed off `SignalReconnectResponseEvent` rather than `SignalReconnectedEvent` — the latter is emitted only after the engine's async ReconnectResponse handling (setConfiguration on both transports, reliable-message replay) and can lose the race against the update that follows it on the wire. Arming off the raw signal message keeps the two ordered. If no snapshot ever arrives the reconciliation simply never fires, so a missing snapshot can never be read as "everyone left". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arming had no expiry, so if the server's post-resume snapshot never arrived it sat there indefinitely and the next *ordinary* participant update — which lists only what changed — was mistaken for a full roster and evicted everyone else. That is the one failure direction this feature must never have. Two guards: - The arming now expires after 5s. The snapshot follows the ReconnectResponse on the same socket, so the window only has to cover scheduling, never a real wait; if it lapses the reconciliation simply never runs. - An update only counts as the snapshot if it carries the local participant. The server includes it so metadata changes propagate, which is exactly what distinguishes a full roster from a partial update. This also covers the RoomMoved path, which reuses the same handler with `otherParticipants` (no local entry). Both failure modes are now "no reconciliation", never "evict a live participant". Tests cover a partial update arriving while armed, an update after the window lapses, and escalation to a full reconnect while armed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| _resumeRosterSnapshotTimeout = Timer(_resumeRosterSnapshotWindow, () { | ||
| if (_resumeRosterSnapshot != null) { | ||
| logger.fine('resume roster snapshot never arrived, skipping reconciliation'); | ||
| } | ||
| _disarmResumeRosterSnapshot(); |
There was a problem hiding this comment.
🟡 Slow snapshots skip roster reconciliation
When processing an arrived roster takes five seconds, _resumeRosterSnapshotTimeout disarms it before reconciliation. Participant and track updates run sequentially. Participants absent during the outage remain in the room indefinitely.
Learn more
The timer measures time until reconciliation finishes, not time until the snapshot arrives. The roster handler captures the armed set, then awaits each participant update before checking that the set remains armed. Existing participants can perform asynchronous track reconciliation in updateFromInfo. If those operations exceed five seconds, this callback clears the set, so the final identity check rejects the already-arrived snapshot.
Example: A 200-participant snapshot arrives immediately, but sequential track updates take 5.2 seconds. The timer clears _resumeRosterSnapshot at five seconds. The absent participant leaver is never reconciled and remains in remoteParticipants.
Recommended fix: Detect a qualifying batch synchronously when _onParticipantUpdateEvent starts and cancel its arrival timeout immediately. Keep the captured set alive until processing and reconciliation finish, while preserving the identity guard for a later resume.
Was this helpful? React with 👍 or 👎 to provide feedback.
Fixes CLT-3323. Stacked on #1197 (base is that branch, not
main) — it carries the mocksetConfigurationfix the resume path needs. GitHub will retarget this tomainonce #1197 merges.Problem
After a signal resume (node migration, transient signal drop), participants who left while the link was down are never removed. Their
DISCONNECTEDupdate went to a socket we no longer had, and a resume — unlike a full reconnect — never rebuilds the roster from aJoinResponse. They stay inroom.remoteParticipantsindefinitely.This became reachable once #1197 stopped migrations from full reconnecting. The full reconnect used to unwind the whole roster and rebuild it, which masked the gap; now that migrations correctly keep participants, stale entries are the failure mode.
client-sdk-js has the same gap. rust-sdks fixed it (
reconcile_absent_participants).Fix
The server answers a resume with the
ReconnectResponsefollowed immediately by a full roster snapshot on the same socket (livekit/pkg/rtc/room.go—HandleReconnectAndSendResponse, thenSendParticipantUpdate). That snapshot is authoritative: any participant we still hold that is absent from it left while we were away, so its disconnect is synthesized through the existing_handleParticipantDisconnectpath.The whole design question here is: what counts as the snapshot? Getting that wrong in the permissive direction evicts live participants, so the arming is fenced three ways:
SignalReconnectResponseEvent, notSignalReconnectedEvent. The latter is emitted only after the engine's async ReconnectResponse handling (setConfigurationon both transports, reliable-message replay) and can lose the race against the roster update that follows it on the wire — I hit exactly that while writing the tests.RoomMovedpath, which reuses the same handler withotherParticipants(no local entry).ReconnectResponseon the same socket, so the window only has to cover scheduling, never a real wait.Net: every failure mode is "no reconciliation, ghost survives", never "evict a live participant". Disarmed on full restart (which rebuilds the roster itself) and in
_cleanUp, which also covers dispose.This differs from rust-sdks, which accumulates a union of updates and reconciles after a 1 s
PC_RECONNECT_SETTLE_DELAY. Keying off the snapshot is deterministic and adds no delay toRoomReconnectedEvent.Tests
test/core/resume_roster_reconcile_test.dart, ported from rust-sdks'test_resume_synthesizes_disconnect_for_participant_that_left. That test needs a live SFU plus adrop_disconnected_updatesfault-injection switch; the mock transport gives the same setup for free — we simply never deliver the leaver's disconnect. The observer/leaver/witness shape is kept, so the witness proves reconciliation only removes participants that actually left.ParticipantDisconnectedEvent, witness retainedFull suite (415 tests),
flutter analyze, format and import_sorter all clean.🤖 Generated with Claude Code