Skip to content

fix: reconcile the participant roster after a signal resume - #1198

Open
xianshijing-lk wants to merge 2 commits into
sxian/CLT-3322/flutter-session-migration-triggers-a-full-reconnect-insteadOf-resumefrom
sxian/CLT-3323/resume-leaves-stale-remote-participants-no-roster-reconciliation-after
Open

fix: reconcile the participant roster after a signal resume#1198
xianshijing-lk wants to merge 2 commits into
sxian/CLT-3322/flutter-session-migration-triggers-a-full-reconnect-insteadOf-resumefrom
sxian/CLT-3323/resume-leaves-stale-remote-participants-no-roster-reconciliation-after

Conversation

@xianshijing-lk

@xianshijing-lk xianshijing-lk commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Fixes CLT-3323. Stacked on #1197 (base is that branch, not main) — it carries the mock setConfiguration fix the resume path needs. GitHub will retarget this to main once #1197 merges.

Problem

After a signal resume (node migration, transient signal drop), participants who left while the link was down are never removed. Their DISCONNECTED update went to a socket we no longer had, and a resume — unlike a full reconnect — never rebuilds the roster from a JoinResponse. They stay in room.remoteParticipants indefinitely.

This became reachable once #1197 stopped migrations from full reconnecting. The full reconnect used to unwind the whole roster and rebuild it, which masked the gap; now that migrations correctly keep participants, stale entries are the failure mode.

client-sdk-js has the same gap. rust-sdks fixed it (reconcile_absent_participants).

Fix

The server answers a resume with the ReconnectResponse followed immediately by a full roster snapshot on the same socket (livekit/pkg/rtc/room.goHandleReconnectAndSendResponse, then SendParticipantUpdate). That snapshot is authoritative: any participant we still hold that is absent from it left while we were away, so its disconnect is synthesized through the existing _handleParticipantDisconnect path.

The whole design question here is: what counts as the snapshot? Getting that wrong in the permissive direction evicts live participants, so the arming is fenced three ways:

  • Armed off SignalReconnectResponseEvent, not SignalReconnectedEvent. The latter is emitted only after the engine's async ReconnectResponse handling (setConfiguration on both transports, reliable-message replay) and can lose the race against the roster update that follows it on the wire — I hit exactly that while writing the tests.
  • An update only counts if it carries the local participant. The server includes it so metadata changes propagate, which is what distinguishes a full roster from an ordinary partial update listing only what changed. This also covers the RoomMoved path, which reuses the same handler with otherParticipants (no local entry).
  • The arming expires after 5s. Without an expiry it would sit armed indefinitely if the snapshot never arrived, and the next ordinary update would be mistaken for a full roster. The snapshot follows the ReconnectResponse on the same socket, so the window only has to cover scheduling, never a real wait.

Net: every failure mode is "no reconciliation, ghost survives", never "evict a live participant". Disarmed on full restart (which rebuilds the roster itself) and in _cleanUp, which also covers dispose.

This differs from rust-sdks, which accumulates a union of updates and reconciles after a 1 s PC_RECONNECT_SETTLE_DELAY. Keying off the snapshot is deterministic and adds no delay to RoomReconnectedEvent.

Tests

test/core/resume_roster_reconcile_test.dart, ported from rust-sdks' test_resume_synthesizes_disconnect_for_participant_that_left. That test needs a live SFU plus a drop_disconnected_updates fault-injection switch; the mock transport gives the same setup for free — we simply never deliver the leaver's disconnect. The observer/leaver/witness shape is kept, so the witness proves reconciliation only removes participants that actually left.

  • leaver absent from the snapshot → synthesized ParticipantDisconnectedEvent, witness retained
  • both present → no disconnects
  • no snapshot at all → roster untouched
  • partial update while armed → roster untouched, arming survives
  • update after the window lapses → roster untouched
  • full reconnect while armed → the restart unwind runs once, the arming does not double-fire

Full suite (415 tests), flutter analyze, format and import_sorter all clean.

🤖 Generated with Claude Code

A resume, unlike a full reconnect, never rebuilds the roster from a
JoinResponse, and the DISCONNECTED update for anyone who left during the
outage went to a socket we no longer had. Those participants stayed in
`room.remoteParticipants` forever.

The server answers a resume with the ReconnectResponse followed
immediately by a full roster snapshot on the same socket, so the snapshot
is authoritative: any participant we still hold that is absent from it
left while we were away and its disconnect is synthesized. Mirrors
`reconcile_absent_participants` in rust-sdks.

The reconciliation is armed off `SignalReconnectResponseEvent` rather than
`SignalReconnectedEvent` — the latter is emitted only after the engine's
async ReconnectResponse handling (setConfiguration on both transports,
reliable-message replay) and can lose the race against the update that
follows it on the wire. Arming off the raw signal message keeps the two
ordered. If no snapshot ever arrives the reconciliation simply never
fires, so a missing snapshot can never be read as "everyone left".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

The arming had no expiry, so if the server's post-resume snapshot never
arrived it sat there indefinitely and the next *ordinary* participant
update — which lists only what changed — was mistaken for a full roster
and evicted everyone else. That is the one failure direction this feature
must never have.

Two guards:

- The arming now expires after 5s. The snapshot follows the
  ReconnectResponse on the same socket, so the window only has to cover
  scheduling, never a real wait; if it lapses the reconciliation simply
  never runs.
- An update only counts as the snapshot if it carries the local
  participant. The server includes it so metadata changes propagate,
  which is exactly what distinguishes a full roster from a partial
  update. This also covers the RoomMoved path, which reuses the same
  handler with `otherParticipants` (no local entry).

Both failure modes are now "no reconciliation", never "evict a live
participant". Tests cover a partial update arriving while armed, an
update after the window lapses, and escalation to a full reconnect while
armed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Devin Review

Comment thread lib/src/core/room.dart
Comment on lines +922 to +926
_resumeRosterSnapshotTimeout = Timer(_resumeRosterSnapshotWindow, () {
if (_resumeRosterSnapshot != null) {
logger.fine('resume roster snapshot never arrived, skipping reconciliation');
}
_disarmResumeRosterSnapshot();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Slow snapshots skip roster reconciliation

When processing an arrived roster takes five seconds, _resumeRosterSnapshotTimeout disarms it before reconciliation. Participant and track updates run sequentially. Participants absent during the outage remain in the room indefinitely.

Learn more

The timer measures time until reconciliation finishes, not time until the snapshot arrives. The roster handler captures the armed set, then awaits each participant update before checking that the set remains armed. Existing participants can perform asynchronous track reconciliation in updateFromInfo. If those operations exceed five seconds, this callback clears the set, so the final identity check rejects the already-arrived snapshot.

Example: A 200-participant snapshot arrives immediately, but sequential track updates take 5.2 seconds. The timer clears _resumeRosterSnapshot at five seconds. The absent participant leaver is never reconciled and remains in remoteParticipants.

Recommended fix: Detect a qualifying batch synchronously when _onParticipantUpdateEvent starts and cancel its arrival timeout immediately. Keep the captured set alive until processing and reconciliation finish, while preserving the identity guard for a later resume.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant