Skip to content

fix: fail over to other regions when Cloud rejects a connection with 403 - #1200

Merged
hiroshihorie merged 3 commits into
mainfrom
sxian/CLT-3325/some-client-sdks-bail-on-region-pinning-403-insteadOf-failing-over
Sep 14, 2026
Merged

fix: fail over to other regions when Cloud rejects a connection with 403#1200
hiroshihorie merged 3 commits into
mainfrom
sxian/CLT-3325/some-client-sdks-bail-on-region-pinning-403-insteadOf-failing-over

Conversation

@xianshijing-lk

@xianshijing-lk xianshijing-lk commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Part of CLT-3325. Companion PRs: client-sdk-js#2097, client-sdk-swift.

Problem

LiveKit Cloud enforces project-level region pinning by returning 403 on the RTC paths (/rtc, /rtc/validate) when a project is not allowed in the region the client geo-routed to. /settings/regions is deliberately excluded from that gate so the client can discover its allowed regions and connect there — per the server-side comment, "allow other APIs to pass because clients will permanently give up if they fail."

Room.connect excluded every NotAllowed error from region failover:

(e is WebSocketException || (e is ConnectException && e.reason != ConnectionErrorReason.NotAllowed))

and the validate handler maps any status >= 400 to NotAllowed (signal_client.dart:209-215). So a pinned project that geo-routed to a disallowed region gave up before ever fetching the region list, and never reached the region it was allowed in.

Fix

Extract the decision into canFailOverToAnotherRegion and key it on the HTTP status:

  • 403 → retry other regions. This is the region-pinning signal.
  • 401 → terminal. No other region will accept a token this one rejected.
  • WebSocketException and non-NotAllowed connect errors → retry, unchanged. Unrelated error types → terminal, unchanged.

Why status and not the error message

The server's body for this case is "project not allowed in this region.", but that is an unversioned human-readable string. Matching it would mean five SDKs carrying identical literals forever, and a server-side copy edit — dropping the period, rewording — would silently break already-shipped clients. That is especially bad on mobile, where a released app cannot be hot-patched.

The cost of not discriminating is bounded: if a 403 really was a permissions failure, every region attempt fails the same way and the original error still surfaces, one region lookup later. A genuinely bad token is capped tighter still — /settings/regions returns 401 for it, and the existing handler already rethrows on that (room.dart:375-377).

Cross-SDK status

This bug is not universal. Rust and Android already fail over on any non-cancellation error and are unaffected; JS, Flutter and Swift all bail. Agents SDKs route through the Rust core, so agents are unaffected.

Testing

test/support/region_failover_test.dart covers the predicate directly. dart analyze is clean on all three touched files.

I could not execute the test suite locally — this repo requires Dart >= 3.10.0 and the toolchain on hand is 3.9.2, so flutter test fails at dependency resolution before running anything. The test needs a CI run to confirm.

Note, not fixed here

signal_client.dart:209-215 classifies any status >= 400 as NotAllowed, so 5xx responses from the validate endpoint are also treated as permission failures. That looks wrong independently of region pinning — a 5xx should probably be retryable — but it is out of scope for this change and behaviour there is unchanged.

Open question

How often the RTC 403 actually fires for pinned projects has not been confirmed with the Cloud team — geo-routing may normally land clients in-region, making this an edge case (bad GeoDNS, anycast flap, VPN). The opt-in prepareConnection() warm-up also does region selection up front and would mask it for apps that call it. The fix is correct either way and works against today's servers, but severity is unconfirmed.

🤖 Generated with Claude Code

Reviewer note (hiroshi)

Checked against the Cloud and OSS server code. The pin gate in the Cloud auth middleware matches /rtc and /rtc/v1 only, and the OSS validate handler has no 403 path, so /rtc/validate answers 200 for a pinned project that geo-routed to a disallowed region. The 403 lands on the websocket upgrade, which Flutter already surfaces as WebSocketException and fails over on. This change therefore does not alter behaviour on today's Cloud. It is kept as parity with client-sdk-js#2097 and stays correct if the gate is ever widened to validate.

Follow-ups in #1204, stacked on this branch: retry every listed region instead of one, treat a 5xx or failed validate request as retryable, and stop retried attempts from emitting a disconnect that races the next attempt's cleanup.

LiveKit Cloud enforces project-level region pinning by returning 403 on
the RTC paths when a project is not allowed in the region the client
geo-routed to. /settings/regions is deliberately left reachable so the
client can discover its allowed regions and connect there.

Room.connect excluded every NotAllowed error from region failover, and
the validate response maps any status >= 400 to NotAllowed, so a pinned
project that geo-routed to a disallowed region gave up before ever
fetching the region list.

Key on the status rather than the server's message: that message is an
unversioned human-readable string, and matching it would let a server-side
copy edit break already-shipped clients. 401 stays terminal since no other
region will accept the same token.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 2 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

@hiroshihorie
hiroshihorie merged commit 00e4c48 into main Sep 14, 2026
15 checks passed
@hiroshihorie
hiroshihorie deleted the sxian/CLT-3325/some-client-sdks-bail-on-region-pinning-403-insteadOf-failing-over branch September 14, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants