[ZEPPELIN-6540] Make ConnectionManager.userSocketMap thread-safe#5306
Open
HwangRock wants to merge 1 commit into
Open
[ZEPPELIN-6540] Make ConnectionManager.userSocketMap thread-safe#5306HwangRock wants to merge 1 commit into
HwangRock wants to merge 1 commit into
Conversation
userSocketMap was a plain HashMap accessed without synchronization from multiple Jetty WebSocket threads (login onMessage, disconnect onClose, note-list broadcast). Concurrent access could throw ConcurrentModificationException while iterating keySet(), spin during HashMap resize, or NPE via the containsKey/get TOCTOU in multicastToUser. The sibling noteSocketMap is guarded by synchronized everywhere; only userSocketMap was left unprotected. Switch userSocketMap to ConcurrentHashMap and make the compound operations atomic: compute() for add, computeIfPresent() for remove, and a single get()+null check for the read paths. Iteration over keySet() is safe under the weakly-consistent iterator. noteSocketMap keeps synchronized because it needs multi-entry atomic operations that a per-key guarantee cannot cover.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this PR for?
ConnectionManager.userSocketMap(user -> Queue<NotebookSocket>) is a plainHashMap, but every access to it is unsynchronized:addUserConnection/removeUserConnection(writes) andmulticastToUser/unicastParagraph/forAllUsers/broadcastNoteListExcept(reads andkeySet()iteration). The sibling fieldnoteSocketMapis guarded bysynchronized (noteSocketMap)at every access — onlyuserSocketMapwas left unprotected.These paths run on different Jetty WebSocket threads (login
onMessage, disconnectonClose, note-list broadcastbroadcastNoteListUpdate). Under concurrent connect/disconnect — e.g. a burst of client reconnects after a server restart — this leads to:ConcurrentModificationExceptionwhile iteratingkeySet(), aborting the note-list broadcast so some users stop receiving updatesHashMapresizeNullPointerExceptionfrom thecontainsKey+getTOCTOU inmulticastToUser/unicastParagraphThe map value is already a
ConcurrentLinkedQueue, so only the map itself was unprotected.Fix: switch
userSocketMaptoConcurrentHashMapand make the compound operations atomic:addUserConnection:compute(...)so the queue create-or-reuse and theaddhappen in one atomic map operation (closing the add-after-remove window that a barecomputeIfAbsent(...).add(...)would leave open)removeUserConnection:computeIfPresent(...), removing the key when the queue becomes emptymulticastToUser/unicastParagraph: a singleget()+ null check, removing the TOCTOU/NPEforAllUsers/broadcastNoteListExcept: unchanged —keySet()iteration is safe underConcurrentHashMap's weakly-consistent iteratornoteSocketMapintentionally keepssynchronized: it needs multi-entry atomic operations (removeConnectionFromAllNote,checkCollaborativeStatus) that a per-keyConcurrentHashMapguarantee does not cover, so a single-strategy migration would not be correct there.What type of PR is it?
Bug Fix
Todos
What is the Jira issue?
How should this be tested?
Added two unit tests in
ConnectionManagerTest:userSocketMapConcurrentAccessTest: 8 writer threads add/remove connections while 2 reader threads iterate viaforAllUsers. On the oldHashMapthis reproducesConcurrentModificationException/NullPointerException; with the fix it passes with no thrown exception.userSocketMapConcurrentAddPreservesAllConnectionsTest: 16 threads concurrently add unique sockets for the same user, then assert every socket is present and the final queue size matches — validates the atomic publishcompute()guarantees.Ran the full
ConnectionManagerTest5× consecutively — stable, no intermittent failures.Note on the add-after-remove window: it is extremely narrow and did not reproduce as a deterministic failing test even under heavy contention (16 churn threads, 16×5000 iterations). The
compute()fix closes it by construction (atomic per-key create-and-add underConcurrentHashMap); the correctness rests on that plus the concurrent-add invariant test rather than a RED→GREEN of the exact interleaving.Screenshots (if appropriate)
Questions: