proxy-io.h: Add Connection disconnect and waitDrained methods - #335
Open
ryanofsky wants to merge 5 commits into
Open
proxy-io.h: Add Connection disconnect and waitDrained methods#335ryanofsky wants to merge 5 commits into
disconnect and waitDrained methods#335ryanofsky wants to merge 5 commits into
Conversation
Split connection teardown out of ~Connection into an idempotent disconnect() method, with the destructor delegating to it. This is a behavior-neutral refactor: the same steps run in the same order on destruction. Having a separate disconnect() method allows severing a connection while keeping the Connection object alive, which the next commits use to let shutdown code wait for in-flight server call bodies to finish after a disconnect (bitcoin/bitcoin#35845). Two details are new: - disconnect() cancels the m_on_disconnect handlers before severing the connection. Previously they were implicitly canceled when the TaskSet member was destroyed. When disconnect() is called separately from destruction, this is required for correctness: severing the stream completes m_network.onDisconnect(), and the registered handlers (_Serve, ConnectStream) destroy the Connection object out from under the caller. - disconnect() explicitly releases m_thread_pool and m_thread_map so worker thread teardown happens at disconnect time whether or not the object is destroyed right away. Previously this happened implicitly during member destruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a per-connection ServerObjectTracker counting live ProxyServer objects, incremented in the ProxyServerBase constructor and decremented in its destructor, with Connection::waitDrained() blocking until the count reaches zero and Connection::pendingServerObjects() exposing it for logging. Disconnecting a connection cancels the KJ promise of an in-flight call, but a C++ server method body already dispatched to a worker thread runs to completion. Counting live server objects turns Cap'n Proto's object lifetime rules into a usable quiescence signal: a ProxyServer object is not destroyed until its outstanding calls finish (the target capability is kept alive for the duration of a call and pinned by post()/PassField via thisCap()), so after disconnect() the count drains to zero exactly when no server call body is still executing. Waiting for that lets shutdown code avoid freeing application state that a still-running call body dereferences (bitcoin/bitcoin#35845). The tracker is held via shared_ptr by the Connection and by every ProxyServer object because objects kept alive by in-flight calls can outlive the Connection on some teardown paths (see ~ProxyServerBase), and their destructors must decrement state that is still valid. It must be declared before m_rpc_system, whose construction creates the bootstrap server object that registers itself with the tracker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a deterministic mptest regression test for bitcoin/bitcoin#35845: hold a server method body in flight on a worker thread, call Connection::disconnect(), and assert that Connection::waitDrained() blocks until the body finishes and its server object is destroyed. Also covers destroying an already-disconnected connection (~Connection noticing disconnect() has run). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ReviewsSee the guideline and AI policy for information on the review process.
If your review is incorrectly listed, please copy-paste |
Contributor
|
Concept ACK |
This was referenced Aug 7, 2026
… the m_incoming_connections list. Currently the list holds Connection by value so the view yields Connection&. When keepconn+notrack later changes the list to list<shared_ptr<Connection>>, the accessor will be updated to return a transform view, so Bitcoin Core code that iterates via this accessor compiles unchanged across that type change. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
CI seems upset? |
Collaborator
Author
|
Updated 39ed2ca -> a40189f ( Added 1 commits a40189f -> 11929f1 ( Updated 11929f1 -> 901a090 ( |
Fix a race between a thread exiting after making IPC calls and a
connection being destroyed by its onDisconnect handler on the event loop
thread. The race was between ~ThreadContext destroying the thread-local
request_threads/callback_threads maps with no locking, and the SetThread
cleanup function (run by Connection::disconnect) erasing entries from
those maps on the event loop thread. When the two ran concurrently, both
could destroy the same ProxyClient<Thread> object: the SetThread cleanup
reset m_disconnect_cb just before ~ProxyClient<Thread> checked it
unsynchronized, so the exiting thread proceeded to destroy the object
while the event loop's map erase destroyed it too. The doubled
destruction consumed m_context.cleanup_fns on one thread, so the other
never unregistered the ProxyClientBase disconnect callback, and
Connection::disconnect then invoked that callback on the freed map node
(heap-use-after-free reading m_client, followed by a double free of the
node reported by glibc as "double free or corruption").
Fix by making map entry removal the synchronization point deciding which
side destroys each ProxyClient<Thread>:
- Add an explicit ~ThreadContext that removes map entries one at a time
under Waiter::m_mutex and destroys each removed node after releasing
the mutex (so ~ProxyClient<Thread> can lock EventLoop::m_mutex without
violating lock order), instead of destroying the maps unlocked.
- Change the SetThread cleanup function to look its entry up by
connection key under Waiter::m_mutex instead of dereferencing the
captured map iterator, extract it, and destroy the node outside the
lock, following the same pattern PassField already uses for mp.Context
arguments. If the entry is gone, the owning thread extracted it first
and is responsible for destroying it.
- Guard the removeSyncCleanup call in ~ProxyClient<Thread> with a
m_context.connection check, because when the entry was extracted by
~ThreadContext first, a concurrent disconnect still runs both the
SetThread cleanup (a no-op now) and the ProxyClientBase disconnect
callback, leaving m_disconnect_cb set but pointing at a spliced-out
list iterator that must not be passed to removeSyncCleanup. The
disconnect callback nulls m_context.connection, and posted functions
cannot interleave with Connection::disconnect on the event loop
thread, so a null connection reliably indicates this case.
The race is long-standing and reachable on master via connections
created by ConnectStream, whose onDisconnect handler deletes the client
Connection on the event loop thread when the peer disconnects while an
exiting thread may be running ~ThreadContext. It was exposed by the
"Waiting for in-flight server call to finish after disconnect" test
because commit bb47369f202b62b8b64f5a52984ff2c40d64ecdd ("Fix error
handling when creating clients") extended the delete-on-disconnect
handler to every ProxyClient created with destroy_connection=true,
including the test setup's directly-created client connection: the
server-side disconnect in the test then deleted the client Connection on
the event loop thread exactly while the test's call thread was exiting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
ConnectionclassdisconnectandwaitDrainedmethods to provide more flexibility when forcibly disconnecting from remote clients or servers.Without these methods, the only way to forcibly close IPC connections is to delete
Connectionobjects. This works but is not ideal because once aConnectionobject is gone, it is difficult to track state still associated with the connection, particularly:ProxyServerobjects that may still be alive because they are executing asynchronous requests made before the disconnect. Without a way to track these objects, there is no generic way to wait for requests to finish existing after disconnecting. So individual IPC interfaces like the Bitcoin mining interface would need to implement custom synchronization to avoid race conditions during shutdown. Followup PR ipc: make ipc::disconnectIncoming wait for in-progress calls to complete bitcoin/bitcoin#35932 builds on this PR, calling the newwaitDrainedmethod introduced here to avoid IPC mining crashes on Bitcoin core shutdown without needing to change the mining code. A unit test is added here simulating these mining crashes.ProxyClientobjects that contain pointers toConnectionobjects. CurrentlyProxyClientobject need to register cleanup handlers withConnectionobjects to deal with Connections being deleted, which consumes memory and complicatesProxyClientshutdown logic. After this change, a followup PR will drop the cleanup handlers soConnectionobjects no longer need to track lists ofProxyClientobjects associated with them. This is implemented in proxy-io: Reference-count Connection objects #336.