Repository navigation
fix(auth): live connections follow the account — event sockets, event streams and terminals close when its access changes (abilityai/trinity-enterprise#863) - #4426
Conversation
…equests (Abilityai/trinity-enterprise#863) `decode_token` is the shared decoder the agent terminal and the system-agent terminal authenticate through. It now applies the #995 account-state check that `get_current_user` applies: a deactivated account (`users.suspended_at` set) gets no identity, and both doors close 4001 before any container lookup. Clearing the column (None or "") admits the account again. A future socket door that uses the decoder inherits the rule. Tests: tests/unit/test_ent863_socket_door_account_state.py drives the real decoder and both real door handlers. Mutation record: with the new check removed, test_a_deactivated_account_gets_no_identity[creator,admin], test_a_deactivated_account_gets_no_terminal[creator,admin] and test_a_deactivated_admin_gets_no_terminal go red; with the check made strict about an empty string, the four reactivated-account controls on "" go red.
… socket decoder (Abilityai/trinity-enterprise#863) The door tests drove the session manager with the decoder passed in, so nothing pinned that the WebSocket route hands it `decode_token`. The new test drives `routers.agents.agent_terminal` itself: a deactivated account closes 4001 before the container lookup. It goes red when the route is given a different decoder, and when the account-state check is removed.
…hanges (Abilityai/trinity-enterprise#863) A non-admin /ws/events socket re-reads its agent list from the key owner's current account on an access change and on the refresh command. Every non-admin /ws and /ws/events agent list is also re-read about once a minute on each worker, and a change that lands while a list is being read is followed by one more read of it.
…bilityai/trinity-enterprise#863) - Every non-admin /ws/events key gets a roster read that re-reads the key's owner on each call (ws_identity_service.event_stream_roster_reader). An owner that is gone, deactivated or Workspace-only reads as an empty list, and the owner's current email is the one used. - The refresh command reads through the same function. - A roster read asked for while one is in flight for that slot is followed by one more read when the in-flight read ends; the refresh command marks the slot the same way. This applies to /ws and /ws/events slots. - StreamDispatcher re-reads every non-admin roster it holds about once a minute (ROSTER_REREAD_INTERVAL_S = 60, a code constant). The dispatcher starts and stops the loop itself; it runs in every worker with no lease (the slots are the process's own), runs its reads one at a time and reads nothing while no non-admin roster is held. Tests: tests/unit/test_ent863_event_stream_roster.py (25 tests); 22 of them fail without this change. Reverting each production hunk on its own turns its named test red: wiring, refresh, owner checks, current email, in-flight re-read, refresh-command re-read, periodic start, one read at a time, disconnected slot, admin slots.
…s change and once a minute (Abilityai/trinity-enterprise#863) integrations.md property (5) describes the two triggers, the event-stream owner re-read and the follow-up read; background-services.md gets a row for the per-worker roster re-read loop (no lease, and why).
… reading it (Abilityai/trinity-enterprise#863) The once-a-minute cycle claimed every non-admin roster up front and then read them one after another. A slot it had not reached yet was already marked as being read, so an access-changing event for that slot only asked for a later read: the slot kept its previous roster until every earlier read in the cycle had finished. The cycle now claims each group just before its own read, so an access change is read for an unreached slot at once, as it is outside a cycle. Reads still run one at a time. Test: test_an_access_change_during_a_periodic_cycle_is_read_at_once blocks the cycle on its first read and fires an unshare; without this change the second slot still holds the unshared agent after 1 s.
…bilityai/trinity-enterprise#863) A slot marked to read again while its read was in flight (an access change, or a refresh command that set a fresh list) was still given that read's result and only corrected by the follow-up read. The in-flight read may predate the change, so its result is now dropped for a slot marked to read again, and only the follow-up read is applied. Tests: test_a_read_that_a_refresh_command_superseded_is_not_applied and test_a_read_that_an_access_change_superseded_is_not_applied hold the follow-up read open and check the list in between; without this change the slot holds the older read's extra agent in that window.
…nected (Abilityai/trinity-enterprise#863) A cycle builds its list of rosters when it starts and reads them one after another, so a slot that disconnected meanwhile was still read. Each group is now checked against the registered slots just before its read. Test: test_a_slot_that_disconnects_before_its_periodic_read_is_not_read disconnects a slot while the cycle is blocked on an earlier read; without this change its resolver still runs.
…ycle claims each slot at its turn (Abilityai/trinity-enterprise#863)
…inity-enterprise#863) The event socket and the event stream close when the account's access changes and reconnect at the new scope. Each worker re-checks its open connections every 15 seconds, so a change made anywhere reaches every worker.
…nterprise#863) The event socket (/ws) and the event stream (/ws/events) close when the account's access changes, and reconnect at the new scope. - services/live_connections.py: a per-worker registry of open connections. A door registers before it reads the account and records a snapshot of the row it read (role, deactivation, email, a digest of the password hash; the key id for /ws/events). One comparison, snapshot against the current row, serves every re-check: gone, deactivated or key inactive closes with 4001; any other difference closes with 4000. - Each worker re-checks its own connections every 15 s (a code constant) with one batched read, and reads nothing when it holds none. It runs in every worker with no leader lease, by design. A failed read keeps the connections open and logs. - notify_account_changed(username) / notify_key_changed(key_id) run the same comparison at once for one account or key, on the serving worker. Both are synchronous and safe from any thread; off the registry's loop they hand over with call_soon_threadsafe. A failed read here closes the reported connections with 4000. - The role route calls notify_account_changed after its write, only when the role changed (a raise closes too). - A closer unregisters its dispatcher slot, then closes the socket; it never cancels the handler's task. A change that lands while a door is opening refuses the connection. - db: get_users_access_state(usernames) and get_keys_active(key_ids), SQLAlchemy Core, no schema change. Tests: tests/unit/test_ent863_live_connections.py (38). Red without the fix: see the mutation record in the slice report.
…count state (Abilityai/trinity-enterprise#863) Each event door reads the account more than once as it opens: `/ws` takes the snapshot and then resolves its scope; `/ws/events` resolves its scope with the email its key read returned and takes the snapshot from the owner read after it. A door now refuses with 4000 (the client reconnects) when those reads disagree on what sets the scope (role, Workspace-only, email), so the scope a socket is served at is always the one its snapshot records and every later re-check compares against. The fleet-reach guard allowlists the new comparison: it reads a role to compare two reads of one person's account and selects no fleet.
…se (Abilityai/trinity-enterprise#863) A re-check now closes the connections it decided on side by side, each still bounded by CLOSE_TIMEOUT_S, instead of one after another. One connection that does not finish closing therefore delays neither the other closes in the same re-check nor the next tick. The test also pins the bound itself: without it the tick would never end.
…t once on every worker (Abilityai/trinity-enterprise#863)
…very worker (Abilityai/trinity-enterprise#863) notify_account_changed and notify_key_changed now also publish one control event through the event bus, scope "control". Each worker's dispatcher hands that event to its live-connection registry, which runs the same targeted re-check against its own read and publishes nothing. A control event never reaches a client: _fanout routes it to the handler and returns, and the visibility rule has no branch for it, so neither live delivery nor a replay can queue it. The handler and the publisher are injected from main.py, so the bus and the registry still import nothing from each other. The per-worker re-check every 15 seconds is unchanged.
…close live connections at once (Abilityai/trinity-enterprise#863) The MCP key revoke and delete routes call notify_key_changed(key_id) after the write: only the event streams opened with that key close, on every worker, whoever owns the key, and the owner's other connections stay open. A password change calls notify_account_changed(username) after the sessions and the optional key revocation are written, so the account's connections close at once on every worker (a revoked key's stream as ended, the rest to reconnect).
… connections through a control event no client receives (Abilityai/trinity-enterprise#863)
…prise#863) Both terminal doors (the agent terminal and the system-agent terminal) register with the worker's live-connection registry in the same step as the account read that authorizes them, and close when the account's access changes: an error frame naming the reason, then 4003. - A shared TerminalAccess helper holds the session's entry. Its closer sends the frame and the close code and cancels the two forwarder tasks the session now owns; it never cancels the handler's own task, so the handler returns through its cleanup (container socket closed, session slot freed, entry released) and the server logs nothing per close. - A close decided while the shell is starting only marks the entry. The door checks once the start has returned, closes the socket it got and never reports the session as started. - The snapshot records the role and email the door authorized on. - A session closed for an access change also ends its shell session in the container, best effort: the shell starts with a per-session marker, and an exec as the terminal's user ends the marked processes and their session. Mutation record (each production hunk reverted on its own; the named tests go red, and green again on restore): closer does not cancel the forwarders (7 red); session not registered with the worker's registry (9); no start-time check, agent door (1) and system door (1); snapshot from the row only (1); no check after a browser read (1); no end of shell (6); end of shell on every close (1); no session marker on the shell (4); closer cancels the handler task (8); forwarders not owned as tasks (7); system door not registered (3); an await between the account read and the registration (1, shape); the end-of-shell script without its session pass (1) or its marker match (1), both run on Linux.
…ore forwarding (Abilityai/trinity-enterprise#863) Two properties the slice relies on had no test that a revert would turn red: a terminal opened by an account with no email keeps its session across a re-check of the unchanged row, and a close that lands after the start is reported but before the forwarders run still ends the session and its shell instead of leaving the forwarders waiting.
…ss passes (Abilityai/trinity-enterprise#863) The end-of-shell script found the processes to end afresh on each pass, and found a session only through a live marked leader. A process in that session that ignored HUP and no longer carried the marker was therefore lost once the HUP had ended the leader, and the script reported the session ended. The sessions found on the first pass are now kept for the later passes, so such a process is still sent KILL. The Linux script test gains that process.
…as survivors (Abilityai/trinity-enterprise#863) The end-of-shell script keeps a terminal's sessions across passes, so its last pass also listed a session member that had already died but was not yet reaped by its parent. It then reported 'still running' and exited 1. On Linux this failed test_the_end_of_shell_script_ends_the_marked_session_and_nothing_else every time: the test's own children stay zombies until it waits. Zombies are now skipped; the same sed|cut read gives state and session.
…863-live-connections
…t863-live-connections # Conflicts: # docs/memory/architecture/security.md
…t863-live-connections # Conflicts: # docs/memory/architecture/security.md # docs/memory/requirements/auth.md # tests/registry.json
|
Resolve by running |
|
Resolve by merging |
Conflict: tests/registry.json (both sides appended entries at the end of test_files); kept every entry from both sides.
…f digesting it (Abilityai/trinity-enterprise#863) The access snapshot only needs to notice that the stored password hash changed. It held a SHA-256 of that hash, which code scanning reads as a password hashed with a fast algorithm (py/weak-sensitive-data-hashing). The input was already the stored hash and the value was never persisted or checked against a password, but a cryptographic digest was the wrong tool for a change detector. A snapshot is compared only with another read made by the same worker process, so Python's per-process salted hash() of the stored hash is enough: it tells two hashes apart and the snapshot still never holds the hash itself. A new test pins both properties.
…63-live-connections
# Conflicts: # tests/registry.json
|
merge-train (2026-11-11): validated READY at Pushed Two non-blocking notes from validation, for a follow-up: the own-email change ( |
#4426 landed the same in-flight roster re-read on dev, so dev's src/backend/services/event_bus.py is taken whole. In integrations.md dev's text already describes the queued re-read; only the share-writer broadcast rule is re-applied on top of it. test_1220_ws_roster_share_refresh.py passes against dev's event bus. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of abilityai/trinity-enterprise#863: the public core. The rest is one change in the enterprise module, a separate PR in abilityai/trinity-enterprise. This PR carries no closing keyword; the issue is closed by hand once both are on
main.Merge order.
dev. It depends on no other open PR.services.live_connections.notify_account_changed, which this PR adds, and that repo's CI checks out publicdev, so its CI stays red until this PR is ondev. Re-run its CI, then merge.src/backend/enterprisegitlink.This branch contains
devatb99c23f6e. The guardrail-events PR for abilityai/trinity-enterprise#165 (#4425, and #4427 stacked on it) touches the sametests/registry.json; whichever merges second keeps both sets of entries. That is the only file where this PR and those two overlap (checked withgit merge-tree; their other conflicts are withdevitself). No migration.What changes
Every live connection is held to the account it was opened for, the same way an HTTP request is: the shared socket decoder applies the account-state checks, an event stream's agent list follows access changes, and the event socket, the event stream and both terminals close when the account's access changes, then reopen at the new scope. Nothing is re-scoped in place.
The shared socket decoder applies the account-state checks
dependencies.decode_token, the decoder behind the agent terminal (/api/agents/{name}/terminal) and the system-agent terminal (/api/system-agent/terminal), applies the account-state checkget_current_userapplies: a token whose account is deactivated (users.suspended_atset) decodes to no identity and the door closes4001. It is the same truthiness test asget_current_user's, right after the missing-row check, so a reactivated account (suspended_atcleared,Noneor"") is admitted again with the same token.An event stream's agent list follows access changes
ws_identity_service.event_stream_roster_reader(key_info)(new) returns a read that re-reads the key's owner on every call. An owner that is gone, deactivated or Workspace-only reads as no agents; otherwise the list is computed with the owner's current email. A failing read keeps the previous list and is logged./ws/events(main.py). Every non-admin key's slot gets that reader as its roster resolver, so an access change (share, unshare and the other roster events the dispatcher already handles) re-reads the list; therefreshcommand reads through the same reader. Creating the reader reads nothing, so no new database read lands beforeaccept().event_bus.StreamDispatcher: a change during a read in flight is followed by one more read, and a read superseded while in flight (by another change or arefresh) is not applied — for/wsand/ws/eventsslots alike. Every 60 s (ROSTER_REREAD_INTERVAL_S) each worker re-reads every non-admin roster it holds, one at a time, claiming each slot just before reading it, so an access event for a slot later in the cycle is read at once. Slots that disconnect during a cycle are skipped. The loop starts and stops with the dispatcher (no new lifespan phase) and runs per worker without a lease, since each worker holds its own sockets. The claim logic is one_claim_roster_reads(), and_refresh_slots(slots, read)is the single entry for any roster read.Event sockets and event streams follow the account
services/live_connections.py(new). One registry per worker holds every open connection with a snapshot of the account taken when the door authorized it: whether the row exists, role, deactivation, email, and a fingerprint of the stored password hash: Python's per-process saltedhash()of it. A snapshot is only compared with another read made by the same worker process, so the fingerprint never leaves the process, and the snapshot never holds the hash itself. The module imports nothing from the event bus.ENDED; a key that is no longer active or no longer exists →ENDED; any other difference from the snapshot →CHANGED(any role change counts, a promotion included); otherwise the connection stays.RECHECK_INTERVAL_S), so a change made by any writer reaches every worker within that bound. Idle, a tick reads nothing; otherwise it is one thread hop and at most two batched, indexed queries per worker, whatever the number of connections.CHANGED, so the clients reconnect and are checked again at the door.CLOSE_TIMEOUT_S(5 s), so one connection slow to close holds up no other close and no tick.finally. A re-check that reaches an entry while it is connecting makes the door refuse (4000) or close at once, so a change that commits while a door is opening is never missed./wsrefuses (4000) when its scope read and its snapshot disagree on admin, Workspace-only or email;/ws/eventsrefuses when the owner row's email differs from the key read's. A socket is therefore always served at the scope its snapshot describes./ws:CHANGED → 4000 "scope changed",ENDED → 4001 "Account access ended"./ws/events: 4000, and 4001 with its existing connect-refusal text.finally, so no close logs an ASGI error.except WebSocketDisconnectbecamefinally(unregister and release on every exit)._start_event_bus(even when the bus fails to start) and stops first on shutdown. No phase added.Changes reach every worker at once
notify_account_changed(username)/notify_key_changed(key_id): sync and safe from any thread. Each re-checks the named account or key at once on this worker, then publishes one control event throughevent_bus.publishat the newSCOPE_CONTROL(Invariant Feature/process engine #10: no direct stream write). The payload names the account or key only:{"recheck": "account", "username": …}or{"recheck": "key", "key_id": …}.StreamDispatcher._fanouthands a control entry to the handlermain.pyinjects (set_control_handler, the same injection pattern as the roster resolver) and returns before any delivery, roster read or event id. Replay already skips an unknown scope, and a test pins that alast-event-idreconnect never receives a control entry. A worker on the released code drops the entry, delivered to no one.live_connections.handle_controlcalls the registry's own re-check, nevernotify_*, so an event is never published again. The publishing worker reads its own event back and re-checks; a connection that has already reconnected matches its snapshot and stays open. A malformed or unknown control event is logged and closes nothing; a handler that raises is logged and delivery carries on; a failed publish still leaves the local close done and raises nothing into the writer.routers/users.py) callsnotify_account_changedwhen the role actually changes.routers/mcp_keys.pyrevoke and delete callnotify_key_changed(key_id)after the write. Keyed on the key, not the caller, so an admin revoking another person's key closes that person's streams; a refused revoke notifies nothing.password_service.change_own_passwordcallsnotify_account_changed(username)after the account's earlier logins are revoked and after any key revocation, so the account's browser socket closes 4000 and the streams of revoked keys close 4001. The person changing their password loses their own open connections too.Terminals follow the account and end their shell
TerminalAccess(services/agent_service/terminal.py), one helper both terminal doors use (routers/system_agent.pyincluded); the doors themselves stay separate. Terminals register in the same per-worker registry, so the 15 s re-check and the cross-worker control event reach them too.{"type":"error","message":…,"close":true}and thenclose(4003, "Access changed" | "Access ended"). The two forwarder tasks the terminal owns are cancelled; the handler task is not, and returns through its ownfinally, which closes the container socket, frees the terminal slot and releases the entry. Input that arrives during the close is not forwarded.api_exec_startreturns, the door checksgo_live()and, if revoked, sends the frame, closes 4003 and returns withoutauth_success.developer) sends HUP, waits, then KILLs every process whose environment carries the marker plus every process sharing thesidof a marked shell, excluding pid 1's and its own. It keeps what it found across its passes, skips zombies (their parent reaps them), and exits non-zero if anything survived the KILL. A failure is logged, never raised. An ordinary close by the browser ends nothing, as today. It runs asdeveloperbecause a container's root withoutCAP_SYS_PTRACEcannot read another user's/proc/<pid>/environ(measured).Docs
requirements/auth.md§2.10;architecture/security.md§1,integrations.md(property 5 and Real-time Delivery),background-services.md(the roster re-read and the re-check loop: per worker, no lease, what runs one at a time),backend.md;feature-flows/websocket-event-bus.md(including the control scope),trinity-connect.md,web-terminal.md("Access changes", error rows) andagent-terminal.md; the migration noteWORKSPACE_ONLY_USER_ROLE_2026-10.md(a role change reaches open event sockets and terminals without a restart); two test docstrings intest_ent467_ws_agent_scope.pyandtest_ent872_fleet_reach.py.Tests
test_ent863_socket_door_account_state.py(14): the realdecode_token, both real door handlers and the realrouters.agents.agent_terminalroute. Each stub row holds a non-ambient email and the controls assert it comes back, proving the stubbed row is the one the decoder read. A deactivated creator and admin get no identity, and each door closes 4001 before any container lookup and registers nothing; reactivated (Noneand"") the identity comes back.test_ent863_event_stream_roster.py(29): the real/ws/eventshandler throughTestClient(a user key's slot gets the reader; after an unshare fan-out it receives only its own agent's event; an admin key gets no reader;refreshafter a role lowering returns the smaller list and after a deactivation none), the reader on its own (each owner state with a control, reactivation, the current email), in-flight and superseded reads, and the periodic loop. Patches go through the dispatcher method's own module globals and each test asserts the patch landed.test_ent863_live_connections.py(43): the realrouters.users.update_user_roleon a real DB; two registries for the cross-worker tick; each changed field (deactivation, row deleted, email, password hash, role) with an unchanged control first; reactivation in both forms; key revoke and delete on real keys; a change injected during the door's read and during its connect, through the real/ws; the registry's thread hand-off; the batched reads; the lifespan shape; a shape test that each door registers before it reads and goes live after it connects; and a snapshot test that two password hashes give different snapshots while no field holds the hash.test_ent863_control_events.py(29): two dispatchers fed one control entry; two workers over one stream through the realEventBus.publish→ read →_fanout, the second closing from the event alone with no tick; replay; the real revoke and delete routes on two registries (only that key's streams close; the owner's/ws, a terminal entry and the other key's stream stay); an admin revoking another person's key; a realchange_own_passwordwith and without key revocation; the echo; malformed events; visibility for every slot kind; a worker with no connections still publishing; and themain.pywiring.test_ent863_terminal_close.py(15): the real door handlers, blocked inreceive()and in the container read — the handler returns within 1 s and is not cancelled, the frame and 4003, the container peer reads EOF, slot and registry empty, no ERROR record; the shell ended with its own marker asdeveloper; an unchanged account keeps its terminal and shell across both re-check paths; deactivation (Access ended); input during the close not forwarded; a change during the shell's start refused on both doors; an account with no email (the built-in admin's shape) keeps its terminal; a failed end-of-shell is logged and the terminal still ends; and the end-of-shell script itself on Linux (/proc), with a marked leader, an unmarked child, anohupchild, a detached marked process, another terminal's process and a bystander. macOS: 14 passed, 1 skipped (the Linux script test); in a Linux container (Python 3.13, dash) the whole file passes.Validation: each area was re-run on
devat13d32e22fbefore the areas were merged into this branch, with the same commands as on the base it was reviewed on and the same results there: the area's files plus the pinned guards — 1140 passed, 9 skipped (decoder); 1155 and 9 (rosters); 1197 and 9 (registry); 1226 and 9 (cross-worker); 1211 and 10 (terminals, the extra skip being the Linux-only script test);dev's new or changed tests over the touched files, 1025 passed; pytest-randomly seeds 1, 2 and 3 over the ent863 files,test_event_busand ent467, 160 passed (with the cross-worker file) and 145 passed, 1 skipped (with the terminal file) each. The pinned guards include 1310 ×2, 1816 adoption and convergence, 186, 2322, 293, 3825, 837 floor and everytest_837_*, ent79, ent858, ent467, ent872, connector auth, 1028 lifespan, 1483 ×2, 1854, 2338, 2996 ×2, 3109, 595, ws_ticket, mcp_key, models-centralized, users-db dual backend and websocket auth; the otherdecode_tokenusers (2322, 3185, ent1107, ent79, ent872) passed 252. The merges into this branch conflicted only inarchitecture/security.md,requirements/auth.mdandtests/registry.json; the two live-connection doc bullets were folded into one carrying both the cross-worker and the terminal changes.After merging
devat97a86f1dc(one conflict,tests/registry.json: both sides appended entries, all kept; 776 entries, no duplicates) and the password-fingerprint change: the ent863 files,test_event_bus*, ent467 and the pinned guards above plusdev's new tests on the touched files (ent1148, ent629, ent1174, ent1202, 293, 2996, 1890, ent435, models-centralized) — 1783 passed, 11 skipped; pytest-randomly seeds 12345, 67890 and 99999 over the ent863 files andtest_event_bus— 152 passed, 1 skipped each;lint_sys_modules,lint_root_test_placementand the Alembic single-head check pass.This branch then merged
devatb99c23f6e(#4447:httpcorepinned so asys.modulespatch cannot split it), which fixes the order-dependenttest_ent926_runtime_turn_scope::test_an_unreachable_platform_is_no_answerfailure thattest_1484_create_agent_characterization.pycaused when run before it. Replayed in that order with no randomization, plustest_4439_httpcore_pinned.py: 104 passed; the ent863 files andtest_event_bus: 152 passed, 1 skipped.Mutation: each fix reverted or mutated on its own, from a scratch copy restored byte-identical, turns named tests red:
decode_token, 6 go red —test_a_deactivated_account_gets_no_identity[creator,admin],test_a_deactivated_account_gets_no_terminal[creator,admin],test_a_deactivated_admin_gets_no_terminaland the route test that drivesrouters.agents.agent_terminal. Made strict (is not None), the four""reactivation controls go red; with the route handing the manager a different decoder, the route test goes red.refreshreusing the connect-time owner row (2); the owner checks reduced (7); the connect-time email (2); the in-flight follow-up dropped (2);refreshnot marking the in-flight read (1); the superseded read applied (test_a_read_that_a_refresh_command_superseded_is_not_applied,test_a_read_that_an_access_change_superseded_is_not_applied); the periodic loop not started (4); concurrent periodic reads (1); slots claimed up front for the whole cycle (test_an_access_change_during_a_periodic_cycle_is_read_at_once); a disconnected slot read (1); admin slots read (1).test_a_role_change_through_the_route_closes_that_accounts_connections_at_once,test_a_promotion_closes_too_and_a_same_role_write_closes_nothing); the comparison dropped or the snapshot read at tick time (13 each); deactivation or key state dropped from the comparison; a failed tick read closing (test_a_failed_tick_read_keeps_every_connection_open_and_the_next_tick_retries); a failed immediate read keeping connections open (test_a_failed_read_after_a_reported_change_closes_that_accounts_connections); notify running on the calling thread (test_notify_from_a_plain_thread_closes_on_the_registrys_loop);/wsregistering after it reads (5); the scope/snapshot check removed (test_a_browser_socket_is_refused_when_its_scope_read_disagrees_with_its_snapshot,test_an_event_stream_is_refused_when_its_owner_email_differs_from_its_key_read); closes run one after another or without the timeout (test_a_close_that_hangs_delays_neither_the_other_closes_nor_the_tick); the loop not started in_start_event_bus; the password fingerprint constant (test_a_snapshot_tells_password_hashes_apart_without_holding_oneand the password-hash case oftest_the_tick_closes_a_connection_whose_account_changed); the snapshot holding the hash itself (test_a_snapshot_tells_password_hashes_apart_without_holding_one).test_an_account_change_rechecks_here_and_publishes_one_control_event,test_a_key_change_rechecks_here_and_publishes_one_key_scoped_control_event);handle_controlcallingnotify_*(test_a_control_event_rechecks_locally_and_is_never_published_again);_fanoutnot routing control, or routing it and carrying on (test_a_control_entry_goes_to_the_handler_and_reaches_no_client,test_a_control_entry_does_not_trigger_a_roster_read); the revoke or delete route not notifying, or notifying the caller's account (test_revoking_or_deleting_a_key_closes_only_its_streams_on_every_worker); the password change not notifying (test_a_password_change_closes_the_accounts_connections_at_once_on_every_worker);main.pywiring no handler or publishing atSCOPE_ALL; visibility admitting control for admins (test_no_slot_of_any_kind_may_see_a_control_entry); replay delivering control (test_a_replay_never_delivers_a_control_entry_nor_reruns_it).test_the_snapshot_holds_what_the_door_authorized); no revoked check after a browser read (test_input_that_arrives_while_the_terminal_closes_is_not_forwarded); no end of shell (6, includingtest_a_terminal_closed_for_an_access_change_ends_its_shell); the shell ended on every close (test_an_unchanged_account_keeps_its_terminal_and_its_shell); no marker (4); the closer cancelling the handler task (8); anawaitbetween the read and the registration (the shape test); the no-emailor ""dropped (test_an_account_with_no_email_keeps_its_terminal_across_a_recheck); and, on Linux, the end-of-shell script without its sid pass, its marker match, its memory across passes, or counting zombies as survivors (the end-of-shell script test).Unverified: the browser's reconnect at the new scope within about 5 s after a 4000 (frontend behaviour, read in
websocket.js, not driven in a browser); a bare-metal Linux Docker host (the container checks ran on Docker Desktop's Linux VM kernel); Python 3.13 and the full unit suite (CI). The live run below predatesdev's 67 commits since and the password-fingerprint change; no code conflicted with them.Verified on a live stack
Production images built from one tree holding these changes together with the public ent#165 and #4281 changes on the
devof that time, on the prod compose stack (PostgreSQL 16, Redis with ACL users, two backend workers, each socket mapped to the worker serving it), with real agent containers:4001on the agent terminal and the system-agent terminal; clearing the column admitted the same token again./ws/events, after an unshare through the API no further events for that agent arrived (checked at 2 s and 65 s) while the socket stayed open, andrefreshreturned the smaller list; after a database-only share removalrefreshreturned the smaller list too./wswith 4000 on both workers within 0.01 s; a database-only role change closed it by the tick; a database-only deactivation closed/wsand/ws/eventswith 4001 on both workers within 13.2 s, and a new ticket was refused 401; a key owner's role change closed/ws/eventswith 4000. Revoking key 1 closed only key 1's streams with 4001 at once, while key 2's streams on both workers and the browser socket stayed open; deleting key 2 closed its streams on both workers; an admin revoking another person's key closed that person's stream; a password change with key revocation closed the streams 4001 on both workers and the browser socket 4000, in 0.51 s. The stored control entries carry onlyrecheckplus a username or key id; alast-event-idreplay delivered none of them. NoException in ASGI applicationon any close.nohupone among them, were gone within 1.27 s; a second (admin) terminal and the agent server survived; no zombies left. Deactivation on the system-agent terminal: "Your access to this terminal has ended." and 4003 in 1.5 s, shell gone. One terminal on each worker: both closed within 0.02 s. A busy container (307 processes): ended in 1.83 s, inside the 10 s bound. One "Ended the terminal shell" line per close, no ASGI errors; an ordinary close left the shell running, as designed. After the zombie fix reached a rebuilt image, the role-change case again: 4003 in 0.02 s, all three processes gone within 1.27 s.Unfiled follow-ups
decode_tokenand access-check snippets quoted in the two terminal feature flows (doc only).requirements/auth.md§2.10 terminal bullet says other workers close a terminal within about 15 s — with the control event, an account change reaches every worker's terminals at once (measured above); align the text.StreamDispatcher.stop()does not cancel or await in-flight roster reads (event-triggered ones; predates this change).refreshcommand's database read off the event loop (predates this change).revokedcheck in the two event doors with a test (nit).finally— on both doors; read in the code, not run; predates this change.Journey Impact
Journey Impact: none: open sockets, event streams and terminals close and reconnect at the new scope when the account changes; no journey step changes