Skip to content

fix(auth): live connections follow the account — event sockets, event streams and terminals close when its access changes (abilityai/trinity-enterprise#863) - #4426

Merged
vybe merged 47 commits into
devfrom
AndriiPasternak31/ent863-live-connections
Nov 11, 2026
Merged

vybe merged 47 commits into
devfrom
AndriiPasternak31/ent863-live-connections

Conversation

@AndriiPasternak31

@AndriiPasternak31 AndriiPasternak31 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Part of abilityai/trinity-enterprise#863: the public core. The rest is one change in the enterprise module, a separate PR in abilityai/trinity-enterprise. This PR carries no closing keyword; the issue is closed by hand once both are on main.

Merge order.

  1. This PR, to dev. It depends on no other open PR.
  2. The enterprise-module PR: it calls services.live_connections.notify_account_changed, which this PR adds, and that repo's CI checks out public dev, so its CI stays red until this PR is on dev. Re-run its CI, then merge.
  3. A public PR that bumps the src/backend/enterprise gitlink.

This branch contains dev at b99c23f6e. The guardrail-events PR for abilityai/trinity-enterprise#165 (#4425, and #4427 stacked on it) touches the same tests/registry.json; whichever merges second keeps both sets of entries. That is the only file where this PR and those two overlap (checked with git merge-tree; their other conflicts are with dev itself). No migration.

What changes

Every live connection is held to the account it was opened for, the same way an HTTP request is: the shared socket decoder applies the account-state checks, an event stream's agent list follows access changes, and the event socket, the event stream and both terminals close when the account's access changes, then reopen at the new scope. Nothing is re-scoped in place.

The shared socket decoder applies the account-state checks

  • dependencies.decode_token, the decoder behind the agent terminal (/api/agents/{name}/terminal) and the system-agent terminal (/api/system-agent/terminal), applies the account-state check get_current_user applies: a token whose account is deactivated (users.suspended_at set) decodes to no identity and the door closes 4001. It is the same truthiness test as get_current_user's, right after the missing-row check, so a reactivated account (suspended_at cleared, None or "") is admitted again with the same token.
  • One fix in the decoder covers both terminal doors. The decoder accepts JWTs only, so agent, system, MCP and portal-delegate keys are unaffected.

An event stream's agent list follows access changes

  • ws_identity_service.event_stream_roster_reader(key_info) (new) returns a read that re-reads the key's owner on every call. An owner that is gone, deactivated or Workspace-only reads as no agents; otherwise the list is computed with the owner's current email. A failing read keeps the previous list and is logged.
  • /ws/events (main.py). Every non-admin key's slot gets that reader as its roster resolver, so an access change (share, unshare and the other roster events the dispatcher already handles) re-reads the list; the refresh command reads through the same reader. Creating the reader reads nothing, so no new database read lands before accept().
  • event_bus.StreamDispatcher: a change during a read in flight is followed by one more read, and a read superseded while in flight (by another change or a refresh) is not applied — for /ws and /ws/events slots alike. Every 60 s (ROSTER_REREAD_INTERVAL_S) each worker re-reads every non-admin roster it holds, one at a time, claiming each slot just before reading it, so an access event for a slot later in the cycle is read at once. Slots that disconnect during a cycle are skipped. The loop starts and stops with the dispatcher (no new lifespan phase) and runs per worker without a lease, since each worker holds its own sockets. The claim logic is one _claim_roster_reads(), and _refresh_slots(slots, read) is the single entry for any roster read.

Event sockets and event streams follow the account

  • services/live_connections.py (new). One registry per worker holds every open connection with a snapshot of the account taken when the door authorized it: whether the row exists, role, deactivation, email, and a fingerprint of the stored password hash: Python's per-process salted hash() of it. A snapshot is only compared with another read made by the same worker process, so the fingerprint never leaves the process, and the snapshot never holds the hash itself. The module imports nothing from the event bus.
    • Comparison: a missing row or a deactivated account → ENDED; a key that is no longer active or no longer exists → ENDED; any other difference from the snapshot → CHANGED (any role change counts, a promotion included); otherwise the connection stays.
    • Every worker re-checks its open connections every 15 s (RECHECK_INTERVAL_S), so a change made by any writer reaches every worker within that bound. Idle, a tick reads nothing; otherwise it is one thread hop and at most two batched, indexed queries per worker, whatever the number of connections.
    • A failed read on the tick keeps every connection open and logs an ERROR; the next tick retries. A failed read on the immediate path closes that account's connections with CHANGED, so the clients reconnect and are checked again at the door.
    • Closes run side by side, each bounded by CLOSE_TIMEOUT_S (5 s), so one connection slow to close holds up no other close and no tick.
  • Door protocol, used by both event doors: register a provisional entry, read the account, activate with the snapshot, connect, go live, release in finally. A re-check that reaches an entry while it is connecting makes the door refuse (4000) or close at once, so a change that commits while a door is opening is never missed. /ws refuses (4000) when its scope read and its snapshot disagree on admin, Workspace-only or email; /ws/events refuses when the owner row's email differs from the key read's. A socket is therefore always served at the scope its snapshot describes.
  • Close codes. /ws: CHANGED → 4000 "scope changed", ENDED → 4001 "Account access ended". /ws/events: 4000, and 4001 with its existing connect-refusal text.
  • Handlers. The closer first unregisters the dispatcher slot and then closes the socket; the handler task is never cancelled and returns through its own finally, so no close logs an ASGI error. except WebSocketDisconnect became finally (unregister and release on every exit).
  • Lifespan. The re-check loop starts at the end of _start_event_bus (even when the bus fails to start) and stops first on shutdown. No phase added.

Changes reach every worker at once

  • notify_account_changed(username) / notify_key_changed(key_id): sync and safe from any thread. Each re-checks the named account or key at once on this worker, then publishes one control event through event_bus.publish at the new SCOPE_CONTROL (Invariant Feature/process engine #10: no direct stream write). The payload names the account or key only: {"recheck": "account", "username": …} or {"recheck": "key", "key_id": …}.
  • The dispatcher routes it and no client ever receives it. StreamDispatcher._fanout hands a control entry to the handler main.py injects (set_control_handler, the same injection pattern as the roster resolver) and returns before any delivery, roster read or event id. Replay already skips an unknown scope, and a test pins that a last-event-id reconnect never receives a control entry. A worker on the released code drops the entry, delivered to no one.
  • The receiving side is local only. live_connections.handle_control calls the registry's own re-check, never notify_*, so an event is never published again. The publishing worker reads its own event back and re-checks; a connection that has already reconnected matches its snapshot and stays open. A malformed or unknown control event is logged and closes nothing; a handler that raises is logged and delivery carries on; a failed publish still leaves the local close done and raises nothing into the writer.
  • Triggers.
    • The role route (routers/users.py) calls notify_account_changed when the role actually changes.
    • routers/mcp_keys.py revoke and delete call notify_key_changed(key_id) after the write. Keyed on the key, not the caller, so an admin revoking another person's key closes that person's streams; a refused revoke notifies nothing.
    • password_service.change_own_password calls notify_account_changed(username) after the account's earlier logins are revoked and after any key revocation, so the account's browser socket closes 4000 and the streams of revoked keys close 4001. The person changing their password loses their own open connections too.

Terminals follow the account and end their shell

  • TerminalAccess (services/agent_service/terminal.py), one helper both terminal doors use (routers/system_agent.py included); the doors themselves stay separate. Terminals register in the same per-worker registry, so the 15 s re-check and the cross-worker control event reach them too.
    • Each door registers right after its access checks pass, in the same synchronous step as the token read. The snapshot records the role and email the door actually authorized on, so a change committed between the decoder's read and the snapshot read is a difference, never the opening state.
    • Close: the browser gets {"type":"error","message":…,"close":true} and then close(4003, "Access changed" | "Access ended"). The two forwarder tasks the terminal owns are cancelled; the handler task is not, and returns through its own finally, which closes the container socket, frees the terminal slot and releases the entry. Input that arrives during the close is not forwarded.
    • Close during setup: a not-yet-live entry is only marked; after api_exec_start returns, the door checks go_live() and, if revoked, sends the frame, closes 4003 and returns without auth_success.
    • Ending the shell. The shell starts with an environment marker unique to that terminal. On a close for an access change (only then), one bounded exec (10 s, run as developer) sends HUP, waits, then KILLs every process whose environment carries the marker plus every process sharing the sid of a marked shell, excluding pid 1's and its own. It keeps what it found across its passes, skips zombies (their parent reaps them), and exits non-zero if anything survived the KILL. A failure is logged, never raised. An ordinary close by the browser ends nothing, as today. It runs as developer because a container's root without CAP_SYS_PTRACE cannot read another user's /proc/<pid>/environ (measured).

Docs

requirements/auth.md §2.10; architecture/security.md §1, integrations.md (property 5 and Real-time Delivery), background-services.md (the roster re-read and the re-check loop: per worker, no lease, what runs one at a time), backend.md; feature-flows/websocket-event-bus.md (including the control scope), trinity-connect.md, web-terminal.md ("Access changes", error rows) and agent-terminal.md; the migration note WORKSPACE_ONLY_USER_ROLE_2026-10.md (a role change reaches open event sockets and terminals without a restart); two test docstrings in test_ent467_ws_agent_scope.py and test_ent872_fleet_reach.py.

Tests

  • New test_ent863_socket_door_account_state.py (14): the real decode_token, both real door handlers and the real routers.agents.agent_terminal route. Each stub row holds a non-ambient email and the controls assert it comes back, proving the stubbed row is the one the decoder read. A deactivated creator and admin get no identity, and each door closes 4001 before any container lookup and registers nothing; reactivated (None and "") the identity comes back.
  • New test_ent863_event_stream_roster.py (29): the real /ws/events handler through TestClient (a user key's slot gets the reader; after an unshare fan-out it receives only its own agent's event; an admin key gets no reader; refresh after a role lowering returns the smaller list and after a deactivation none), the reader on its own (each owner state with a control, reactivation, the current email), in-flight and superseded reads, and the periodic loop. Patches go through the dispatcher method's own module globals and each test asserts the patch landed.
  • New test_ent863_live_connections.py (43): the real routers.users.update_user_role on a real DB; two registries for the cross-worker tick; each changed field (deactivation, row deleted, email, password hash, role) with an unchanged control first; reactivation in both forms; key revoke and delete on real keys; a change injected during the door's read and during its connect, through the real /ws; the registry's thread hand-off; the batched reads; the lifespan shape; a shape test that each door registers before it reads and goes live after it connects; and a snapshot test that two password hashes give different snapshots while no field holds the hash.
  • New test_ent863_control_events.py (29): two dispatchers fed one control entry; two workers over one stream through the real EventBus.publish → read → _fanout, the second closing from the event alone with no tick; replay; the real revoke and delete routes on two registries (only that key's streams close; the owner's /ws, a terminal entry and the other key's stream stay); an admin revoking another person's key; a real change_own_password with and without key revocation; the echo; malformed events; visibility for every slot kind; a worker with no connections still publishing; and the main.py wiring.
  • New test_ent863_terminal_close.py (15): the real door handlers, blocked in receive() and in the container read — the handler returns within 1 s and is not cancelled, the frame and 4003, the container peer reads EOF, slot and registry empty, no ERROR record; the shell ended with its own marker as developer; an unchanged account keeps its terminal and shell across both re-check paths; deactivation (Access ended); input during the close not forwarded; a change during the shell's start refused on both doors; an account with no email (the built-in admin's shape) keeps its terminal; a failed end-of-shell is logged and the terminal still ends; and the end-of-shell script itself on Linux (/proc), with a marked leader, an unmarked child, a nohup child, a detached marked process, another terminal's process and a bystander. macOS: 14 passed, 1 skipped (the Linux script test); in a Linux container (Python 3.13, dash) the whole file passes.

Validation: each area was re-run on dev at 13d32e22f before the areas were merged into this branch, with the same commands as on the base it was reviewed on and the same results there: the area's files plus the pinned guards — 1140 passed, 9 skipped (decoder); 1155 and 9 (rosters); 1197 and 9 (registry); 1226 and 9 (cross-worker); 1211 and 10 (terminals, the extra skip being the Linux-only script test); dev's new or changed tests over the touched files, 1025 passed; pytest-randomly seeds 1, 2 and 3 over the ent863 files, test_event_bus and ent467, 160 passed (with the cross-worker file) and 145 passed, 1 skipped (with the terminal file) each. The pinned guards include 1310 ×2, 1816 adoption and convergence, 186, 2322, 293, 3825, 837 floor and every test_837_*, ent79, ent858, ent467, ent872, connector auth, 1028 lifespan, 1483 ×2, 1854, 2338, 2996 ×2, 3109, 595, ws_ticket, mcp_key, models-centralized, users-db dual backend and websocket auth; the other decode_token users (2322, 3185, ent1107, ent79, ent872) passed 252. The merges into this branch conflicted only in architecture/security.md, requirements/auth.md and tests/registry.json; the two live-connection doc bullets were folded into one carrying both the cross-worker and the terminal changes.

After merging dev at 97a86f1dc (one conflict, tests/registry.json: both sides appended entries, all kept; 776 entries, no duplicates) and the password-fingerprint change: the ent863 files, test_event_bus*, ent467 and the pinned guards above plus dev's new tests on the touched files (ent1148, ent629, ent1174, ent1202, 293, 2996, 1890, ent435, models-centralized) — 1783 passed, 11 skipped; pytest-randomly seeds 12345, 67890 and 99999 over the ent863 files and test_event_bus — 152 passed, 1 skipped each; lint_sys_modules, lint_root_test_placement and the Alembic single-head check pass.

This branch then merged dev at b99c23f6e (#4447: httpcore pinned so a sys.modules patch cannot split it), which fixes the order-dependent test_ent926_runtime_turn_scope::test_an_unreachable_platform_is_no_answer failure that test_1484_create_agent_characterization.py caused when run before it. Replayed in that order with no randomization, plus test_4439_httpcore_pinned.py: 104 passed; the ent863 files and test_event_bus: 152 passed, 1 skipped.

Mutation: each fix reverted or mutated on its own, from a scratch copy restored byte-identical, turns named tests red:

  • Decoder: with the check removed from decode_token, 6 go red — test_a_deactivated_account_gets_no_identity[creator,admin], test_a_deactivated_account_gets_no_terminal[creator,admin], test_a_deactivated_admin_gets_no_terminal and the route test that drives routers.agents.agent_terminal. Made strict (is not None), the four "" reactivation controls go red; with the route handing the manager a different decoder, the route test goes red.
  • Rosters: resolver only for agent keys (the unshare endpoint test); refresh reusing the connect-time owner row (2); the owner checks reduced (7); the connect-time email (2); the in-flight follow-up dropped (2); refresh not marking the in-flight read (1); the superseded read applied (test_a_read_that_a_refresh_command_superseded_is_not_applied, test_a_read_that_an_access_change_superseded_is_not_applied); the periodic loop not started (4); concurrent periodic reads (1); slots claimed up front for the whole cycle (test_an_access_change_during_a_periodic_cycle_is_read_at_once); a disconnected slot read (1); admin slots read (1).
  • Registry: the role route not notifying (test_a_role_change_through_the_route_closes_that_accounts_connections_at_once, test_a_promotion_closes_too_and_a_same_role_write_closes_nothing); the comparison dropped or the snapshot read at tick time (13 each); deactivation or key state dropped from the comparison; a failed tick read closing (test_a_failed_tick_read_keeps_every_connection_open_and_the_next_tick_retries); a failed immediate read keeping connections open (test_a_failed_read_after_a_reported_change_closes_that_accounts_connections); notify running on the calling thread (test_notify_from_a_plain_thread_closes_on_the_registrys_loop); /ws registering after it reads (5); the scope/snapshot check removed (test_a_browser_socket_is_refused_when_its_scope_read_disagrees_with_its_snapshot, test_an_event_stream_is_refused_when_its_owner_email_differs_from_its_key_read); closes run one after another or without the timeout (test_a_close_that_hangs_delays_neither_the_other_closes_nor_the_tick); the loop not started in _start_event_bus; the password fingerprint constant (test_a_snapshot_tells_password_hashes_apart_without_holding_one and the password-hash case of test_the_tick_closes_a_connection_whose_account_changed); the snapshot holding the hash itself (test_a_snapshot_tells_password_hashes_apart_without_holding_one).
  • Cross-worker: the account or key notify not publishing (test_an_account_change_rechecks_here_and_publishes_one_control_event, test_a_key_change_rechecks_here_and_publishes_one_key_scoped_control_event); handle_control calling notify_* (test_a_control_event_rechecks_locally_and_is_never_published_again); _fanout not routing control, or routing it and carrying on (test_a_control_entry_goes_to_the_handler_and_reaches_no_client, test_a_control_entry_does_not_trigger_a_roster_read); the revoke or delete route not notifying, or notifying the caller's account (test_revoking_or_deleting_a_key_closes_only_its_streams_on_every_worker); the password change not notifying (test_a_password_change_closes_the_accounts_connections_at_once_on_every_worker); main.py wiring no handler or publishing at SCOPE_ALL; visibility admitting control for admins (test_no_slot_of_any_kind_may_see_a_control_entry); replay delivering control (test_a_replay_never_delivers_a_control_entry_nor_reruns_it).
  • Terminals: the closer not cancelling the forwarders (7); the doors not registered (9); no start-time check on either door (the two "change while the shell starts" tests); the snapshot taken from the row only (test_the_snapshot_holds_what_the_door_authorized); no revoked check after a browser read (test_input_that_arrives_while_the_terminal_closes_is_not_forwarded); no end of shell (6, including test_a_terminal_closed_for_an_access_change_ends_its_shell); the shell ended on every close (test_an_unchanged_account_keeps_its_terminal_and_its_shell); no marker (4); the closer cancelling the handler task (8); an await between the read and the registration (the shape test); the no-email or "" dropped (test_an_account_with_no_email_keeps_its_terminal_across_a_recheck); and, on Linux, the end-of-shell script without its sid pass, its marker match, its memory across passes, or counting zombies as survivors (the end-of-shell script test).

Unverified: the browser's reconnect at the new scope within about 5 s after a 4000 (frontend behaviour, read in websocket.js, not driven in a browser); a bare-metal Linux Docker host (the container checks ran on Docker Desktop's Linux VM kernel); Python 3.13 and the full unit suite (CI). The live run below predates dev's 67 commits since and the password-fingerprint change; no code conflicted with them.

Verified on a live stack

Production images built from one tree holding these changes together with the public ent#165 and #4281 changes on the dev of that time, on the prod compose stack (PostgreSQL 16, Redis with ACL users, two backend workers, each socket mapped to the worker serving it), with real agent containers:

  • Decoder: the token of a deactivated owner and of the deactivated built-in admin (set in the database) was refused with 4001 on the agent terminal and the system-agent terminal; clearing the column admitted the same token again.
  • Rosters: with a user key on /ws/events, after an unshare through the API no further events for that agent arrived (checked at 2 s and 65 s) while the socket stayed open, and refresh returned the smaller list; after a database-only share removal refresh returned the smaller list too.
  • Sockets and streams: a role change through the API closed /ws with 4000 on both workers within 0.01 s; a database-only role change closed it by the tick; a database-only deactivation closed /ws and /ws/events with 4001 on both workers within 13.2 s, and a new ticket was refused 401; a key owner's role change closed /ws/events with 4000. Revoking key 1 closed only key 1's streams with 4001 at once, while key 2's streams on both workers and the browser socket stayed open; deleting key 2 closed its streams on both workers; an admin revoking another person's key closed that person's stream; a password change with key revocation closed the streams 4001 on both workers and the browser socket 4000, in 0.51 s. The stored control entries carry only recheck plus a username or key id; a last-event-id replay delivered none of them. No Exception in ASGI application on any close.
  • Terminals: a role change from an admin: "Your access changed…" and 4003 at once; three background processes the shell had started, a nohup one among them, were gone within 1.27 s; a second (admin) terminal and the agent server survived; no zombies left. Deactivation on the system-agent terminal: "Your access to this terminal has ended." and 4003 in 1.5 s, shell gone. One terminal on each worker: both closed within 0.02 s. A busy container (307 processes): ended in 1.83 s, inside the 10 s bound. One "Ended the terminal shell" line per close, no ASGI errors; an ordinary close left the shell running, as designed. After the zombie fix reached a rebuilt image, the role-change case again: 4003 in 0.02 s, all three processes gone within 1.27 s.

Unfiled follow-ups

  • The terminal owner check passes an email where a username is expected — an owner whose username differs from their email is refused their own terminal (fails closed).
  • Refresh the decode_token and access-check snippets quoted in the two terminal feature flows (doc only).
  • The requirements/auth.md §2.10 terminal bullet says other workers close a terminal within about 15 s — with the control event, an account change reaches every worker's terminals at once (measured above); align the text.
  • StreamDispatcher.stop() does not cancel or await in-flight roster reads (event-triggered ones; predates this change).
  • Roster reads are per slot, not per key or owner, on each access event — many MCP sockets mean a burst into the default executor; not measured.
  • Move the refresh command's database read off the event loop (predates this change).
  • Pin the post-receive revoked check in the two event doors with a test (nit).
  • An immediate trigger for agent-key deactivations — those writers return no key ids, so such streams close on the 15 s re-check rather than at once.
  • End the shell on an ordinary terminal close too — it could reuse the same end-of-shell helper.
  • A refused terminal open (4002, terminal limit) frees the other, live terminal's slot in its finally — on both doors; read in the code, not run; predates this change.
  • The end-of-shell script's "still running" message can list a pid twice (cosmetic).

Journey Impact

Journey Impact: none: open sockets, event streams and terminals close and reconnect at the new scope when the account changes; no journey step changes

…equests (Abilityai/trinity-enterprise#863)

`decode_token` is the shared decoder the agent terminal and the
system-agent terminal authenticate through. It now applies the #995
account-state check that `get_current_user` applies: a deactivated
account (`users.suspended_at` set) gets no identity, and both doors
close 4001 before any container lookup. Clearing the column (None or "")
admits the account again. A future socket door that uses the decoder
inherits the rule.

Tests: tests/unit/test_ent863_socket_door_account_state.py drives the
real decoder and both real door handlers. Mutation record: with the new
check removed, test_a_deactivated_account_gets_no_identity[creator,admin],
test_a_deactivated_account_gets_no_terminal[creator,admin] and
test_a_deactivated_admin_gets_no_terminal go red; with the check made
strict about an empty string, the four reactivated-account controls on
"" go red.
… socket decoder (Abilityai/trinity-enterprise#863)

The door tests drove the session manager with the decoder passed in, so
nothing pinned that the WebSocket route hands it `decode_token`. The new
test drives `routers.agents.agent_terminal` itself: a deactivated account
closes 4001 before the container lookup. It goes red when the route is
given a different decoder, and when the account-state check is removed.
…hanges (Abilityai/trinity-enterprise#863)

A non-admin /ws/events socket re-reads its agent list from the key owner's
current account on an access change and on the refresh command. Every
non-admin /ws and /ws/events agent list is also re-read about once a
minute on each worker, and a change that lands while a list is being read
is followed by one more read of it.
…bilityai/trinity-enterprise#863)

- Every non-admin /ws/events key gets a roster read that re-reads the key's
  owner on each call (ws_identity_service.event_stream_roster_reader). An
  owner that is gone, deactivated or Workspace-only reads as an empty list,
  and the owner's current email is the one used.
- The refresh command reads through the same function.
- A roster read asked for while one is in flight for that slot is followed
  by one more read when the in-flight read ends; the refresh command marks
  the slot the same way. This applies to /ws and /ws/events slots.
- StreamDispatcher re-reads every non-admin roster it holds about once a
  minute (ROSTER_REREAD_INTERVAL_S = 60, a code constant). The dispatcher
  starts and stops the loop itself; it runs in every worker with no lease
  (the slots are the process's own), runs its reads one at a time and reads
  nothing while no non-admin roster is held.

Tests: tests/unit/test_ent863_event_stream_roster.py (25 tests); 22 of
them fail without this change. Reverting each production hunk on its own
turns its named test red: wiring, refresh, owner checks, current email,
in-flight re-read, refresh-command re-read, periodic start, one read at
a time, disconnected slot, admin slots.
…s change and once a minute (Abilityai/trinity-enterprise#863)

integrations.md property (5) describes the two triggers, the event-stream
owner re-read and the follow-up read; background-services.md gets a row for
the per-worker roster re-read loop (no lease, and why).
… reading it (Abilityai/trinity-enterprise#863)

The once-a-minute cycle claimed every non-admin roster up front and then
read them one after another. A slot it had not reached yet was already
marked as being read, so an access-changing event for that slot only
asked for a later read: the slot kept its previous roster until every
earlier read in the cycle had finished. The cycle now claims each group
just before its own read, so an access change is read for an unreached
slot at once, as it is outside a cycle. Reads still run one at a time.

Test: test_an_access_change_during_a_periodic_cycle_is_read_at_once
blocks the cycle on its first read and fires an unshare; without this
change the second slot still holds the unshared agent after 1 s.
…bilityai/trinity-enterprise#863)

A slot marked to read again while its read was in flight (an access
change, or a refresh command that set a fresh list) was still given that
read's result and only corrected by the follow-up read. The in-flight
read may predate the change, so its result is now dropped for a slot
marked to read again, and only the follow-up read is applied.

Tests: test_a_read_that_a_refresh_command_superseded_is_not_applied and
test_a_read_that_an_access_change_superseded_is_not_applied hold the
follow-up read open and check the list in between; without this change
the slot holds the older read's extra agent in that window.
…nected (Abilityai/trinity-enterprise#863)

A cycle builds its list of rosters when it starts and reads them one
after another, so a slot that disconnected meanwhile was still read.
Each group is now checked against the registered slots just before its
read.

Test: test_a_slot_that_disconnects_before_its_periodic_read_is_not_read
disconnects a slot while the cycle is blocked on an earlier read; without
this change its resolver still runs.
…inity-enterprise#863)

The event socket and the event stream close when the account's access
changes and reconnect at the new scope. Each worker re-checks its open
connections every 15 seconds, so a change made anywhere reaches every
worker.
…nterprise#863)

The event socket (/ws) and the event stream (/ws/events) close when the
account's access changes, and reconnect at the new scope.

- services/live_connections.py: a per-worker registry of open connections.
  A door registers before it reads the account and records a snapshot of
  the row it read (role, deactivation, email, a digest of the password
  hash; the key id for /ws/events). One comparison, snapshot against the
  current row, serves every re-check: gone, deactivated or key inactive
  closes with 4001; any other difference closes with 4000.
- Each worker re-checks its own connections every 15 s (a code constant)
  with one batched read, and reads nothing when it holds none. It runs in
  every worker with no leader lease, by design. A failed read keeps the
  connections open and logs.
- notify_account_changed(username) / notify_key_changed(key_id) run the
  same comparison at once for one account or key, on the serving worker.
  Both are synchronous and safe from any thread; off the registry's loop
  they hand over with call_soon_threadsafe. A failed read here closes the
  reported connections with 4000.
- The role route calls notify_account_changed after its write, only when
  the role changed (a raise closes too).
- A closer unregisters its dispatcher slot, then closes the socket; it
  never cancels the handler's task. A change that lands while a door is
  opening refuses the connection.
- db: get_users_access_state(usernames) and get_keys_active(key_ids),
  SQLAlchemy Core, no schema change.

Tests: tests/unit/test_ent863_live_connections.py (38). Red without the
fix: see the mutation record in the slice report.
…count state (Abilityai/trinity-enterprise#863)

Each event door reads the account more than once as it opens: `/ws` takes
the snapshot and then resolves its scope; `/ws/events` resolves its scope
with the email its key read returned and takes the snapshot from the owner
read after it. A door now refuses with 4000 (the client reconnects) when
those reads disagree on what sets the scope (role, Workspace-only, email),
so the scope a socket is served at is always the one its snapshot records
and every later re-check compares against.

The fleet-reach guard allowlists the new comparison: it reads a role to
compare two reads of one person's account and selects no fleet.
…se (Abilityai/trinity-enterprise#863)

A re-check now closes the connections it decided on side by side, each
still bounded by CLOSE_TIMEOUT_S, instead of one after another. One
connection that does not finish closing therefore delays neither the other
closes in the same re-check nor the next tick. The test also pins the
bound itself: without it the tick would never end.
…very worker (Abilityai/trinity-enterprise#863)

notify_account_changed and notify_key_changed now also publish one control
event through the event bus, scope "control". Each worker's dispatcher hands
that event to its live-connection registry, which runs the same targeted
re-check against its own read and publishes nothing. A control event never
reaches a client: _fanout routes it to the handler and returns, and the
visibility rule has no branch for it, so neither live delivery nor a replay
can queue it. The handler and the publisher are injected from main.py, so the
bus and the registry still import nothing from each other. The per-worker
re-check every 15 seconds is unchanged.
…close live connections at once (Abilityai/trinity-enterprise#863)

The MCP key revoke and delete routes call notify_key_changed(key_id) after the
write: only the event streams opened with that key close, on every worker,
whoever owns the key, and the owner's other connections stay open. A password
change calls notify_account_changed(username) after the sessions and the
optional key revocation are written, so the account's connections close at
once on every worker (a revoked key's stream as ended, the rest to reconnect).
…prise#863)

Both terminal doors (the agent terminal and the system-agent terminal)
register with the worker's live-connection registry in the same step as the
account read that authorizes them, and close when the account's access
changes: an error frame naming the reason, then 4003.

- A shared TerminalAccess helper holds the session's entry. Its closer sends
  the frame and the close code and cancels the two forwarder tasks the
  session now owns; it never cancels the handler's own task, so the handler
  returns through its cleanup (container socket closed, session slot freed,
  entry released) and the server logs nothing per close.
- A close decided while the shell is starting only marks the entry. The door
  checks once the start has returned, closes the socket it got and never
  reports the session as started.
- The snapshot records the role and email the door authorized on.
- A session closed for an access change also ends its shell session in the
  container, best effort: the shell starts with a per-session marker, and an
  exec as the terminal's user ends the marked processes and their session.

Mutation record (each production hunk reverted on its own; the named tests
go red, and green again on restore): closer does not cancel the forwarders
(7 red); session not registered with the worker's registry (9); no start-time
check, agent door (1) and system door (1); snapshot from the row only (1); no
check after a browser read (1); no end of shell (6); end of shell on every
close (1); no session marker on the shell (4); closer cancels the handler
task (8); forwarders not owned as tasks (7); system door not registered (3);
an await between the account read and the registration (1, shape); the
end-of-shell script without its session pass (1) or its marker match (1),
both run on Linux.
…ore forwarding (Abilityai/trinity-enterprise#863)

Two properties the slice relies on had no test that a revert would turn
red: a terminal opened by an account with no email keeps its session across
a re-check of the unchanged row, and a close that lands after the start is
reported but before the forwarders run still ends the session and its shell
instead of leaving the forwarders waiting.
…ss passes (Abilityai/trinity-enterprise#863)

The end-of-shell script found the processes to end afresh on each pass,
and found a session only through a live marked leader. A process in that
session that ignored HUP and no longer carried the marker was therefore
lost once the HUP had ended the leader, and the script reported the
session ended. The sessions found on the first pass are now kept for the
later passes, so such a process is still sent KILL. The Linux script test
gains that process.
…as survivors (Abilityai/trinity-enterprise#863)

The end-of-shell script keeps a terminal's sessions across passes, so its
last pass also listed a session member that had already died but was not
yet reaped by its parent. It then reported 'still running' and exited 1.
On Linux this failed test_the_end_of_shell_script_ends_the_marked_session_and_nothing_else
every time: the test's own children stay zombies until it waits. Zombies
are now skipped; the same sed|cut read gives state and session.
…t863-live-connections

# Conflicts:
#	docs/memory/architecture/security.md
…t863-live-connections

# Conflicts:
#	docs/memory/architecture/security.md
#	docs/memory/requirements/auth.md
#	tests/registry.json
@github-actions

Copy link
Copy Markdown

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

@github-actions

Copy link
Copy Markdown

⚠️ Live-instance suite skipped — merge conflict against dev.

Resolve by merging dev locally and pushing the result; the next nightly re-tests.

Conflict: tests/registry.json (both sides appended entries at the end of
test_files); kept every entry from both sides.
…f digesting it (Abilityai/trinity-enterprise#863)

The access snapshot only needs to notice that the stored password hash
changed. It held a SHA-256 of that hash, which code scanning reads as a
password hashed with a fast algorithm (py/weak-sensitive-data-hashing).
The input was already the stored hash and the value was never persisted
or checked against a password, but a cryptographic digest was the wrong
tool for a change detector.

A snapshot is compared only with another read made by the same worker
process, so Python's per-process salted hash() of the stored hash is
enough: it tells two hashes apart and the snapshot still never holds the
hash itself. A new test pins both properties.
@vybe

vybe commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

merge-train (2026-11-11): validated READY at 45ed3762c (lane C: /validate-pr, /review, /cso, six wiring mutations all red) and green on train #4463.

Pushed fe2229afc to your branch: a merge of dev after #4243 landed. The only conflict was tests/registry.json, rebuilt from dev's entries plus your five test_ent863_* entries, with no duplicates. No source change.

Two non-blocking notes from validation, for a follow-up: the own-email change (routers/users.py) and the setup admin-email write (routers/setup.py) don't call notify_account_changed, so those sockets close on the 15-second tick rather than at once. The immediate suspension path also depends on the enterprise-module PR.

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge-train: batch validated on train/20261111-0613 (#4463); this head adds only a dev merge with the registry rebuild

@vybe
vybe merged commit 4448f54 into dev Nov 11, 2026
28 checks passed
trinity-ability pushed a commit that referenced this pull request Nov 11, 2026
#4426 landed the same in-flight roster re-read on dev, so dev's
src/backend/services/event_bus.py is taken whole. In integrations.md
dev's text already describes the queued re-read; only the
share-writer broadcast rule is re-applied on top of it.
test_1220_ws_roster_share_refresh.py passes against dev's event bus.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants