Skip to content

fix(audit): record a terminate that ended the turn when another writer wins the status race (#4475) - #4550

Merged
obasilakis merged 4 commits into
devfrom
feature/4475-terminate-audit-race
Nov 13, 2026
Merged

obasilakis merged 4 commits into
devfrom
feature/4475-terminate-audit-race

Conversation

@obasilakis

@obasilakis obasilakis commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • A terminate now writes its execution_terminate audit row when the agent answers terminated (container arm) or the turn is found parked (parked arm). Before, the row depended on winning the CANCELLED compare-and-swap, which the in-flight /task / /chat caller (or the parked dispatcher) usually wins, so on a real instance the row was lost.
  • Repeat and concurrent terminates write one row: a Redis first-claim terminate_audit:{agent}:{execution_id} (SET NX, 24h TTL) on an id the agent or the call limiter verified. Fails open when Redis is unreachable; the claim is released when the audit write fails (the hook returns log's event id; None or a raise releases it).
  • The CAS still decides the status row and the dispatch-activity close. The queued arm is unchanged.

Behaviour change

test_a_container_terminate_of_a_turn_already_terminal_writes_no_row (#4035) becomes ..._writes_one_row: the agent killed a live process whose row was already terminal, so the terminate ended work and is recorded.

Changes

  • src/backend/services/chat_execution_service.py: _audit_terminate_once, proxy and parked arms
  • src/backend/routers/chat.py: comment
  • tests/unit/test_4475_terminate_audit_race.py (new), tests/unit/test_3180_operational_write_audit.py
  • docs/memory/requirements/security.md, docs/memory/feature-flows/execution-termination.md, learnings fragment, CSO diff report (no findings)

Test Plan

Out of scope (follow-up #4551, needs an agent-image rebuild)

  • A terminated chat turn ends failed (agent /api/chat has no cancel relabel).
  • Dedupe in the agent's process registry, which would let the Redis claim go.

Fixes #4475

🤖 Generated with Claude Code

…r wins the status race (#4475)

The execution_terminate audit row fired only when the terminate request won
the CANCELLED compare-and-swap. The agent's terminate route waits for the
process to exit before it answers, so the synchronous /task or /chat caller
that started the turn usually writes the terminal status first, and the
terminate wrote no audit row although its kill ended the work. The parked
arm had the same race with the parked dispatcher.

The container arm now records the row when the agent answers `terminated`,
and the parked arm when it finds the turn parked. A Redis first-claim
(terminate_audit:{agent}:{execution_id}, SET NX, 24h) on an id the agent or
the call limiter verified keeps a repeat or concurrent terminate to one row.
It fails open when Redis is unreachable and releases the claim if the audit
write raises. The CAS still decides the status row and the activity close.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@webmixgamer

Copy link
Copy Markdown
Contributor

/review + /validate-pr — REQUEST CHANGES (head c03104530)

The fix is right and proven by mutation; nothing below asks for a change to its logic. One blocking item: CI's regression diff is red on a pinned guard this PR trips (C1). It is a one-entry fix.

Scope: #4475 (P2, theme-security, found on v1.0.5-rc2): terminating a running execution returns 200 terminated but writes no execution_terminate audit row, because the row was gated on the terminate winning the CANCELLED compare-and-swap and the in-flight /task / /chat caller usually writes the terminal first. The PR decides the row on the arm's own outcome instead (the agent answered terminated, or the turn was found parked) and deduplicates repeats with a Redis first-claim terminate_audit:{agent}:{execution_id} (SET NX, 24 h, fail-open). The queued arm is unchanged. Scope is clean: two backend files (one is a comment-only edit), one new test file, one flipped assertion in the #3180 suite (declared in the PR body as a behaviour change), docs, a learnings fragment and a CSO diff report. The issue's secondary item (a terminated chat turn ends failed) is deferred in the PR body without an issue number.

Execution coverage

changed symbol executed by live consumer verdict
_audit_terminate_once (src/backend/services/chat_execution_service.py:3349) all 7 tests of tests/unit/test_4475_terminate_audit_race.py; TestExecutionTerminate in tests/unit/test_3180_operational_write_audit.py (through the real route) chat_execution_service.py:3554 (container arm), :3750 (parked arm) executed
container arm: audit on the agent's terminated answer, not on cancel_won (chat_execution_service.py:3553-3554) test_4475 task-raced, chat-raced, redis-down, second-task-id; test_3180 ..._already_terminal_writes_one_row reached from src/backend/routers/chat.py:1205-1210 (on_ended=_audit_terminate) executed; mutation M2 red (5 tests)
parked arm: audit when the turn is found parked, not on cancel_won (chat_execution_service.py:3750) test_4475 ..._parked_turn_finalized_by_its_dispatcher... same route executed; mutation M3 red (1 test)
the first-claim (SET NX) test_4475 second-task-id, claim-helper test; test_3180 the two "repeated" tests chat_execution_service.py:3365-3368 executed; mutation M4 red (4 tests)
claim keyed on execution_id, not the caller-supplied task_execution_id test_4475 ..._second_task_execution_id... chat_execution_service.py:3554 executed; mutation M6 red (1 test)
fail-open when the Redis client is None test_4475 test_with_redis_down_the_row_is_still_written chat_execution_service.py:3364-3365 executed; mutation M9 red (1 test)
fail-open when the Redis call raises (chat_execution_service.py:3369-3371) — same not executed by any test (mutation M5 stays green) — Informational 2
claim released when the hook raises (chat_execution_service.py:3374-3380) test_4475 test_a_failed_audit_write_gives_the_claim_back no production hook raises — Informational 1 executed, but by a hook shape production does not have
route hand-off on_ended=_audit_terminate (routers/chat.py:1210, unchanged line) every row-asserting test goes through the real router — pinned: deleting the line turns 14 tests red (M10)

The tests do not assert source text. They build an app from the real agents and chat routers, resolve credentials with the real resolver, and read audit_log back from a real SQLite store; Redis is fakeredis; the agent HTTP boundary and the capacity manager are stubs.

How the race is reproduced (PROVED by reading + probe). The CAS result is not mocked. _inflight_writer_wins (tests/unit/test_4475_terminate_audit_race.py:51-66) wraps db.update_execution_status so that, on the first call, the competing writer's real terminal write runs first (:61, through the real DB function) and then the terminate's own real CAS runs (:63). assert raced == [True] (:80, :95, :118) proves the competitor won its CAS. The tests do not assert the terminate's CAS returned False; a reviewer probe did: with a competing failed write, the terminate's CAS returned [False], the row kept failed / agent returned 502, the slot release was per-execution, and one audit row was written. What the harness does not model is the competitor running in another coroutine or worker during the agent's wait; from the terminate's point of view the ordering is the same (the row is terminal when its CAS runs).

Fix mutation (PROVED). In a scratch worktree at the head, src/backend/services/chat_execution_service.py and src/backend/routers/chat.py were reverted to the merge-base (6bbff0d87), tests kept. pytest -p no:randomly unit/test_4475_terminate_audit_race.py unit/test_3180_operational_write_audit.py: 8 failed, 33 passed (41 passed at the head).

Finer mutations at the head, same two files run each time (41 tests):

mutation result
M2 container arm gated on cancel_won again 5 failed
M3 parked arm gated on cancel_won again 1 failed
M4 claim never refuses (no dedupe) 4 failed
M5 Redis error on the claim re-raised instead of failing open 0 failed (41 passed)
M6 container claim keyed on task_execution_id 1 failed
M7 claim not released when the hook raises 1 failed
M8 container arm audits any 200 answer (incl. already_finished) 1 failed
M9 no row when the Redis client is None 1 failed
M10 route no longer passes on_ended 14 failed

Orderings (each row: what was run, or "read").

ordering audit rows status row evidence
terminate wins the CAS, agent answered terminated 1, status=terminated, actor = caller cancelled test_3180 test_a_container_terminate_writes_one_row (session and user key) — run, green
in-flight caller wins with CANCELLED or FAILED, kill landed 1, terminated the caller's terminal stands test_4475 task-raced / chat-raced — run, green; red under M1/M2
row already failed or success, process still live, kill landed 1, terminated unchanged test_3180 ..._already_terminal_writes_one_row (failed); reviewer probe (success) — run
the run finished on its own, kill was a no-op (already_finished) 0 unchanged test_3180 test_terminating_an_already_finished_turn_writes_no_row — run, green; red under M8
second terminate, agent answers 404 0 unchanged test_3180 test_a_failed_terminate_writes_no_row[agent-404] — run, green
second terminate, agent answers terminated again (pending entry kept, or two kills of one live process) still 1, the first claimant unchanged test_3180 repeated-container test — run, green; red under M4
parked turn, dispatcher wins the CAS 1, cancelled_while_parked cancelled test_4475 parked-raced — run, green; red under M3
parked cancel, then a container terminate of the same id still 1 (cancelled_while_parked) — reviewer probe — run
Redis unavailable, two terminates 2 (documented fail-open cost) — reviewer probe — run

When the agent answers terminated is INFERRED from docker/base-image/agent_server/services/process_registry.py:318-343 and :420: only for a process whose poll() was None at the kill, or for a pending entry whose cancel flag it set; an exited process answers already_finished, an unknown id 404. So the new gate does not fire for a run that finished on its own. The audit row's actor is the principal of the request that took the claim, the target is the agent, and details.status is the arm's own outcome — honest in every ordering above. With two people terminating at once only the first claimant is recorded (stated in the code comment and the CSO report); before the fix only the CAS winner was.

The architecture rule (every other terminal side effect stays behind the CAS). PROVED by the diff: the only executable changes are the two on_ended call sites and the new helper. _close_dispatch_activity_cancelled(task_execution_id, cancel_won) is untouched in both arms (chat_execution_service.py:3533, :3733); the per-execution release_if_matches and _record_cancel_event were already outside the CAS at the merge-base and did not move. tests/unit/test_1804_terminal_activity_parity.py and tests/unit/test_3588_close_cancelled_claimed_activities.py are green at the head. The terminate path sends no notification and wakes no caller.

Auth. No route dependency changed: routers/chat.py differs only in a comment (:1183-1187). The route keeps Depends(get_authorized_agent) + Depends(get_current_user) + @ops_write_route, and the below-owner-tier "only a turn you started" check (routers/chat.py:1168-1177). The claim is taken only after the agent answered terminated for that id on that agent, or after the call limiter found the id parked on that agent, so a caller cannot spend the claim for an id they could not terminate; a caller with no hook (the public-link and Workspace routes) takes no claim (test_4475 test_no_hook_takes_no_claim).

Multiple workers / scheduler. Nothing in the fix is process-local: the claim lives in Redis, the row in the database. The scheduler process does not call this path (read, not run).

Critical

C1. regression diff is red: the new claim is a set(..., nx=True) outside the single-flight allowlist (confidence 10/10)

_audit_terminate_once takes its first-claim with set(nx=True, ex=…) in src/backend/services/chat_execution_service.py. tests/unit/test_1920_no_hand_rolled_single_flight.py::test_no_unlisted_hand_rolled_single_flight_lock scans for that call shape and fails for any file not in its allowlist. PROVED, on this head:

cd tests && python -m pytest -p no:randomly unit/test_1920_no_hand_rolled_single_flight.py -q
E   AssertionError: New `set(..., nx=True, …)` call(s) outside redis_breaker_util.SingleFlightLock:
E       services/chat_execution_service.py

CI agrees: regression diff reports one new failure introduced by the head, this test. It is the PR's own red, not a flake.

The guard's own message gives the two ways out: adopt SingleFlightLock, or add one justified allowlist entry for a deliberate non-lock use. This claim is a write-once dedupe marker with a 24 h life, not a lock that is released when work ends, so an allowlist entry with that reason looks like the right one; the author decides. My first pass missed this guard: the targeted run did not include it.

Informational (non-blocking)

  1. The "claim is released if the audit write raises" branch cannot fire for the real hook (confidence 8/10). platform_audit_service.log catches every exception and returns None (src/backend/services/platform_audit_service.py:211-218), so _audit_terminate never raises. When the audit insert fails, the claim stays for 24 h and a repeat writes nothing. Probe (scratch, db.create_audit_entry made to raise): first terminate 200, rows: 0, claim: 1, ttl: 86400; second terminate 200, rows: 0. This is not a regression — before the fix a failed write was lost the same way — but docs/memory/feature-flows/execution-termination.md:485, the CSO report's A10 row and test_4475:163-178 describe a recovery production does not have. Either let the hook return log's event id and release the claim on None, or reword the docs.
  2. The fail-open branch for a Redis call that raises is untested (confidence 8/10). get_breaker_redis caches the client after the first successful ping (src/backend/redis_breaker_util.py:58), so an outage after boot shows up as .set raising, not as a None client — the branch at chat_execution_service.py:3369-3371. Mutation M5 (re-raise there) leaves all 41 tests green. The code is right: a probe with a client whose .set raises ConnectionError wrote the row. Add that case beside test_with_redis_down_the_row_is_still_written.
  3. Stale docstrings in tests/unit/test_3180_operational_write_audit.py:522-526 and :611-614 still say only the call that wins the CANCELLED CAS ended the turn.
  4. The new test file is not in tests/registry.json (convention, not enforced: 154 of the 222 unit test files added on dev in the last two weeks are registered).
  5. The live re-test is still open. The PR's manual Test Plan item is unchecked, and the issue states its cause was "read from source, not confirmed by server logs". The fix no longer depends on the CAS at all once the agent answers terminated (which the issue observed), so the reported case is covered by construction; row B20 still needs a re-test on the next release candidate by someone other than the author.
  6. The deferred items have no issue number. The PR body lists "a terminated chat turn ends failed" (the issue's secondary point) and "dedupe in the agent's process registry" as follow-ups without filing them; Fixes #4475 will promote and later close the issue that carries the first one.
  7. Feature-flow Revision History (docs/memory/feature-flows/execution-termination.md:573) has no entry for the new section; the PR body carries no Journey Impact: line (the gate passes: "allowed for now").

Low confidence (appendix):

  • Same class, not introduced here: when the kill lands but the answer is lost (the backend's 15 s proxy timeout against the agent's up-to-7 s wait plus its sweep, or a termination_unconfirmed 5xx), no row is written and the retry gets a 404. Reachability not checked.
  • The production and hosted compose files run Redis with allkeys-lru (docker-compose.prod.yml:668-669); under memory pressure the claim can be evicted, which fails toward a duplicate row, not a missing one.
  • The execution id can be re-used by an in-line retry on the agent; a second, distinct kill of the same id within 24 h would be deduplicated. No reachable path found (the first terminate leaves the row terminal).
  • The claim is a blocking Redis call on the event loop (1 s socket timeout, once per terminate), as at other call sites of the same client.

Methodology (validate-pr)

  • Lane: B (no auth/credential/CI/image path; 6 deleted lines under tests/, all in the declared flipped test). No schema change.
  • Issue link / closing keyword: Fixes #4475 in the body; the issue exists, priority-p2, type-bug, theme-security, status-in-progress. Base is dev. One commit, conventional fix(audit): ….
  • Journey Impact: scripts/ci/check_journey_impact.py on the PR's body and SHAs → PASSED (no declaration; allowed for now).
  • Docs tier: bug fix — a commit message would do; the PR also updates the feature flow (new "Audit Row" section), docs/memory/requirements/security.md (the execution_terminate line) and adds a learnings fragment in the required shape. No architecture file needs a change (no endpoint, schema or service added).
  • Requirements: updated (docs/memory/requirements/security.md:58).
  • Tests registered: no (Informational 4).
  • Security report: docs/security-reports/cso-diff-2026-11-13-4475-terminate-audit-race.{md,json} present, no findings; its A10 row repeats the claim in Informational 1.
  • Security greps (§4.1-4.6, §4.9, §4.10): no key, email, public IP, env file, hardcoded secret, credential file, new env var, symlink, mode change or host path in the diff. Enterprise-docs pattern over the added lines: no hit.
  • Audit rows (§4.11): audit_rows.py → ➖ no gated write handler touched (the handler's gate is the access-tier dependency, outside the script's owner/admin set). The handler does write its row through _audit_terminate.
  • Cross-surface drift (§5.5): nothing printed. routers/chat.py changed only comment lines; its # mcp: header already lists terminate as UI-only; no agent-server twin of the changed service file.
  • Product fit (6.6): bug, not incubating; the one user-visible change (the row now appears) is the issue's "Expected". No new route, tool, setting or env key.
  • Targeted runs at the head (local, -p no:randomly, SQLite): the two PR files 41 passed; 19 further files covering the touched functions and the guards that read them (test_1804_terminal_activity_parity, test_3588_…, test_2433_dispatch_wiring, test_ent155_chat_cancel, test_ent875_exec_control_routes, test_2795_room_stop, both test_4284_*, test_ent279_scrub_parity, test_2106_…, test_2789_…, test_terminate_async_executor, test_timeout_termination, test_3180_ops_write_scope, test_3180_ops_write_route_census, test_1310_auth_wiring, test_186_enumeration_uniformity, test_2996_human_only_routes, test_2338_journey_catalog): 800 passed, 19 errors. All 19 errors are in test_3180_ops_write_route_census.py, which imports the full app and fails locally on a missing opentelemetry.exporter module; the same file gives the same 19 errors at the merge-base, so they are environmental. test_1560_agent_redis_key_parity, test_1560_breaker_cleared_on_lifecycle, test_models_centralized, test_2019_pytest_timeout_guard: 39 passed.
  • CI on head c03104530: 19 pass, 6 skipped, 1 failed: regression diff (C1). pytest (head, seed 12345) passes because it only produces the results file. Pass also: Analyze (python), Analyze (javascript-typescript), CodeQL, schema-parity, verify-non-root, gitleaks, journey-impact, prod-image-smoke, backend boots without enterprise submodule, cached dev baseline, the guard and lint jobs. Skipped: e2e, journey-smoke, report, auto-merge, the two fallback pytest jobs.
  • Merge conflicts with dev: none (git merge-tree of the head onto the current origin/dev tip 54f663a61; dev changed docs/memory/requirements/security.md after the merge-base and it still merges). GitHub reports MERGEABLE, REVIEW_REQUIRED, no reviews, no labels.

Not checked

  • The full pinned-guard list of the review skill's Step 2.5 (303 files by its pattern) and the full unit suite — only the files named above were run; the diff adds no route, protected path or registry entry. CI's regression diff is the signal for the rest; it reports only the C1 guard.
  • PostgreSQL: every local run was the [sqlite] parametrisation.
  • A live instance: no terminate against a running agent with a working model, no second uvicorn worker, no real Redis. The agent's answers are a stub in every test; when the real agent answers terminated was read from source, not run. Agents on an older base image were not considered.
  • The CSO report's variant-analysis claim (no other audit hook gated on a CAS result) was not re-derived.
  • /cso --diff was not re-run; frontend, MCP server and the scheduler were not exercised (the diff does not touch them).

@webmixgamer webmixgamer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on c03104530: one blocking finding (C1, regression diff is red on the single-flight guard test_1920, which the new set(nx=True) claim trips). The fix logic is confirmed by mutation and needs no change; one justified allowlist entry, or the shared lock helper, clears it. The /review + /validate-pr report is in the comment above.

@webmixgamer webmixgamer added the status-needs-fix PR has an unaddressed review/validation finding; cleared by the author's next push (#2815) label Nov 13, 2026
…ils (#4475)

- The route hook returns platform_audit_service.log's event id; a None
  result releases the terminate_audit claim, so the next terminate
  records the turn. log swallows insert errors, so the raise branch
  alone never fired for the real hook.
- Allowlist the terminate_audit dedupe claim in the #1920 single-flight
  guard (once-guard, 24 h TTL, deleted only on a failed audit write).
- Tests: the Redis claim raising still writes the row; a failed audit
  insert through the real route releases the claim and a repeat writes
  one row.
- Refresh the #3180 test docstrings, the feature-flow Audit Row text and
  Revision History, the CSO report A10 row; register the #4475 test file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@obasilakis

Copy link
Copy Markdown
Contributor Author

Pushed 57f0d5a2f.

…-audit-race

# Conflicts:
#	tests/registry.json
@github-actions github-actions Bot removed the status-needs-fix PR has an unaddressed review/validation finding; cleared by the author's next push (#2815) label Nov 13, 2026
@webmixgamer

Copy link
Copy Markdown
Contributor

Re-review — APPROVE (head ca3654987, previous c03104530)

The blocking finding is resolved and nothing new blocks. CI is green on this head, regression diff included.

The blocking finding (C1)

Resolved (PROVED). pytest -p no:randomly unit/test_1920_no_hand_rolled_single_flight.py at ca3654987: 21 passed. The same file with the previous head's chat_execution_service.py put back: 1 failed, 20 passed, naming services/chat_execution_service.py — the failure CI reported.

The allowlist entry (tests/unit/test_1920_no_hand_rolled_single_flight.py:128-131) is one row for one file with one reason, which is the only granularity this guard has. Its text matches the code: nothing waits on or renews the claim, it expires after 24 h, it is deleted only when the audit write fails or the hook raises, and it fails open. One thing it leaves out is in "New in this push", point 1.

Guards run this time. The first pass ran a hand-picked set and missed this guard, so the set was derived mechanically and widened:

  • every unit test file matching the review skill's guard pattern, or an allowlist / _EXPECTED_ / ast.walk / rglob marker, or any file-reading or tree-walking idiom (open(, read_text, getsource, glob, os.walk, iterdir, subprocess) together with a backend path, or a route-table / getsource idiom, or a Redis key shape (nx=True, keyspace, key prefix): 703 files, run sequentially in four processes at ca3654987;
  • result: 24,651 passed, 50 failed, 101 errors, 39 skipped;
  • every one of the 151 failing or erroring test ids was re-run against the dev tip 76f640054 (the merge's second parent): 143 fail identically there, so they are not this PR's (the causes were not all traced; the ones looked at are local-environment gaps such as a missing OpenTelemetry exporter module, which breaks every test that imports the full app). The other 8 are all in test_ent1030_skills.py and are order-dependent in the combination run here: that file gives 24 passed alone at the head, and with the same eleven-file set the head and the dev tip fail the identical 92 ids;
  • a small recording plugin logged which tests open src/backend/services/chat_execution_service.py or src/backend/routers/chat.py as text (the scanners, as opposed to tests that import them): 174 tests in 98 files. Five of those files have a failing test at the head (test_3180_ops_write_route_census.py 19, test_4499_vendor_egress_switch.py 4, test_3434_removed_routes.py 1, test_3785_operator_queue_package.py 1, test_ent872_fleet_reach.py 1) and all 26 fail identically at the dev tip.
The 98 files whose tests read one of the two files as text (tests that did so)

test_1068_task_timeout_demotion.py (1), test_1168_agent_ssh_bind.py (2), test_1310_auth_wiring.py (2), test_1560_agent_redis_key_parity.py (2), test_1677_operator_alert_emitters.py (2), test_1804_terminal_activity_parity.py (3), test_1871_log_config_parity.py (1), test_1920_no_hand_rolled_single_flight.py (3), test_2048_pull_pilot_reach.py (1), test_2094_dependency_path_param_pairing.py (1), test_2104_agent_type_retired.py (1), test_2186_start_stopped_agent_recreate.py (1), test_2241_magicmock_assert_guard.py (2), test_2323_machine_identities.py (3), test_2339_testing_docs_consolidated.py (1), test_2376_approval_response_membership.py (1), test_2541_restart_policy_parity.py (1), test_2661_task_timeout_receipt.py (2), test_2669_minting_accessor_callers.py (1), test_2794_room_file_awareness.py (1), test_2806_inter_agent_depth.py (1), test_2889_readiness_classifier.py (4), test_293_admin_gate_rejects_agent_keys.py (1), test_2996_human_only_routes.py (1), test_3102_host_header_fences.py (1), test_3106_agent_server_fastapi_telemetry.py (1), test_3109_agent_ws_type.py (1), test_3114_pull_route_interactive.py (1), test_3180_ops_write_route_census.py (5), test_3214_volume_instance_scope.py (1), test_3245_async_replay_after_failure.py (2), test_3246_platform_alerts_registry.py (6), test_3317_loop_dispatch_claim.py (1), test_3434_removed_routes.py (1), test_3590_rebuild_pull_env.py (2), test_3785_operator_queue_package.py (7), test_3881_public_url_trailing_dot.py (1), test_4105_pull_trigger_census.py (1), test_4257_receipt_wait_reason.py (1), test_4284_no_exception_text_in_5xx.py (1), test_4499_vendor_egress_switch.py (2), test_837_workspace_only_floor.py (1), test_837_workspace_route_census.py (5), test_848_inline_auth_compose_wiring.py (1), test_937_ent781_census.py (1), test_agent_auth_header_guard.py (1), test_backlog.py (1), test_chat_dispatched_marker.py (2), test_database_facade_delegation.py (1), test_ent1073_env_read_fail_closed.py (1), test_ent1095_readdress_role.py (1), test_ent1101_mcp_bind_address.py (2), test_ent1111_reserve_id_prefix.py (1), test_ent1117_s13_service_error_text.py (4), test_ent1153_people_search_seam.py (1), test_ent1258_suspended_workspace_door.py (1), test_ent279_scrub_parity.py (4), test_ent314_hardened_yaml.py (2), test_ent329_operator_resume.py (1), test_ent419_memory_poisoning_gates.py (1), test_ent435_settings_sink_guard.py (1), test_ent467_ws_agent_scope.py (2), test_ent568_delegation_contract.py (1), test_ent614_source_agent_attribution.py (4), test_ent615_credential_helper_parity.py (1), test_ent615_token_free_remotes.py (2), test_ent629_agent_reach.py (2), test_ent715_queue_person_fields.py (4), test_ent719_credential_and_key_routes_human_only.py (1), test_ent719_listed_agent_uses.py (1), test_ent719_no_agent_only_grant_guard.py (1), test_ent719_own_settings_identity_git_human_only.py (1), test_ent719_room_uses_delegated.py (1), test_ent719_schedule_webhook_person_or_grant.py (1), test_ent719_self_gate_identity.py (1), test_ent719_workspace_grants_person_only.py (1), test_ent720_email_binding.py (1), test_ent751_gate_guards.py (6), test_ent752_self_approval_clearance.py (1), test_ent753_assignment_hooks.py (1), test_ent819_no_platform_links_in_home.py (1), test_ent830_idempotency_caller_scope.py (1), test_ent840_agent_self_gate.py (1), test_ent849_email_code_policy.py (1), test_ent872_fleet_reach.py (6), test_ent875_exec_control_routes.py (5), test_ent885_keyless_refused.py (1), test_ent901_withdraw_ask.py (1), test_ent926_agent_key_turn_scope.py (1), test_ent926_agent_route_census.py (4), test_ent966_delegation_routes.py (2), test_ent99_no_sql_order_by_next_run_at.py (1), test_models_centralized.py (2), test_redis_url_no_fallback.py (2), test_timeout_termination.py (1), test_whatsapp_adapter.py (1), test_whatsapp_inbound_media.py (1), test_workspace_flag_retired.py (2)

So no guard that scans either changed file, and none in the wider 703-file set, is red because of this PR. The remaining 766 unit test files were not run locally; CI's regression diff covers them and is green.

The other points of the reply

point claim verified how result
I1 the hook returns log's event id; a None releases the claim; a route-level test covers it; docs and the A10 row updated read chat_execution_service.py:3374-3387 and routers/chat.py:1188-1189; ran the new test; two mutations (below); probes done. With the release removed (M11) the new test is red; with the route hook dropping log's return value (M12) three repeat tests are red. Probe: audit store down for two terminates → 0 rows, claim absent; store back up for two more → 1 row, claim present. log has exactly one None path, the swallowed exception (platform_audit_service.py:209, :218), so None never means "skipped on purpose". The only hook is the route's; the queued arm ignores the return value (chat_execution_service.py:3329-3330). Flow doc :485 and both CSO report files reworded.
I2 a test where .set raises ConnectionError; M5 now red re-ran M5 done. M5 (re-raise on the Redis error): 1 failed, 42 passed, the new test_with_the_redis_claim_raising_the_row_is_still_written (test_4475:134).
I3 both #3180 docstrings updated read the diff done (test_3180_operational_write_audit.py:522-526, :611-614).
I4 test file registered git diff c03104530 57f0d5a2f -- tests/registry.json: 13 added lines, 0 deleted; after the merge, against dev: 13 added, 0 deleted; the file parses, 789 entries, no duplicate; test_2338_journey_catalog.py 51 passed done, additive.
I5 live re-test still open — still open, as the author says. Row B20 needs a re-test on the next release candidate.
I6 deferred items tracked in #4551 read the issue done. #4551 (open, type-bug, priority-p3) carries both: a terminated chat turn ends failed, and repeat terminates are indistinguishable at the agent, with acceptance criteria. The PR body links it.
I7 Revision History row added read the diff done (execution-termination.md:577).
merge ca3654987 merges dev compared the merge commit's tree with a plain automatic merge of 57f0d5a2f and 76f640054 clean. The only file that differs from the automatic merge is tests/registry.json, which conflicted and was resolved by keeping both sides. dev had not touched the two backend files or the two edited test files.

Mutations at ca3654987 (the two PR test files, 43 tests, all green unmutated):

mutation result
M2 container arm gated on cancel_won again 7 failed
M3 parked arm gated on cancel_won again 1 failed
M4 claim never refuses (no dedupe) 4 failed
M5 Redis error on the claim re-raised 1 failed (was 0)
M6 container claim keyed on task_execution_id 1 failed
M7 claim not released when the hook raises 1 failed
M8 container arm audits any 200 answer 1 failed
M9 no row when the Redis client is None 1 failed
M10 route no longer passes on_ended 16 failed
M11 claim not released when the hook returns None (new) 1 failed
M12 route hook drops log's return value (new) 3 failed
M13 claim released after every write, also a successful one (new) 4 failed

Probes re-run at ca3654987 (scratch only): a client whose .set raises → 1 row; parked cancel then container terminate of the same id → 1 row (cancelled_while_parked); Redis client None, two terminates → 2 rows (the documented fail-open cost); Redis up, two terminates by two credentials → 1 row; lost CAS → the row keeps the other writer's status and error, slot release per execution, 1 audit row; a slow write that succeeds while a second terminate arrives → the second is refused, 1 row, claim kept.

New in this push

None of these blocks. The release-on-None is new behaviour, so it was reviewed as new code.

  1. The release is a tokenless DEL of a constant-value claim (chat_execution_service.py:3368 sets "1", :3374-3379 deletes the key unconditionally). That is the shape test_1920's own docstring calls the worst class for a lock, and the neighbouring once-guard rows in its allowlist say "no release, no token" while this one has a release. Control flow keeps it safe in normal operation: only the caller whose SET NX succeeded reaches _release. It can delete another caller's claim only if the claim vanished while the first write was in flight (a flush or an eviction; expiry cannot do it, 24 h against one insert) and a second caller took it. Probe: first caller claims and its write stalls, the key is deleted, a second caller claims and writes, the first write returns None and releases, a third caller writes → 2 rows. The cost is one duplicate row in a case that should not occur. Either say so in the allowlist reason, or store a per-call token and compare before deleting (redis_breaker_util has lock_token_matches).
  2. A write that commits and then raises releases the claim (chat_execution_service.py:3386-3387). Probe: the insert lands, the call then raises, log returns None; a repeat terminate → 2 rows. Same direction as the stated policy (a duplicate costs less than a missing row).
  3. "So the next terminate records the turn" is true for some turns only (flow doc :485, the registry description, the allowlist reason). A later terminate reaches the claim again while the turn is still parked or pending, or when two kills overlap. After a completed kill the agent drops the process from its registry (docker/base-image/agent_server/services/process_registry.py:416-418) and answers 404, which is an error arm and writes nothing — INFERRED from source, not run; the new route-level test uses a stub agent that always answers terminated. A second terminate that arrives while a failing write is still in flight is refused and is not recorded either (probe: hooks run A-failed, then nothing for the concurrent call, then a third call records). So a failed audit insert on the common path still loses the row, as it did before this PR. Wording only.
  4. platform_audit_service.log's docstring now disagrees with a caller: "Callers should not branch on the return value" (platform_audit_service.py:127). The hook's result is branched on, for the claim only; the response does not change. A one-line docstring update would keep the next reader from "fixing" it.
  5. A Redis error during the release is swallowed without a log line (chat_execution_service.py:3376-3379). Probe: .delete raises → both terminates answer 200, 0 rows, the claim stays 24 h. Same outcome as before this push for that case; a warning line would make it visible.

CI and merge state

gh pr checks 4550 on head ca3654987: 22 pass, 5 skipped, nothing failed or pending. pytest (head, seed 12345), pytest (base, inline fallback) and regression diff all pass, with the security, schema, image and guard jobs. Skipped: e2e, journey-smoke, report, auto-merge, pytest (push, absolute failures).

GitHub: MERGEABLE, review decision CHANGES_REQUESTED (the earlier request), no labels. The branch already contains the dev tip 76f640054.

Not checked

  • The 766 unit test files outside the 703-file set, and PostgreSQL (local runs are SQLite only). regression diff is the signal for the unit files and it passes on this head.
  • The private submodule is not mounted in the reviewed tree: its own twin of the single-flight guard and any caller there were not looked at.
  • A live instance, a second worker, a real Redis and the real agent: the agent's answers are a stub in every test, so point 3 above is read from source.
  • The read-recording plugin sees open-based reads only; a guard that shells out to git grep would not be listed among the 98 (it would still have been run if it matched the set).

@webmixgamer webmixgamer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving ca3654987: the single-flight guard passes with one narrow allowlist entry, the release-on-failed-write behaviour is covered and every mutation goes red, and CI is green including regression diff. Five non-blocking notes on the new release path are in the re-review comment above. Row B20 still needs a re-test on the next RC by someone other than the author.

…-audit-race

# Conflicts:
#	tests/registry.json
@obasilakis
obasilakis merged commit 548e1ff into dev Nov 13, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants