Repository navigation
fix(audit): record a terminate that ended the turn when another writer wins the status race (#4475) - #4550
Conversation
…r wins the status race (#4475) The execution_terminate audit row fired only when the terminate request won the CANCELLED compare-and-swap. The agent's terminate route waits for the process to exit before it answers, so the synchronous /task or /chat caller that started the turn usually writes the terminal status first, and the terminate wrote no audit row although its kill ended the work. The parked arm had the same race with the parked dispatcher. The container arm now records the row when the agent answers `terminated`, and the parked arm when it finds the turn parked. A Redis first-claim (terminate_audit:{agent}:{execution_id}, SET NX, 24h) on an id the agent or the call limiter verified keeps a repeat or concurrent terminate to one row. It fails open when Redis is unreachable and releases the claim if the audit write raises. The CAS still decides the status row and the activity close. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/review + /validate-pr — REQUEST CHANGES (head
|
| changed symbol | executed by | live consumer | verdict |
|---|---|---|---|
_audit_terminate_once (src/backend/services/chat_execution_service.py:3349) |
all 7 tests of tests/unit/test_4475_terminate_audit_race.py; TestExecutionTerminate in tests/unit/test_3180_operational_write_audit.py (through the real route) |
chat_execution_service.py:3554 (container arm), :3750 (parked arm) |
executed |
container arm: audit on the agent's terminated answer, not on cancel_won (chat_execution_service.py:3553-3554) |
test_4475 task-raced, chat-raced, redis-down, second-task-id; test_3180 ..._already_terminal_writes_one_row |
reached from src/backend/routers/chat.py:1205-1210 (on_ended=_audit_terminate) |
executed; mutation M2 red (5 tests) |
parked arm: audit when the turn is found parked, not on cancel_won (chat_execution_service.py:3750) |
test_4475 ..._parked_turn_finalized_by_its_dispatcher... |
same route | executed; mutation M3 red (1 test) |
the first-claim (SET NX) |
test_4475 second-task-id, claim-helper test; test_3180 the two "repeated" tests |
chat_execution_service.py:3365-3368 |
executed; mutation M4 red (4 tests) |
claim keyed on execution_id, not the caller-supplied task_execution_id |
test_4475 ..._second_task_execution_id... |
chat_execution_service.py:3554 |
executed; mutation M6 red (1 test) |
fail-open when the Redis client is None |
test_4475 test_with_redis_down_the_row_is_still_written |
chat_execution_service.py:3364-3365 |
executed; mutation M9 red (1 test) |
fail-open when the Redis call raises (chat_execution_service.py:3369-3371) |
— | same | not executed by any test (mutation M5 stays green) — Informational 2 |
claim released when the hook raises (chat_execution_service.py:3374-3380) |
test_4475 test_a_failed_audit_write_gives_the_claim_back |
no production hook raises — Informational 1 | executed, but by a hook shape production does not have |
route hand-off on_ended=_audit_terminate (routers/chat.py:1210, unchanged line) |
every row-asserting test goes through the real router | — | pinned: deleting the line turns 14 tests red (M10) |
The tests do not assert source text. They build an app from the real agents and chat routers, resolve credentials with the real resolver, and read audit_log back from a real SQLite store; Redis is fakeredis; the agent HTTP boundary and the capacity manager are stubs.
How the race is reproduced (PROVED by reading + probe). The CAS result is not mocked. _inflight_writer_wins (tests/unit/test_4475_terminate_audit_race.py:51-66) wraps db.update_execution_status so that, on the first call, the competing writer's real terminal write runs first (:61, through the real DB function) and then the terminate's own real CAS runs (:63). assert raced == [True] (:80, :95, :118) proves the competitor won its CAS. The tests do not assert the terminate's CAS returned False; a reviewer probe did: with a competing failed write, the terminate's CAS returned [False], the row kept failed / agent returned 502, the slot release was per-execution, and one audit row was written. What the harness does not model is the competitor running in another coroutine or worker during the agent's wait; from the terminate's point of view the ordering is the same (the row is terminal when its CAS runs).
Fix mutation (PROVED). In a scratch worktree at the head, src/backend/services/chat_execution_service.py and src/backend/routers/chat.py were reverted to the merge-base (6bbff0d87), tests kept. pytest -p no:randomly unit/test_4475_terminate_audit_race.py unit/test_3180_operational_write_audit.py: 8 failed, 33 passed (41 passed at the head).
- Red for the behaviour: the three raced tests and the redis-down test (
assert 0 == 1— no row), the second-task-id test (assert 2 == 1— two rows), and the flippedtest_3180 ..._already_terminal_writes_one_row(assert 0 == 1). - Red because the helper does not exist at the merge-base: the two
TestTheClaimHelpertests (AttributeError). - Green without the fix: the other 33 tests of the feat: a bounded, attributable machine credential for operational WRITES that survives enforced 2FA (write tier for the ops key scope) #3180 file (pre-existing behaviour).
Finer mutations at the head, same two files run each time (41 tests):
| mutation | result |
|---|---|
M2 container arm gated on cancel_won again |
5 failed |
M3 parked arm gated on cancel_won again |
1 failed |
| M4 claim never refuses (no dedupe) | 4 failed |
| M5 Redis error on the claim re-raised instead of failing open | 0 failed (41 passed) |
M6 container claim keyed on task_execution_id |
1 failed |
| M7 claim not released when the hook raises | 1 failed |
M8 container arm audits any 200 answer (incl. already_finished) |
1 failed |
M9 no row when the Redis client is None |
1 failed |
M10 route no longer passes on_ended |
14 failed |
Orderings (each row: what was run, or "read").
| ordering | audit rows | status row | evidence |
|---|---|---|---|
terminate wins the CAS, agent answered terminated |
1, status=terminated, actor = caller |
cancelled |
test_3180 test_a_container_terminate_writes_one_row (session and user key) — run, green |
| in-flight caller wins with CANCELLED or FAILED, kill landed | 1, terminated |
the caller's terminal stands | test_4475 task-raced / chat-raced — run, green; red under M1/M2 |
row already failed or success, process still live, kill landed |
1, terminated |
unchanged | test_3180 ..._already_terminal_writes_one_row (failed); reviewer probe (success) — run |
the run finished on its own, kill was a no-op (already_finished) |
0 | unchanged | test_3180 test_terminating_an_already_finished_turn_writes_no_row — run, green; red under M8 |
| second terminate, agent answers 404 | 0 | unchanged | test_3180 test_a_failed_terminate_writes_no_row[agent-404] — run, green |
second terminate, agent answers terminated again (pending entry kept, or two kills of one live process) |
still 1, the first claimant | unchanged | test_3180 repeated-container test — run, green; red under M4 |
| parked turn, dispatcher wins the CAS | 1, cancelled_while_parked |
cancelled |
test_4475 parked-raced — run, green; red under M3 |
| parked cancel, then a container terminate of the same id | still 1 (cancelled_while_parked) |
— | reviewer probe — run |
| Redis unavailable, two terminates | 2 (documented fail-open cost) | — | reviewer probe — run |
When the agent answers terminated is INFERRED from docker/base-image/agent_server/services/process_registry.py:318-343 and :420: only for a process whose poll() was None at the kill, or for a pending entry whose cancel flag it set; an exited process answers already_finished, an unknown id 404. So the new gate does not fire for a run that finished on its own. The audit row's actor is the principal of the request that took the claim, the target is the agent, and details.status is the arm's own outcome — honest in every ordering above. With two people terminating at once only the first claimant is recorded (stated in the code comment and the CSO report); before the fix only the CAS winner was.
The architecture rule (every other terminal side effect stays behind the CAS). PROVED by the diff: the only executable changes are the two on_ended call sites and the new helper. _close_dispatch_activity_cancelled(task_execution_id, cancel_won) is untouched in both arms (chat_execution_service.py:3533, :3733); the per-execution release_if_matches and _record_cancel_event were already outside the CAS at the merge-base and did not move. tests/unit/test_1804_terminal_activity_parity.py and tests/unit/test_3588_close_cancelled_claimed_activities.py are green at the head. The terminate path sends no notification and wakes no caller.
Auth. No route dependency changed: routers/chat.py differs only in a comment (:1183-1187). The route keeps Depends(get_authorized_agent) + Depends(get_current_user) + @ops_write_route, and the below-owner-tier "only a turn you started" check (routers/chat.py:1168-1177). The claim is taken only after the agent answered terminated for that id on that agent, or after the call limiter found the id parked on that agent, so a caller cannot spend the claim for an id they could not terminate; a caller with no hook (the public-link and Workspace routes) takes no claim (test_4475 test_no_hook_takes_no_claim).
Multiple workers / scheduler. Nothing in the fix is process-local: the claim lives in Redis, the row in the database. The scheduler process does not call this path (read, not run).
Critical
C1. regression diff is red: the new claim is a set(..., nx=True) outside the single-flight allowlist (confidence 10/10)
_audit_terminate_once takes its first-claim with set(nx=True, ex=…) in src/backend/services/chat_execution_service.py. tests/unit/test_1920_no_hand_rolled_single_flight.py::test_no_unlisted_hand_rolled_single_flight_lock scans for that call shape and fails for any file not in its allowlist. PROVED, on this head:
cd tests && python -m pytest -p no:randomly unit/test_1920_no_hand_rolled_single_flight.py -q
E AssertionError: New `set(..., nx=True, …)` call(s) outside redis_breaker_util.SingleFlightLock:
E services/chat_execution_service.py
CI agrees: regression diff reports one new failure introduced by the head, this test. It is the PR's own red, not a flake.
The guard's own message gives the two ways out: adopt SingleFlightLock, or add one justified allowlist entry for a deliberate non-lock use. This claim is a write-once dedupe marker with a 24 h life, not a lock that is released when work ends, so an allowlist entry with that reason looks like the right one; the author decides. My first pass missed this guard: the targeted run did not include it.
Informational (non-blocking)
- The "claim is released if the audit write raises" branch cannot fire for the real hook (confidence 8/10).
platform_audit_service.logcatches every exception and returnsNone(src/backend/services/platform_audit_service.py:211-218), so_audit_terminatenever raises. When the audit insert fails, the claim stays for 24 h and a repeat writes nothing. Probe (scratch,db.create_audit_entrymade to raise): first terminate200,rows: 0,claim: 1,ttl: 86400; second terminate200,rows: 0. This is not a regression — before the fix a failed write was lost the same way — butdocs/memory/feature-flows/execution-termination.md:485, the CSO report's A10 row andtest_4475:163-178describe a recovery production does not have. Either let the hook returnlog's event id and release the claim onNone, or reword the docs. - The fail-open branch for a Redis call that raises is untested (confidence 8/10).
get_breaker_rediscaches the client after the first successful ping (src/backend/redis_breaker_util.py:58), so an outage after boot shows up as.setraising, not as aNoneclient — the branch atchat_execution_service.py:3369-3371. Mutation M5 (re-raise there) leaves all 41 tests green. The code is right: a probe with a client whose.setraisesConnectionErrorwrote the row. Add that case besidetest_with_redis_down_the_row_is_still_written. - Stale docstrings in
tests/unit/test_3180_operational_write_audit.py:522-526and:611-614still say only the call that wins the CANCELLED CAS ended the turn. - The new test file is not in
tests/registry.json(convention, not enforced: 154 of the 222 unit test files added ondevin the last two weeks are registered). - The live re-test is still open. The PR's manual Test Plan item is unchecked, and the issue states its cause was "read from source, not confirmed by server logs". The fix no longer depends on the CAS at all once the agent answers
terminated(which the issue observed), so the reported case is covered by construction; row B20 still needs a re-test on the next release candidate by someone other than the author. - The deferred items have no issue number. The PR body lists "a terminated chat turn ends
failed" (the issue's secondary point) and "dedupe in the agent's process registry" as follow-ups without filing them;Fixes #4475will promote and later close the issue that carries the first one. - Feature-flow Revision History (
docs/memory/feature-flows/execution-termination.md:573) has no entry for the new section; the PR body carries noJourney Impact:line (the gate passes: "allowed for now").
Low confidence (appendix):
- Same class, not introduced here: when the kill lands but the answer is lost (the backend's 15 s proxy timeout against the agent's up-to-7 s wait plus its sweep, or a
termination_unconfirmed5xx), no row is written and the retry gets a 404. Reachability not checked. - The production and hosted compose files run Redis with
allkeys-lru(docker-compose.prod.yml:668-669); under memory pressure the claim can be evicted, which fails toward a duplicate row, not a missing one. - The execution id can be re-used by an in-line retry on the agent; a second, distinct kill of the same id within 24 h would be deduplicated. No reachable path found (the first terminate leaves the row terminal).
- The claim is a blocking Redis call on the event loop (1 s socket timeout, once per terminate), as at other call sites of the same client.
Methodology (validate-pr)
- Lane: B (no auth/credential/CI/image path; 6 deleted lines under
tests/, all in the declared flipped test). No schema change. - Issue link / closing keyword:
Fixes #4475in the body; the issue exists,priority-p2,type-bug,theme-security,status-in-progress. Base isdev. One commit, conventionalfix(audit): …. - Journey Impact:
scripts/ci/check_journey_impact.pyon the PR's body and SHAs →PASSED(no declaration; allowed for now). - Docs tier: bug fix — a commit message would do; the PR also updates the feature flow (new "Audit Row" section),
docs/memory/requirements/security.md(theexecution_terminateline) and adds a learnings fragment in the required shape. No architecture file needs a change (no endpoint, schema or service added). - Requirements: updated (
docs/memory/requirements/security.md:58). - Tests registered: no (Informational 4).
- Security report:
docs/security-reports/cso-diff-2026-11-13-4475-terminate-audit-race.{md,json}present, no findings; its A10 row repeats the claim in Informational 1. - Security greps (§4.1-4.6, §4.9, §4.10): no key, email, public IP, env file, hardcoded secret, credential file, new env var, symlink, mode change or host path in the diff. Enterprise-docs pattern over the added lines: no hit.
- Audit rows (§4.11):
audit_rows.py→➖ no gated write handler touched(the handler's gate is the access-tier dependency, outside the script's owner/admin set). The handler does write its row through_audit_terminate. - Cross-surface drift (§5.5): nothing printed.
routers/chat.pychanged only comment lines; its# mcp:header already lists terminate as UI-only; no agent-server twin of the changed service file. - Product fit (6.6): bug, not incubating; the one user-visible change (the row now appears) is the issue's "Expected". No new route, tool, setting or env key.
- Targeted runs at the head (local,
-p no:randomly, SQLite): the two PR files 41 passed; 19 further files covering the touched functions and the guards that read them (test_1804_terminal_activity_parity,test_3588_…,test_2433_dispatch_wiring,test_ent155_chat_cancel,test_ent875_exec_control_routes,test_2795_room_stop, bothtest_4284_*,test_ent279_scrub_parity,test_2106_…,test_2789_…,test_terminate_async_executor,test_timeout_termination,test_3180_ops_write_scope,test_3180_ops_write_route_census,test_1310_auth_wiring,test_186_enumeration_uniformity,test_2996_human_only_routes,test_2338_journey_catalog): 800 passed, 19 errors. All 19 errors are intest_3180_ops_write_route_census.py, which imports the full app and fails locally on a missingopentelemetry.exportermodule; the same file gives the same 19 errors at the merge-base, so they are environmental.test_1560_agent_redis_key_parity,test_1560_breaker_cleared_on_lifecycle,test_models_centralized,test_2019_pytest_timeout_guard: 39 passed. - CI on head
c03104530: 19 pass, 6 skipped, 1 failed:regression diff(C1).pytest (head, seed 12345)passes because it only produces the results file. Pass also:Analyze (python),Analyze (javascript-typescript),CodeQL,schema-parity,verify-non-root,gitleaks,journey-impact,prod-image-smoke,backend boots without enterprise submodule,cached dev baseline, the guard and lint jobs. Skipped:e2e,journey-smoke,report,auto-merge, the two fallback pytest jobs. - Merge conflicts with
dev: none (git merge-treeof the head onto the currentorigin/devtip54f663a61;devchangeddocs/memory/requirements/security.mdafter the merge-base and it still merges). GitHub reportsMERGEABLE,REVIEW_REQUIRED, no reviews, no labels.
Not checked
- The full pinned-guard list of the review skill's Step 2.5 (303 files by its pattern) and the full unit suite — only the files named above were run; the diff adds no route, protected path or registry entry. CI's
regression diffis the signal for the rest; it reports only the C1 guard. - PostgreSQL: every local run was the
[sqlite]parametrisation. - A live instance: no terminate against a running agent with a working model, no second uvicorn worker, no real Redis. The agent's answers are a stub in every test; when the real agent answers
terminatedwas read from source, not run. Agents on an older base image were not considered. - The CSO report's variant-analysis claim (no other audit hook gated on a CAS result) was not re-derived.
/cso --diffwas not re-run; frontend, MCP server and the scheduler were not exercised (the diff does not touch them).
webmixgamer
left a comment
There was a problem hiding this comment.
Requesting changes on c03104530: one blocking finding (C1, regression diff is red on the single-flight guard test_1920, which the new set(nx=True) claim trips). The fix logic is confirmed by mutation and needs no change; one justified allowlist entry, or the shared lock helper, clears it. The /review + /validate-pr report is in the comment above.
…ils (#4475) - The route hook returns platform_audit_service.log's event id; a None result releases the terminate_audit claim, so the next terminate records the turn. log swallows insert errors, so the raise branch alone never fired for the real hook. - Allowlist the terminate_audit dedupe claim in the #1920 single-flight guard (once-guard, 24 h TTL, deleted only on a failed audit write). - Tests: the Redis claim raising still writes the row; a failed audit insert through the real route releases the claim and a repeat writes one row. - Refresh the #3180 test docstrings, the feature-flow Audit Row text and Revision History, the CSO report A10 row; register the #4475 test file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Pushed
|
…-audit-race # Conflicts: # tests/registry.json
Re-review — APPROVE (head
|
| point | claim | verified how | result |
|---|---|---|---|
| I1 | the hook returns log's event id; a None releases the claim; a route-level test covers it; docs and the A10 row updated |
read chat_execution_service.py:3374-3387 and routers/chat.py:1188-1189; ran the new test; two mutations (below); probes |
done. With the release removed (M11) the new test is red; with the route hook dropping log's return value (M12) three repeat tests are red. Probe: audit store down for two terminates → 0 rows, claim absent; store back up for two more → 1 row, claim present. log has exactly one None path, the swallowed exception (platform_audit_service.py:209, :218), so None never means "skipped on purpose". The only hook is the route's; the queued arm ignores the return value (chat_execution_service.py:3329-3330). Flow doc :485 and both CSO report files reworded. |
| I2 | a test where .set raises ConnectionError; M5 now red |
re-ran M5 | done. M5 (re-raise on the Redis error): 1 failed, 42 passed, the new test_with_the_redis_claim_raising_the_row_is_still_written (test_4475:134). |
| I3 | both #3180 docstrings updated | read the diff | done (test_3180_operational_write_audit.py:522-526, :611-614). |
| I4 | test file registered | git diff c03104530 57f0d5a2f -- tests/registry.json: 13 added lines, 0 deleted; after the merge, against dev: 13 added, 0 deleted; the file parses, 789 entries, no duplicate; test_2338_journey_catalog.py 51 passed |
done, additive. |
| I5 | live re-test still open | — | still open, as the author says. Row B20 needs a re-test on the next release candidate. |
| I6 | deferred items tracked in #4551 | read the issue | done. #4551 (open, type-bug, priority-p3) carries both: a terminated chat turn ends failed, and repeat terminates are indistinguishable at the agent, with acceptance criteria. The PR body links it. |
| I7 | Revision History row added | read the diff | done (execution-termination.md:577). |
| merge | ca3654987 merges dev |
compared the merge commit's tree with a plain automatic merge of 57f0d5a2f and 76f640054 |
clean. The only file that differs from the automatic merge is tests/registry.json, which conflicted and was resolved by keeping both sides. dev had not touched the two backend files or the two edited test files. |
Mutations at ca3654987 (the two PR test files, 43 tests, all green unmutated):
| mutation | result |
|---|---|
M2 container arm gated on cancel_won again |
7 failed |
M3 parked arm gated on cancel_won again |
1 failed |
| M4 claim never refuses (no dedupe) | 4 failed |
| M5 Redis error on the claim re-raised | 1 failed (was 0) |
M6 container claim keyed on task_execution_id |
1 failed |
| M7 claim not released when the hook raises | 1 failed |
| M8 container arm audits any 200 answer | 1 failed |
M9 no row when the Redis client is None |
1 failed |
M10 route no longer passes on_ended |
16 failed |
M11 claim not released when the hook returns None (new) |
1 failed |
M12 route hook drops log's return value (new) |
3 failed |
| M13 claim released after every write, also a successful one (new) | 4 failed |
Probes re-run at ca3654987 (scratch only): a client whose .set raises → 1 row; parked cancel then container terminate of the same id → 1 row (cancelled_while_parked); Redis client None, two terminates → 2 rows (the documented fail-open cost); Redis up, two terminates by two credentials → 1 row; lost CAS → the row keeps the other writer's status and error, slot release per execution, 1 audit row; a slow write that succeeds while a second terminate arrives → the second is refused, 1 row, claim kept.
New in this push
None of these blocks. The release-on-None is new behaviour, so it was reviewed as new code.
- The release is a tokenless
DELof a constant-value claim (chat_execution_service.py:3368sets"1",:3374-3379deletes the key unconditionally). That is the shapetest_1920's own docstring calls the worst class for a lock, and the neighbouring once-guard rows in its allowlist say "no release, no token" while this one has a release. Control flow keeps it safe in normal operation: only the caller whoseSET NXsucceeded reaches_release. It can delete another caller's claim only if the claim vanished while the first write was in flight (a flush or an eviction; expiry cannot do it, 24 h against one insert) and a second caller took it. Probe: first caller claims and its write stalls, the key is deleted, a second caller claims and writes, the first write returnsNoneand releases, a third caller writes → 2 rows. The cost is one duplicate row in a case that should not occur. Either say so in the allowlist reason, or store a per-call token and compare before deleting (redis_breaker_utilhaslock_token_matches). - A write that commits and then raises releases the claim (
chat_execution_service.py:3386-3387). Probe: the insert lands, the call then raises,logreturnsNone; a repeat terminate → 2 rows. Same direction as the stated policy (a duplicate costs less than a missing row). - "So the next terminate records the turn" is true for some turns only (flow doc
:485, the registry description, the allowlist reason). A later terminate reaches the claim again while the turn is still parked or pending, or when two kills overlap. After a completed kill the agent drops the process from its registry (docker/base-image/agent_server/services/process_registry.py:416-418) and answers 404, which is an error arm and writes nothing — INFERRED from source, not run; the new route-level test uses a stub agent that always answersterminated. A second terminate that arrives while a failing write is still in flight is refused and is not recorded either (probe: hooks runA-failed, then nothing for the concurrent call, then a third call records). So a failed audit insert on the common path still loses the row, as it did before this PR. Wording only. platform_audit_service.log's docstring now disagrees with a caller: "Callers should not branch on the return value" (platform_audit_service.py:127). The hook's result is branched on, for the claim only; the response does not change. A one-line docstring update would keep the next reader from "fixing" it.- A Redis error during the release is swallowed without a log line (
chat_execution_service.py:3376-3379). Probe:.deleteraises → both terminates answer 200, 0 rows, the claim stays 24 h. Same outcome as before this push for that case; a warning line would make it visible.
CI and merge state
gh pr checks 4550 on head ca3654987: 22 pass, 5 skipped, nothing failed or pending. pytest (head, seed 12345), pytest (base, inline fallback) and regression diff all pass, with the security, schema, image and guard jobs. Skipped: e2e, journey-smoke, report, auto-merge, pytest (push, absolute failures).
GitHub: MERGEABLE, review decision CHANGES_REQUESTED (the earlier request), no labels. The branch already contains the dev tip 76f640054.
Not checked
- The 766 unit test files outside the 703-file set, and PostgreSQL (local runs are SQLite only).
regression diffis the signal for the unit files and it passes on this head. - The private submodule is not mounted in the reviewed tree: its own twin of the single-flight guard and any caller there were not looked at.
- A live instance, a second worker, a real Redis and the real agent: the agent's answers are a stub in every test, so point 3 above is read from source.
- The read-recording plugin sees
open-based reads only; a guard that shells out togit grepwould not be listed among the 98 (it would still have been run if it matched the set).
webmixgamer
left a comment
There was a problem hiding this comment.
Approving ca3654987: the single-flight guard passes with one narrow allowlist entry, the release-on-failed-write behaviour is covered and every mutation goes red, and CI is green including regression diff. Five non-blocking notes on the new release path are in the re-review comment above. Row B20 still needs a re-test on the next RC by someone other than the author.
…-audit-race # Conflicts: # tests/registry.json
Summary
execution_terminateaudit row when the agent answersterminated(container arm) or the turn is found parked (parked arm). Before, the row depended on winning the CANCELLED compare-and-swap, which the in-flight/task//chatcaller (or the parked dispatcher) usually wins, so on a real instance the row was lost.terminate_audit:{agent}:{execution_id}(SET NX, 24h TTL) on an id the agent or the call limiter verified. Fails open when Redis is unreachable; the claim is released when the audit write fails (the hook returnslog's event id;Noneor a raise releases it).Behaviour change
test_a_container_terminate_of_a_turn_already_terminal_writes_no_row(#4035) becomes..._writes_one_row: the agent killed a live process whose row was already terminal, so the terminate ended work and is recorded.Changes
src/backend/services/chat_execution_service.py:_audit_terminate_once, proxy and parked armssrc/backend/routers/chat.py: commenttests/unit/test_4475_terminate_audit_race.py(new),tests/unit/test_3180_operational_write_audit.pydocs/memory/requirements/security.md,docs/memory/feature-flows/execution-termination.md, learnings fragment, CSO diff report (no findings)Test Plan
cd tests && pytest unit/test_4475_terminate_audit_race.py -v -p no:randomly/task, terminate it, confirm oneexecution_terminaterow inGET /api/audit-logOut of scope (follow-up #4551, needs an agent-image rebuild)
failed(agent/api/chathas no cancel relabel).Fixes #4475
🤖 Generated with Claude Code