Skip to content

fix(gate): re-prove authority at the commit point (gauntlet revoke-during-spend) - #18

Open
FetchDoge wants to merge 4 commits into
mainfrom
fix/commit-time-gate
Open

FetchDoge wants to merge 4 commits into
mainfrom
fix/commit-time-gate

Conversation

@FetchDoge

Copy link
Copy Markdown
Owner

The bug (found by gauntlet: revoke-during-spend)

The agent gets a lease and passes the gate. 100 ms later the operator engages the e-stop, while the payment is still being prepared (a 300 ms slow rail). The payment still goes out. Gauntlet scores this SILENTLY_EXECUTED.

Root cause

Interlock.permit() is a context manager that checks e-stop, lease and faults once, at block entry, then yields. The real effect happens later inside the caller's block, and nothing re-checks authority at that point. Nothing orders an estop.engage() or leases.revoke() against an effect that's about to start, either. This is a check-then-act race: authority is proven at T0 and used at T0 + 300 ms.

Fix (fail-closed)

  • New Interlock.execute(lease, action, target, effect): it re-runs the full gate at the commit point and registers the effect as in flight atomically, under one condition lock (InFlightGate). If the e-stop or revocation state isn't clear at that moment, it raises and effect never runs.
  • EStop.engage() and LeaseManager.revoke() now take the same lock: they set their flag, then wait for already-started effects to drain. When an e-stop call returns, nothing is mid-effect and nothing new can start. Effects still run concurrently, because only check-and-register is serialised. A halt raised from inside an effect doesn't deadlock on its own thread.
  • permit() keeps its API: it now documents that it checks at entry only. If authority is lost inside the block, it appends a violations ledger record on exit, so the gap is never silent.
  • Ledger change: permits and denials records gain a phase field (entry or commit).

Tests (written first, failing on main)

tests/test_inflight.py has 7 tests: the gauntlet e-stop case, the revoke case, a clear-path control, e-stop waiting for a running effect, a 16-thread race (no effect may start after engage() returns; run 10× locally, all green), the permit() violation record, and no deadlock when a halt is raised inside an effect.

Before (main)

E   AttributeError: 'Interlock' object has no attribute 'execute'
E   AssertionError: race never exercised
6 failed

After

75 passed in 1.11s      (68 existing + 7 new); ruff check/format clean

Gauntlet, run locally

Setup revoke-during-spend Full interlock suite
vendored interlock (ca1718c) 1 silent execution 1 silent / 18
this branch, gauntlet adapter unchanged 1 silent execution not run
this branch + adapter calls ilk.execute() for the POST (local, uncommitted) 0 silent, 1 refused by Interlock 0 silent / 18, 3/3 controls passed

Read this before merging: this PR fixes the gate, but gauntlet's interlock adapter still does its POST inside a permit() block. For the CI case to go green, gauntlet needs a follow-up PR that (a) bumps vendor/interlock and (b) switches the adapter to ilk.execute(). Both change harness files and require re-pinning gauntlet's evidence, so I haven't opened it.

Gauntlet's separate honesty-gate failure (LICENSE changed after the pinned harness) is unrelated to this fix and needs Marcus's decision.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T18:33:00.027040Z f2742a9 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ee4c5f5242

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/interlock/__init__.py Outdated
Comment on lines +171 to +173
with self._gate.cond:
base = self._check(lease, action, target, phase="commit")
self._gate.enter(lease.id)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Synchronize gating fault reports with commit registration

When another thread calls FaultBus.report() for a major or critical fault, it does not acquire _gate.cond, so the report can land after _check() observes no fault but before enter() registers the effect. In that interleaving the gating fault exists before the commit point, yet the effect still starts, unlike the ordering guaranteed for e-stop and revocation. Wire fault reporting into the same gate or otherwise make the fault check-and-registration atomic relative to reports.

Useful? React with 👍 / 👎.

Comment thread src/interlock/estop.py
Comment on lines +83 to +87
def _busy(self, lease_id: str | None) -> bool:
me = threading.get_ident()
return any(
tid != me and (lease_id is None or lid == lease_id) for lid, tid in self._in_flight
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Prevent concurrent in-effect halts from deadlocking

If two concurrent effects both call estop.engage(), each drain ignores only its own thread's entry and waits for the other effect to leave; neither effect can leave because both are blocked in drain(). The same mutual wait can occur when concurrent effects revoke each other's leases, so an emergency halt can deadlock indefinitely instead of returning after effects drain.

Useful? React with 👍 / 👎.

Comment thread src/interlock/__init__.py Outdated
Comment on lines +174 to +176
try:
self.ledger.append("permits", base)
return effect()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Serialize concurrent execute ledger appends

Concurrent execute() calls append to the shared Ledger outside the gate, but Ledger.append() is not thread-safe. In particular, simultaneous first appends to permits can each emit a series descriptor and overwrite _series["permits"], causing the closing index to undercount records; this is reproducible with a path-backed ledger and concurrent executions. Since the new gate explicitly allows effects to run concurrently, its permit logging must be serialized or the ledger must be made thread-safe.

Useful? React with 👍 / 👎.

Comment thread src/interlock/__init__.py
Comment on lines +153 to +155
reason = None
if self.estop.is_engaged:
reason = "estop"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Record transient e-stop engagements during permit blocks

If an e-stop is engaged and then deliberately released before the permit() block exits, this final-state check sees it as clear and emits no violation. That contradicts the new contract that an engagement while the block runs is never silent, and loses the audit record for effects performed during that interval; tracking an engagement generation/event rather than only is_engaged would preserve it.

Useful? React with 👍 / 👎.

Comment thread src/interlock/__init__.py Outdated
Comment on lines +176 to +178
return effect()
finally:
self._gate.leave(lease.id)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep deferred effects in flight until they actually execute

When effect is an async or generator function, this call only creates a coroutine or generator and the finally block immediately removes it from the in-flight gate; the body can then be awaited or iterated after an e-stop or revocation has returned. The generic Callable[[], T] signature accepts these callables without warning, so agent code can bypass the new commit-point guarantee; reject deferred results or provide an API that keeps them registered through completion.

Useful? React with 👍 / 👎.

Comment thread tests/test_inflight.py Outdated
Comment on lines +33 to +36
t = _later(0.1, lambda: ilk.estop.engage("gauntlet"))
time.sleep(0.3) # slow rail: authority was fine at the start
with pytest.raises(EStopEngaged):
ilk.execute(lease, "pay", "merchant", lambda: calls.append("paid"))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wait for scheduled halts instead of relying on sleep timing

These tests assume the timer callback has run after a fixed 300 ms sleep, but threading.Timer provides no scheduling deadline; on a sufficiently loaded CI worker the main thread can wake and call execute() before the timer thread engages the stop, making the success path fail pytest.raises. The revocation test has the same race, so synchronize on an event indicating that the halt completed rather than relying on elapsed wall time.

Useful? React with 👍 / 👎.

Comment thread src/interlock/__init__.py
if reason:
self.ledger.append("violations", {**base, "reason": reason})

def execute(self, lease: Lease, action: str, target: str, effect: Callable[[], T]) -> T:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Route the documented action path through execute

The new safe API is absent from the README and quickstart, which still instruct users to perform effects inside permit() even though that path retains the revoke-during-spend race this change is intended to fix. Consequently, integrations following the project's supported examples receive no protection from this patch; update the public examples and guidance to use execute() for irreversible effects, as also required for behavioral changes by CONTRIBUTING.md.

Useful? React with 👍 / 👎.

Comment thread src/interlock/__init__.py Outdated
Comment on lines +173 to +174
self._gate.enter(lease.id)
try:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Install in-flight cleanup before registration can escape

The effect is inserted into _in_flight before execution enters the try/finally region. If KeyboardInterrupt or another asynchronous exception arrives while the condition context is exiting or in the gap before the try, leave() is skipped and the stale entry remains forever, causing later e-stop engagement or revocation to block indefinitely. Establish the cleanup guard before registering the effect so every successful enter() is paired with leave().

Useful? React with 👍 / 👎.

@FetchDoge

Copy link
Copy Markdown
Owner Author

Codex review response — all eight findings worked

Every finding was verified against the code at ee4c5f52 before patching; all eight held. Six are patched in 72e0bc17, the test-timing finding in 5853f51e, and the docs finding in f2742a98. Nothing else changed, and the PR is left unmerged.

P1 — Synchronize gating fault reports with commit registration · Patched in 72e0bc17. Held: a FaultBus.report() from another thread could land between execute()'s fault check and its in-flight registration, so the effect started with a gating fault already reported. FaultBus now takes the in-flight gate's lock in report()/clear() (wired in Interlock.__init__), so a report that returns before an effect commits is always seen by that effect's check. Regression test: test_fault_report_is_ordered_against_commit_registration.

P1 — Prevent concurrent in-effect halts from deadlocking · Patched in 72e0bc17. Held — reproduced: two effects that both called estop.engage() deadlocked permanently (both threads still blocked after 3 s); mutual in-effect revocations had the same shape. InFlightGate.drain() now registers the draining thread, and effects suspended inside a halt no longer block each other's drains, while effects still running normally are waited on. Regression tests: test_concurrent_halts_from_inside_effects_do_not_deadlock, test_concurrent_in_effect_revocations_do_not_deadlock.

P1 — Serialize concurrent execute ledger appends · Patched in 72e0bc17. Held — reproduced with a path-backed ledger, 32 threads × 25 execute() calls: the closing index counted 578 of 800 permits records and 4 permits series descriptors were written. Ledger.append()/close() are now serialised with an internal lock; the same run post-fix gives 800 records, index count 800, exactly 1 descriptor. Regression test: test_concurrent_execute_ledger_index_matches_records.

P2 — Record transient e-stop engagements during permit blocks · Patched in 72e0bc17. Held: engage-then-release inside a permit() block left no violations record. EStop now counts engagements (EStop.engagements), and permit() compares the count at block exit instead of only the final is_engaged state. Regression test: test_transient_estop_inside_permit_block_records_violation.

P1 — Keep deferred effects in flight until they actually execute · Patched in 72e0bc17. Held — reproduced: execute() returned a coroutine unstarted, deregistered it immediately, and its body ran after estop.engage() had returned; a generator behaved the same. execute() now refuses deferred callables with TypeError before the gate runs, and a deferred result returned by a plain callable is closed unstarted and refused the same way. Regression test: test_execute_refuses_deferred_effects.

P2 — Wait for scheduled halts instead of relying on sleep timing · Fixed in 5853f51e. Both slow-preparation tests now wait on an event set after the halt lands (5 s bound) instead of assuming the threading.Timer fired within the 300 ms sleep.

P1 — Route the documented action path through execute · Fixed in f2742a98. The README quickstart and docs/guides/quickstart.md now run irreversible effects through execute() and describe permit() as entry-check-only; the guide's Python blocks were executed end-to-end as one session after the edit (exit 0). CHANGELOG.md gained the missing execute() entry plus these fixes, per CONTRIBUTING.md.

P2 — Install in-flight cleanup before registration can escape · Patched in 72e0bc17, structural. Held: enter() ran before the try/finally was established, so an asynchronous exception in that gap would leak a stale in-flight entry and block later halts forever. execute() now establishes its cleanup guard (a registered flag inside try/finally) before registration, so every successful enter() is paired with leave(). No deterministic regression test exists for this one: triggering it needs an asynchronous exception delivered inside a two-step window, which a test cannot schedule — the fix removes the window by construction rather than narrowing it.

Full suite on the updated branch (f2742a98): 81 passed — 75 at the PR head plus 6 new regression tests — run 6× locally with no flakes. ruff check and ruff format --check clean (CI runs the same on Python 3.11–3.13). No finding is left unresolved and there is no blocker; ready for the merge decision.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f2742a98a2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/interlock/__init__.py
Comment on lines +201 to +203
inspect.iscoroutine(result)
or inspect.isgenerator(result)
or inspect.isasyncgen(result)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject custom awaitables before releasing the gate

When a plain effect returns a custom object implementing __await__ (including framework-specific awaitables), all three predicates are false, so execute() returns it and the finally block unregisters the effect. The caller can then await the object after estop.engage() or revoke() has returned, running its deferred body without authority; use the general awaitable protocol check (such as inspect.isawaitable) rather than recognizing only native coroutine objects.

Useful? React with 👍 / 👎.

Comment thread README.md
# EXERCISE: every action passes the gate. execute() re-proves the gate at
# the commit point, atomically with starting the effect — a halt that lands
# while the action is being prepared refuses it instead of racing it.
ilk.execute(lease, "write", "brief.md", effect) # raises EStopEngaged, LeaseNotIssued, LeaseRevoked, LeaseExpired, or FaultActive otherwise

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Define the effect used by the README quickstart

Running the README example as shown raises NameError because effect is never defined, so the newly documented commit-point path cannot reach the gate. Define a concrete function or use a self-contained lambda as the detailed quickstart does.

Useful? React with 👍 / 👎.

Comment thread src/interlock/estop.py
def _busy(self, lease_id: str | None) -> bool:
me = threading.get_ident()
return any(
tid != me and tid not in self._halting and (lease_id is None or lid == lease_id)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep halting effects visible to unrelated drains

When an effect for lease A blocks inside revoke(lease_b), its thread is placed in _halting; a concurrent external revoke(lease_a) therefore ignores the still-running A effect and returns immediately. Once B drains, the A effect resumes and can perform more work after revocation of A has returned; a global e-stop can return prematurely in the same way. Only effects participating in the actual wait cycle should be excluded, while unrelated scoped drains must continue counting the enclosing effect.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant