Skip to content
This repository was archived by the owner on Oct 24, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -129,9 +129,6 @@ trust-conformance-live:
adversarial-battery-live:
uv run --with-requirements requirements.txt python scripts/adversarial_battery.py

scoped-smoke-loop:
uv run --with-requirements requirements.txt python scripts/scoped_smoke_loop.py $(SCOPED_SMOKE_ARGS)

# Railway deploy gate. Run under `railway run` (or with DATABASE_URL +
# PUBLIC_URL exported) to check migration parity and live posture together.
railway-preflight:
Expand Down
24 changes: 14 additions & 10 deletions docs/PROOF_MATRIX.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,18 +188,22 @@ Both write real rows to the target. Point them at staging.
| Command | Proves | Requires |
|---|---|---|
| `make trust-conformance-live` | Golden path; sequential replay; 15 identical concurrent requests exposing one receipt identity or explicit `idempotency_in_progress`, followed by a completed replay and one charge; a changed payload under a reused key conflicting rather than replaying; budget denial; expired and forged permit rejection; receipt and audit-chain verification; tenant isolation against a directly-supplied foreign wallet and permit id | `AGENT_MIDDLEWARE_API_KEY`; set `AGENT_MIDDLEWARE_API_URL` or it defaults to production |
| `make scoped-smoke-loop` | Authority readable for the caller's own wallet; scoped permit issued and carrying a nonce; governed invoke; the debit matching the receipt's own stated cost; replay returning the same receipt and charging nothing further; a changed payload under a reused key refused; an out-of-scope but genuinely registered tool denied and not charged; the portable receipt carrying a signature, key id, and named canonicalization | `AGENT_MIDDLEWARE_API_KEY` holding a **wallet-scoped** key (the loop refuses a bootstrap admin key); `--tool` required off loopback |
| `python scripts/constant_test_loop.py` | Scoped permit sized from the tool's advertised `creditsPerCall`; governed invoke; signed success receipt with a ledger entry; replay returning the same `receipt_id` with no second debit; an out-of-scope but genuinely registered tool refused with `permit_tool_not_allowed` and charged nothing | `CI_SMOKE_AGENT_KEY` holding a **wallet-scoped** key (self-provisions on loopback when absent); `--tool` and `--tool-args` required off loopback |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore portable receipt verification in the consolidated loop

The deleted smoke loop fetched /v1/receipts/{receipt_id}/portable and checked its signature, key ID, and canonicalization, but run_constant_test now stops after checking receipt outcome, ledger linkage, and charge. Consequently, a deployment with a broken portable receipt endpoint or missing verification metadata still prints ALL INVARIANTS HELD; the matrix's “signed success receipt” wording overstates what this replacement actually proves. Restore those checks before treating this as the sole continuous receipt proof.

AGENTS.md reference: AGENTS.md:L68-L80

Useful? React with 👍 / 👎.

| `make adversarial-battery-live` | Wallet isolation, invalid-key rejection, forged-receipt rejection, permit key binding, expired permits, revoked keys, replay idempotency; always revokes keys it minted | `API_URL` (no default, by design) and `BOOTSTRAP_KEY` |

`scoped-smoke-loop` is the one built to run continuously: it needs no admin
credential, so the key it runs on cannot mint keys or read another tenant's
audit trail if the runner holding it is compromised. That is also its limit —
tenant isolation, audit-chain verification, and refund reconciliation are
absent from it precisely because a scoped key must not be able to reach them,
and they stay in `trust-conformance-live` for an operator to run deliberately.
It reports SKIP rather than a false PASS for the out-of-scope denial when the
deployment registers only one tool, since invoking an unregistered name proves
"tool not found" rather than "the permit refused it".
`constant_test_loop.py` is the one built to run continuously: it needs no

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve continuous execution in the surviving loop

The surviving script executes run_constant_test exactly once and exits, while this newly updated proof matrix calls it the loop “built to run continuously.” The deleted scoped_smoke_loop.py was the only implementation of --interval, --max-iterations, and continuing after transient failures, so deployments replacing make scoped-smoke-loop with this documented command silently lose continuous monitoring. Either retain equivalent scheduling behavior in the consolidated script or describe the required external scheduler.

Useful? React with 👍 / 👎.

admin credential, so the key it runs on cannot mint keys or read another
tenant's audit trail if the runner holding it is compromised. That is also its
limit — tenant isolation, audit-chain verification, and refund reconciliation
are absent from it precisely because a scoped key must not reach them, and
they stay in `trust-conformance-live` for an operator to run deliberately.

It skips rather than false-passes the out-of-scope denial when the deployment
registers only one tool, since invoking an unregistered name proves "tool not
found" rather than "the permit refused it". Off loopback it refuses to start
without `--tool` and `--tool-args`: arguments derived from a schema satisfy a
tool's declared shape but not its semantics, and the derivation picks the
first enum member, which for a consequential tool could be `delete`.

The battery reports SKIP — never a false PASS — for MCP-invocation checks when
the deployment exposes no invokable `golden-path-echo` tool. It does not
Expand Down
28 changes: 27 additions & 1 deletion docs/constant-test-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,32 @@ The script self-provisions an agent key via `/v1/dev-keys/self-provision` when n

## Production Use

Off loopback the loop refuses to start without `--tool` and `--tool-args`
(or `$CI_SMOKE_TOOL` / the equivalent flags). This is deliberate, not a
papercut:

- **Tool selection.** `partner.echo` and `partner.notes.write` are tried in
order when present, but which tool a monitor invokes on every run is an
operator's decision, not the registry ordering's.
- **Payload.** Arguments derived from a tool's `inputSchema` satisfy its
declared *shape*, not its *semantics*, and the derivation fills required
fields from types, defaults, and the **first enum member** — which for a
consequential tool could be `delete`. Schema validity is not evidence of
safety. Pass `--tool-args '{}'` if the tool genuinely needs no arguments;
the point is that it be a decision.

The permit is sized from the selected tool's advertised
`annotations.creditsPerCall` rather than a fixed cap, so pointing the loop at
a pricier tool does not fail with `permit_budget_exceeded` on a healthy
deployment.

```bash
python scripts/constant_test_loop.py \
--api-url https://api.thisisatest.tech \
--tool partner.echo \
--tool-args '{"text": "constant test loop"}'
```

Set `CI_SMOKE_AGENT_KEY` for a pre-provisioned agent credential. Optionally provide `CI_SMOKE_WALLET_ID` and `CI_SMOKE_KEY_ID` for faster startup (the script will fetch them from the API if not provided):

```bash
Expand All @@ -36,7 +62,7 @@ export CI_SMOKE_AGENT_KEY="$(python scripts/partner_api_key_bootstrap.py \
--agent-id ci-smoke-agent \
--key-name constant-test-loop \
--budget-credits 5000 \
--json | jq -r .api_key)"
--key-only)" # or --json | jq -r .api_key

export CI_SMOKE_WALLET_ID="$(python scripts/partner_api_key_bootstrap.py \
--api-url https://api.thisisatest.tech \
Expand Down
Loading