Repository navigation
feat(guardrails): guardrail event store, own-key ingest and owner-tier read (abilityai/trinity-enterprise#165) - #4424
Closed
AndriiPasternak31 wants to merge 14 commits into
Closed
AndriiPasternak31 wants to merge 14 commits into
AndriiPasternak31 wants to merge 14 commits into
Conversation
… its store (Abilityai/trinity-enterprise#165) One row per guardrail event an agent reported, unique per (agent, event_id) so a resend is idempotent (ON CONFLICT DO NOTHING ... RETURNING on both dialects). SQLite migration plus Alembic 0123 chained after 0122 (one head), DDL in schema.py/tables.py, indexes for the per-agent read, the per-execution counts and the retention sweep. execution_id carries no FK. Rows follow the agent through AGENT_REFS (CASCADE: renamed with it, purged with its hard delete).
…/trinity-enterprise#165) POST /api/agents/{name}/guardrail-events takes a batch from the agent's own key only (the heartbeat predicate, outside get_current_user): 413 past 500 events or 256 KiB, 422 only for a malformed envelope, 429 + Retry-After past the per-agent cap. Each event is validated on its own: an unknown kind or hook is stored as 'other' with the raw value, code is an open string, a malformed event is counted in 'rejected' and never fails the batch, an over-long execution_id is nulled. Only allowlisted fields are kept, free text sanitized and bounded. GET is owner tier with the uniform 404 and returns events newest first, a keyset cursor and the reporting state (never / reporting / stale) kept in Redis and cleared on teardown. count_by_execution and count_by_agent are the seams the execution-status and fleet consumers call. Both route censuses list the routes.
…us guard (Abilityai/trinity-enterprise#165) guardrail_event_retention_days (default 90, 0 disables) is a registered retention window: validated write path, reported by GET /api/settings/retention, not a community-floor key. The cleanup cycle prunes rows past it by the server receive time (an agent-reported timestamp must not make a row immortal), behind its own _guard_allows site, in bounded chunks.
…rchitecture catalogs and the guardrails flow (Abilityai/trinity-enterprise#165)
…cycle harness (Abilityai/trinity-enterprise#165) The happy-path harness mocks db wholesale, so a new sweep's prune count must be a real int or the WAL-checkpoint sum raises. Same shape as the ent#433 and ent#478 sweeps; the test now also asserts the guardrail prune is reached.
…ls its batch or the owner's read (Abilityai/trinity-enterprise#165) A NUL in a raw-stored field made psycopg2 refuse the whole multi-row insert, and a lone surrogate did the same on SQLite: the batch 500ed on every retry and the forwarder stalled behind it. Inside details a surrogate was stored JSON-escaped and then made every later GET of that agent's events fail while the response was encoded. Display text (tool, verb, tier, turn_kind, rule, path) now carries U+FFFD in place of such a character; an id (execution_id, session_id, tool_use_id) that holds one is dropped, like an over-long one.
…dy is in (Abilityai/trinity-enterprise#165) Auth was decided before the body was read, so a body streamed slowly could outlive an agent delete or rename: its rows landed under the retired name after the cascade, and its report re-marked that name as reporting after the reporter was cleared, so a later agent of the same name would read as covered. The key rows follow the agent (cascaded on delete, re-keyed on rename), so the second check refuses the write with the same 403.
…er rule in the guardrails flow (Abilityai/trinity-enterprise#165)
…e time, not counted in every window (Abilityai/trinity-enterprise#165) An agent whose clock runs ahead sent events whose occurred_at sat after every rolling-window cutoff until the row was pruned, so the 24 h counts (and any since-read) kept counting them. Ingest now clamps occurred_at to received_at when the agent's ts is later, and keeps the agent's own value in details.occurred_at_reported. The event is never rejected: a skewed clock must not cost the agent its record.
…ed, so history before it is not read as clean (Abilityai/trinity-enterprise#165) A consumer of reporting could not tell a run from before the agent's first report (never covered) from a covered run with no events: both read as zero. The reporter hash now records first_report_at with HSETNX (first write wins), and reporting carries it: the stored value, else the oldest stored row's received_at when the fast store lost it, else the last report. Every fallback is at or after the true first report, so a lost fast store only moves coverage later, the safe direction. It is null exactly when the state is never. The response model carries the field so the GET does not drop it.
…ARD-005, the API catalog and the guardrails flow (Abilityai/trinity-enterprise#165)
… after the last report, and is read before the counts (Abilityai/trinity-enterprise#165) Three gaps a second review found in first_report_at, each proven by a test that failed first: - A recycled name inherited its predecessor's reporter record when the teardown's fail-open Redis clear failed, or an ingest in flight wrote it back, so a new agent's unreported history read as covered. The create path now clears it beside the #1560 breakers, before the container. - Two racing reports could leave the start after a later-dated last report. The start is capped at last_report_at: the agent did report then, so the earlier value is still at or after the truth. - count_by_execution read the counts before the coverage, so an agent's first batch landing between the two reads paired a clean 0 with the coverage that batch set up. Coverage is now read first.
… the read order in GUARD-005 and the guardrails flow (Abilityai/trinity-enterprise#165)
Contributor
Author
|
Superseded by #4425, which carries all of abilityai/trinity-enterprise#165 in one PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of abilityai/trinity-enterprise#165 (GUARD-005), one of five PRs. The keyword points at another tracker, so merging closes nothing; the issue is closed by hand at the release cut once all five are on
main.The five PRs and their order. This PR,
AndriiPasternak31/ent165-s2(the agent-side forwarder) andAndriiPasternak31/ent165-s3(the Bash output scanner) are independent and targetdev.AndriiPasternak31/ent165-s4(fleet counts and UI) is stacked on this one, andAndriiPasternak31/ent165-s5(scanner on other tools) onAndriiPasternak31/ent165-s3. #4281's slice E (AndriiPasternak31/issue-4281-e) reads the seam this PR adds and is based on a merge of this PR andAndriiPasternak31/ent165-s2.Why
Guardrail decisions (a Bash or file deny, a turn-tier refusal, a credential hidden from the model) were written only to a file inside the agent container. That file is lost on recreate and readable only while the container runs, so nothing on the platform could say that a run was refused, or that an agent had stopped reporting. #4281 is the concrete case: 78 hook denials over about 25 hours while every execution read
success. This PR is the platform half of the event surface: a table, an ingest the agent posts to under its own key, and a per-agent read.What changes
guardrail_eventson both migration tracks (Invariant Fix git pushing bug #9): the SQLite migrationguardrail_events_tableand Alembic0123_guardrail_events, plusdb/schema.pyanddb/tables.py.UNIQUE(agent_name, event_id); indexes on(agent_name, occurred_at),(execution_id)and(received_at).execution_idcarries no foreign key.db/guardrail_events.pywithdatabase.pyfacade methods. The insert ison_conflict_do_nothing(...).returning(id), so accepted and duplicate counts are exact on both dialects. Reads page by keyset on(occurred_at, id).services/guardrail_event_service.py(normalisation, ingest, reporting state, counts) and routerrouters/guardrail_events.py(# mcp: noneheader; an MCP tool is a follow-up). Models are inmodels.py.POST /api/agents/{agent_name}/guardrail-events, the agent's ownTRINITY_MCP_API_KEYonly (a sibling agent's key, a user key and a connector key get 403). Body{"schema":1,"runtime":…,"events":[≤500],"dropped":n}, ≤256 KiB. Answers200 {accepted, duplicates, rejected}, 413 over the caps, 422 for a malformed envelope only, 429 withRetry-Afterpast 60 posts per minute per agent.kindorhookis stored asotherwith the raw value kept; a malformed event is counted inrejected; an over-longexecution_idis nulled;ruleandpathgo through the credential sanitizer.details.occurred_at_reported. It is never rejected, so a skewed agent clock loses nothing.GET /api/agents/{agent_name}/guardrail-events, owner tier throughOwnedAgentByName(uniform 404, SEC: User and Agent Enumeration via differential API responses #186). Filterskind,since/until,limit(≤200), keyset cursor.{state: reporting|stale|never, last_report_at, first_report_at, runtime, dropped_total}.staleafter 15 minutes without a report (three missed 5-minute keepalives).first_report_atis the receive time of the agent's first accepted report (first write wins in Redis; the oldest stored row is the fallback; capped atlast_report_at). A consumer can therefore tell "before this agent was covered" from "covered and clean". A lost Redis moves it later, never earlier, and the create path clears a recycled name's reporter state so a new agent never inherits a predecessor's start.count_by_execution(agent_name, execution_ids), the read-time seam A schedule created by a non-owner-tier person silently loses the shell on every run, and the run still reports success #4281 E consumes. It returns{reporting, executions: {id: {by_kind, by_hook_code}}}, reads the reporting state before the counts, and filters by the authenticated agent.AGENT_REFS); the reporter key is cleared on teardown (CLEARED_KEYSPACES). Retention is 90 days onreceived_at(so an agent-reported date cannot keep a row forever), through the guarded prune (guardrail_event_retention_daysinOPS_SETTINGS_DEFAULTS, bounds 0–3650).selfinagent_route_tiers.jsonandAGENT_CALLABLE_ANYin_route_census.py; the GET is owner tier;FROZEN_BASELINE_COUNTis unchanged.requirements/security.md§28.5 (status 🚧; the whole event contract),architecture/backend.md,api-endpoints.md,database.md, and an S1 subsection infeature-flows/agent-guardrails.md.Tests
test_ent165_guardrail_event_store.py(real rows on SQLite and PostgreSQL),test_ent165_guardrail_event_route.py(the real router over a real DB, only the key lookup stubbed),test_ent165_guardrail_event_retention.py,test_ent165_guardrail_event_contract.py. The contract test compares the backend's kind and hook enums with the agent hooks'EVENT_KINDS/EVENT_HOOKS; it skips untilAndriiPasternak31/ent165-s2is on the same tree, and passed on the combined tree.test_ent165_*with seeds 11, 22 and 4242: 103 passed, 1 skipped (the contract skip) each time. The store file also ran against a throwaway PostgreSQL 16.test_agent_cleanup_parity, schema parity, 186, 1310, models-centralized, 297, the cleanup sweeps, 1644, 2085, 1638, the retention floor andtest_1484(1743 passed).check_alembic_heads.py: one head.check_alembic_parity.py: pass. A realalembic upgrade headon PostgreSQL reached0123_guardrail_eventswith all indexes, and a downgrade and re-upgrade rebuilt them.other; noON CONFLICT; any valid key accepted (sibling, user and connector key tests); counts not filtered by agent; GET at accessor tier; prune onoccurred_at; sweep not in the cleanup cycle; retention default lowered; theAGENT_REFSentry removed (also red intest_agent_cleanup_parity); no rate cap; no byte cap; path and rule not sanitized; no receive-time clamp (3 tests);first_report_atwritten last-wins, read from the newest row, or dropped from the response model; the create-path clear removed (red intest_1484); the unstorable-character handling removed (5 tests); the second own-key check removed (a streamed body that revokes the key mid-upload gets 200).Mutation: n/a — not a fix
On current
devRebased onto
devat13d32e22f(15 commits past the base it was reviewed and verified on). The lines this PR adds and removes are identical to the verified version — a zero-context diff of the old and new change is empty — and the only conflicts were in docs andtests/registry.json, kept on both sides. Dev's newer commits were not part of the live run below. Re-run on the new base:test_ent165_*with seeds 11, 22 and 4242, 103 passed and 1 skipped each; the guards plustest_1484, 1747 passed (the +4 are dev's new cases intest_ent926_runtime_turn_scope.py); dev's 38 new or changed unit test files, 1541 passed with only the 39 macOS-only hook failures. One Alembic head,0123_guardrail_events(its parent is still dev's head).Verified on a live stack
Production images built from a tree that merges this PR with the other public PRs of ent#165, ent#863 and #4281 onto
dev, run on the prod compose stack (PostgreSQL 16, Redis with ACL users, two backend workers), with real agent containers on Claude Code CLI 2.1.281:devimage and on a fresh install (SQLite), and the realinit_databaseon PostgreSQL moved from0122to0123_guardrail_eventswith one head and the table and indexes present.duplicates. The owner and an admin read rows and state; a shared non-owner gets a 404 byte-identical to a missing agent's.toolis stored as U+FFFD on PostgreSQL and a surrogate inpathdoes not break the GET.AndriiPasternak31/ent165-s2in the same tree, the real forwarder delivered and an agent on an older image stayednever.details;first_report_atstayed put across later reports, a Redis restart (AOF on) and a backend restart; after a realFLUSHALLthe database fallback gave the oldest row's receive time and the next report reset it to that report; a same-name recreate got its own start.Not proven here
Unfiled follow-ups
guardrail_event_retention_daysis not inretentionFields.jsor the monitoring retention table.dropped_totalon the read surfaces — lost events are counted but not shown.dropped_totalis cumulative with no time, so a consumer cannot tell which runs a drop may cover.last_report_atis last-write-wins — a delayed report can move it earlier, so an agent can readstaleearly.Journey Impact
Journey Impact: none: adds a guardrail event table, an agent-key ingest and an owner read; no catalog promise covers guardrail events
🤖 Generated with Claude Code
https://claude.ai/code/session_01HWna79L6KER8ABV8gexSFC