feat(hooks): strict search gate with per-agent query evidence - #3291
feat(hooks): strict search gate with per-agent query evidence#3291logan683 wants to merge 12 commits into
Conversation
…se hooks
The generated hook resolved its command via shutil.which("graphify"), which
returns the console-script shim. For uv-tool and pipx installs that shim is
unsigned; on Windows with Smart App Control enabled, code-integrity policy
refuses to load it, so the hook can never run. Both ~/.local/bin/graphify.exe
and the venv Scripts/graphify.exe are NotSigned, so resolving to a different
shim directory does not help.
The hook also had no failure guard. A PreToolUse hook that exits non-zero
blocks the tool call it was meant to advise, so an advisory graph check could
break the host on any launcher failure.
Resolve to '<sys.executable> -m graphify' via the existing hooks._pinned_python(),
the same reasoning already applied to git hooks, and make both the POSIX and
Windows commands fail open with a timeout.
The Graphify-Labs#2165 regression parses the subcommand out of the hook command; it now
handles both supported grammars rather than assuming the subcommand is the
second token, so it keeps proving the same contract.
Refs Graphify-Labs#3280
The generated Windows command may be run by cmd.exe or by PowerShell, and no bare fail-open suffix is correct in both. `& exit /b 0` is cmd syntax; run by PowerShell the bare `&` is the background operator, so the hook spawns a job and exits 1 - the fail-open suffix caused the very failure it was added to prevent (Graphify-Labs#3280). `; exit 0` has the mirror problem: valid in PowerShell, rejected by cmd.exe. Naming the interpreter removes the ambiguity. Measured on Windows, the generated string now exits 0 under both pwsh and cmd.exe; the previous form exited 1 under pwsh. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPaGbwQeyWLwbVm9uRgHtU
Strict Claude hooks recorded no query evidence after the graph-wide stamp
was removed: the Read deny fired once per session unconditionally and the
retry was unconditional, and Bash/Grep search was nudge-only by design. An
agent instructed to read and search through Bash never met a block, and a
nudge that costs nothing is learned in one turn to cost nothing.
- PostToolUse `hook-guard mark-queried` writes cache/hook_sessions/<sid>
[--agent-<id>].queried after mcp__graphify__{query_graph,get_node,
get_neighbors,get_community,god_nodes,shortest_path} or a Bash command
that runs `-m graphify query|explain|path` (project sidecar form included).
- `hook-guard search --strict` denies a recursive in-project corpus search
(grep -r/-R, rg, find, fd, ack, ag, or the Grep tool on a directory) until
that marker exists. Not once-per-session: the way out is a query, not a
retry. Exact-file grep, stdin grep, git grep, out-of-project targets, Glob,
soft mode, no session_id, malformed stdin and GRAPHIFY_HOOK_STRICT=0 never
block. Deny text is constant; parsing never executes the command.
- Quoted spans are protected before the operator split so a regex with
parens or a "$(cat graphify-out/.graphify_python)" interpreter cannot
fragment its segment.
- install: --strict now applies to Bash|Grep; the PostToolUse marker hook is
registered and removed with the others.
Tests: tests/test_search_strict.py (RED on 2a83725: 12 failed / 5 passed;
GREEN: 301 passed across the hook/install suites), skillgen --check 134 OK,
ruff clean. Live proof: a fresh `claude -p --model claude-opus-5` session
in a scratch project with the candidate hooks had its second tool call
(grep -rn ... .) denied, queried the graph next, and was then allowed.
Stacked on Graphify-Labs#3281 (pr/hook-fail-open); follow-up to Graphify-Labs#3280.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTa1HY67hwopSPQtffAPy
Round 1 review findings on the strict search gate: - F1: a Bash-only session never reached the 24-hour marker sweep that lived only inside _mark_session_denied, so .queried markers accumulated and a reused session id stayed pre-authorized. Both writers now call one shared _gc_session_markers. - F2: Claude Code on Windows exposes a PowerShell tool (tool_name "PowerShell", tool_input.command; PreToolUse:PowerShell hook events were measured in a live transcript). Its recursive in-project searches now reach the same guard: matcher Bash|Grep|PowerShell, and two measured shapes are denied until query evidence exists — `Get-ChildItem|gci|ls|dir ... -Recurse ... | Select-String|sls` and `Select-String ... -Path <dir or dir\*>`. Exact-file -Path, Get-Content | Select-String and out-of-project targets only nudge; non-search PowerShell stays silent. Tests: 5 RED on a0a4f1d, 306 passed after; skillgen --check 134 OK; ruff clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTa1HY67hwopSPQtffAPy
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTa1HY67hwopSPQtffAPy
…ll listing Round 3 review findings: - F1: _session_has_queried now enforces the 24-hour TTL itself and best-effort unlinks an expired marker, so a stale .queried never pre-authorizes a reused session id while no writer happens to sweep. One _SESSION_MARKER_TTL shared with the GC. - F2: a recursive PowerShell listing without a Select-String pipe (Get-ChildItem|gci|ls|dir ... -Recurse) is the same corpus-enumeration class as `find .` and now denies in-project, nudges out-of-project, and stays silent when not recursive. Tests: 2 RED on 8d04a72, 326 passed after; skillgen --check 134 OK; ruff clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTa1HY67hwopSPQtffAPy
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. 2 change(s) alter behavior, breaking input(s) attached.
Behavior changes: \_claude\_pretooluse\_hooks changes behavior, here is the input that shows it.
The verifier found a concrete input on which \_claude\_pretooluse\_hooks behaves differently before and after the change. If that change is intended, ship it; if not, this is your bug.
Guarantee: This difference was REPRODUCED, the verifier actually ran both versions on that input and saw them disagree. It is real, not an artifact.
Evidence: On input \{"strict":"True","project":"True"\}, the old code produced \[\{'matcher': 'Bash\|Grep', 'hooks': \[\{'type': 'command', 'command': 'graphify hook\-guard search'\}\]\}, \{'matcher': 'Read\|Glob', 'hooks': \[\{'type': 'command', 'command': 'graphify hook\-guar… but the new code produces \[\{'matcher': 'Bash\|Grep\|PowerShell', 'hooks': \[\{'type': 'command', 'command': '"$\(cat graphify\-out/\.graphify\_python\)" \-m graphify hook\-guard search \-\-strict \|\| true', 'commandWindows…. Paste that input straight into a regression test.
Behavior changes: \_resolve\_graphify\_exe changes behavior, here is the input that shows it.
The verifier found a concrete input on which \_resolve\_graphify\_exe behaves differently before and after the change. If that change is intended, ship it; if not, this is your bug.
Guarantee: This difference was REPRODUCED, the verifier actually ran both versions on that input and saw them disagree. It is real, not an artifact.
Evidence: On input \{"project":"True"\}, the old code produced 'graphify' but the new code produces '"$\(cat graphify\-out/\.graphify\_python\)" \-m graphify'. Paste that input straight into a regression test.
Graphify review — findings
Reworks generated agent guidance to prefer the MCP query_graph tool and fall back to the graph's recorded interpreter via "$(cat graphify-out/.graphify_python)" -m graphify … (with a PowerShell variant), so hardened Windows hosts stop routing through an unsigned graphify.exe shim. Rewires project-scoped hooks to resolve Graphify through each clone's graphify-out/.graphify_python sidecar and fail open (emit no graph decision) until it exists, keeping committed configs portable on Application Control hosts. Fixes strict Claude hooks to key deny markers by session_id plus agent_id instead of one graph-wide 30-minute stamp, records per-session/agent query evidence via a PostToolUse hook-guard mark-queried on the MCP graph tools and graphify query|explain|path, GC's markers older than 24 hours, and now blocks a recursive in-project search (Bash grep -r/rg/find, the Grep tool, or PowerShell Get-ChildItem -Recurse | Select-String) until one traversal runs — while leaving exact-file grep, stdin grep, git grep, out-of-project targets, Glob, and soft mode unblocked.
Worth a look
- hook command interpolates unquoted exe now that caller quoting removed —
graphify/install.py:320· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Strict Claude PreToolUse hooks mask blocking failures —
graphify/install.py:324· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Project-scoped Windows hook executes interpreter path read from project-controlled file —
graphify/install.py:1496· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Wrapper skip loop can drop the wrapped command's positional args —
graphify/cli.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- IndexError on empty graphify command tokens —
graphify/cli.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2314 functions depend on the 2136 functions this change touches.
Health — this change adds coupling hotspots:
- new:
dispatch_command()— 2 callers, 122 callees - new:
codebuddy_install()— 20 callers, 5 callees - new:
claude_install()— 19 callers, 4 callees - new:
_run_hook_guard()— 5 callers, 14 callees - new:
gemini_install()— 10 callers, 7 callees - new:
claude_uninstall()— 17 callers, 4 callees - new:
_project_uninstall()— 5 callers, 13 callees - new:
render()— 13 callers, 5 callees - …and 21 more — each is listed as a finding
Verification — 2314 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 2260 function(s) in the blast radius were not formally verified this run
Formal verification
Behavior changes: \_claude\_pretooluse\_hooks changes behavior, here is the input that shows it.
The verifier found a concrete input on which \_claude\_pretooluse\_hooks behaves differently before and after the change. If that change is intended, ship it; if not, this is your bug.
Guarantee: This difference was REPRODUCED, the verifier actually ran both versions on that input and saw them disagree. It is real, not an artifact.
Evidence: On input \{"strict":"True","project":"True"\}, the old code produced \[\{'matcher': 'Bash\|Grep', 'hooks': \[\{'type': 'command', 'command': 'graphify hook\-guard search'\}\]\}, \{'matcher': 'Read\|Glob', 'hooks': \[\{'type': 'command', 'command': 'graphify hook\-guar… but the new code produces \[\{'matcher': 'Bash\|Grep\|PowerShell', 'hooks': \[\{'type': 'command', 'command': '"$\(cat graphify\-out/\.graphify\_python\)" \-m graphify hook\-guard search \-\-strict \|\| true', 'commandWindows…. Paste that input straight into a regression test.
Behavior changes: \_resolve\_graphify\_exe changes behavior, here is the input that shows it.
The verifier found a concrete input on which \_resolve\_graphify\_exe behaves differently before and after the change. If that change is intended, ship it; if not, this is your bug.
Guarantee: This difference was REPRODUCED, the verifier actually ran both versions on that input and saw them disagree. It is real, not an artifact.
Evidence: On input \{"project":"True"\}, the old code produced 'graphify' but the new code produces '"$\(cat graphify\-out/\.graphify\_python\)" \-m graphify'. Paste that input straight into a regression test.
No difference found (not proven): No behavior difference found in \_bash\_invokes\_search (not a proof).
The verifier ran both versions of \_bash\_invokes\_search on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_mark\_session\_denied (not a proof).
The verifier ran both versions of \_mark\_session\_denied on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
No difference found (not proven): No behavior difference found in \_run\_hook\_guard (not a proof).
The verifier ran both versions of \_run\_hook\_guard on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify claude\_install.
The verifier did not have enough to check claude\_install, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 30 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
Could not verify: Could not verify \_install\_claude\_hook.
The verifier did not have enough to check \_install\_claude\_hook, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `project_dir` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_install\_codebuddy\_hook.
The verifier did not have enough to check \_install\_codebuddy\_hook, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `project_dir` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_install\_codex\_hook.
The verifier did not have enough to check \_install\_codex\_hook, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `project_dir` is annotated `Path` — outside the synthesizable primitive/collection set
No difference found (not proven): No behavior difference found in \_print\_install\_usage (not a proof).
The verifier ran both versions of \_print\_install\_usage on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_strip\_graphify\_hook.
The verifier did not have enough to check \_strip\_graphify\_hook, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `settings_path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_uninstall\_codebuddy\_hook.
The verifier did not have enough to check \_uninstall\_codebuddy\_hook, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `project_dir` is annotated `Path` — outside the synthesizable primitive/collection set
· 5 grounded finding(s) anchored inline below; 24 more finding(s) on lines outside this diff (see the check run).
| return False | ||
|
|
||
|
|
||
| def _run_hook_guard(kind: str, strict: bool = False) -> None: |
There was a problem hiding this comment.
_run_hook_guard()
fans out to 14 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| 'powershell.exe -NoProfile -NonInteractive -Command ' | ||
| f'"{inner}; exit 0"' | ||
| ) | ||
| def _install_codex_hook(project_dir: Path, project: bool = False) -> None: |
There was a problem hiding this comment.
_install_codex_hook()
7 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| print("Strict mode: the first raw file read per session is blocked until one") | ||
| print("graph traversal runs, and recursive in-project searches stay blocked until") | ||
| print("this session has run one (toggle with GRAPHIFY_HOOK_STRICT=0).") | ||
| def _install_claude_hook(project_dir: Path, strict: bool = False, project: bool = False) -> None: |
There was a problem hiding this comment.
_install_claude_hook()
fans out to 6 callees (efferent coupling); 7 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") | ||
| print(f" .claude/{settings_path.name} -> PreToolUse hook removed") | ||
| print(f" .claude/{settings_path.name} -> graphify hooks removed") | ||
| def uninstall_all(project_dir: Path | None = None, purge: bool = False) -> None: |
There was a problem hiding this comment.
uninstall_all()
fans out to 13 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert _is_deny(_invoke("search", _search("grep -rn foo ."), tmp_path, monkeypatch)) | ||
|
|
||
|
|
||
| def test_allows_after_mcp_query(tmp_path, monkeypatch): |
There was a problem hiding this comment.
test_allows_after_mcp_query()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
Stacked on #3281 (
pr/hook-fail-open); follow-up to #3280. Please review #3281 first — this PR's diff againstv8includes it until it merges, after which this branch will be rebased.Problem
After #3281 removed the graph-wide query stamp, strict Claude hooks recorded no query evidence at all. The
Readdeny fires once per session unconditionally and the retry is unconditional;Bash|Grepsearch is nudge-only by design (cli.pydocstring). An agent whose harness instructs it to read and search through Bash (cat,sed,grep -r) never meets a block, and a nudge that costs nothing is learned in one turn to cost nothing. Measured on this machine: seven documented recurrences of "grep the corpus before querying" over twelve days, with a headless Claude run scoring RED for it four days after the previous fix.Fix
hook-guard mark-queriedwritesgraphify-out/cache/hook_sessions/<session_id>[--agent-<agent_id>].queriedafter the MCP graph tools (query_graph,get_node,get_neighbors,get_community,god_nodes,shortest_path) or a Bash command that runs-m graphify query|explain|path(the project sidecar form"$(cat graphify-out/.graphify_python)" -m graphify query …included). Same identity sanitising/hashing as the existing.deniedmarker.hook-guard search --strictdenies a recursive in-project corpus search —grep -r/-R/--recursive,rg,find,fd,ack,ag, or the Grep tool on a directory — until that marker exists. Not once-per-session: the way out is a query, not a retry. Exact-file grep, stdin grep,git grep, out-of-project targets, Glob, soft mode, missingsession_id, malformed stdin andGRAPHIFY_HOOK_STRICT=0never block. The deny text is constant; parsing never executes the command.(/|or a"$(cat …)"interpreter cannot fragment its segment._strip_heredoc_bodiesis shared with_bash_invokes_search.--strictnow applies toBash|Grep; the PostToolUse hook is registered and stripped with the others. README strict paragraph and CHANGELOG updated.Evidence
tests/test_search_strict.py— RED on2a83725: 12 failed / 5 passed (the five are the never-deny safety cases). GREEN: 301 passed acrosstest_search_strict,test_hook_strict,test_search_hook,test_read_hook,test_hook_guard,test_install*,test_settings_merge,test_codebuddy.2a83725export).python -m tools.skillgen --check134 OK,--audit-coverageOK,--schema-singletonOK;ruff check .clean.claude -p --model claude-opus-5 --dangerously-skip-permissionssession in a scratch project with these hooks installed project-scoped, given a neutral relationship question naming no tool, madegrep -rn … .its second call → denied with the strict reason →mcp__graphify__query_graph→ subsequent searches allowed → correct answer with file:line citations.Not covered
Select-String -Recurse) — different matcher, not designed here.🤖 Generated with Claude Code
https://claude.ai/code/session_01HNTa1HY67hwopSPQtffAPy