Skip to content

feat: Claude Code Stop-hook integration - #91

Merged
adaamko merged 3 commits into
mainfrom
feat/claude-code-hook
Aug 13, 2026
Merged

adaamko merged 3 commits into
mainfrom
feat/claude-code-hook

Conversation

@adaamko

@adaamko adaamko commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

What

Implements the Claude Code hook from #50: lettucedetect/integrations/claude_code/ with a Stop-hook script runnable as python -m lettucedetect.integrations.claude_code.check_answer.

  • Reads the hook event from stdin, extracts the last user/assistant messages from the session transcript (pure function, junk-tolerant), checks the answer against user-supplied grounding context (--context-file, default context.md; missing context exits 0 silently).
  • Two modes per the issue: --api-url via LettuceClient.detect_spans (recommended, model stays loaded) and --model-path in-process, with optional --taxonomy-head so typed spans surface unsupported_addition ("the request did not ask for this") in the report.
  • Exit contract: flagged spans print a report to stderr and exit 2 (fed back to the agent); clean answers exit 0. stop_hook_active short-circuits to exit 0 to prevent feedback loops.
  • README with a ready-to-paste settings.json block and a 5-line walkthrough, following the langchain integration pattern.
  • 15 network-free tests (tests/test_claude_code_hook_pytest.py) with stubbed detectors + a transcript fixture; transcript parsing additionally verified against a real Claude Code session transcript (Claude Code 2.x JSONL format, August 2026).

No changes outside lettucedetect/integrations/claude_code/ and tests/.

Closes #50

Checklist

  • python -m pytest tests/test_claude_code_hook_pytest.py -v passes (15/15); python -m pytest full suite green locally; ruff format + check clean.

  • I certify that I have the right to submit this code and that it may be distributed under the repository's MIT license

@adaamko

adaamko commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Added a third mode: --llm-model KRLabsOrg/lettucedect-v2-qwen-2b --llm-base-url http://localhost:8001/v1 routes through the LLM detector, so the generative model (via vLLM or any OpenAI-compatible endpoint) and plain LLM judges both work; the detector's _is_generative heuristic picks the frozen prompt automatically. 17 tests now.

End-to-end verification with the real models (lettucedect-v2-mmbert-base + lettucedect-v2-taxonomy-head, run locally through the actual hook entry point):

Scenario: request says "Fix the bug in normalize_whitespace: consecutive tabs are not collapsed", context is the source file, and the agent's answer describes the correct regex fix plus a sentence nobody asked for: "I also added an LRU cache on normalize_whitespace to speed up repeated calls, and bumped the default max_length from 256 to 512."

Result: exit 2 with

LettuceDetect flagged 1 unsupported span(s) in the answer:
- "I also added an LRU cache on normalize_whitespace to speed up repeated calls, and bumped the default max_length from 256 to 512 for longer documents." (confidence 0.96, unsupported_addition/behavior — the request did not ask for this)

The grounded fix description is not flagged. Negative control (same request and context, honest answer without the extra sentence): exit 0, no report. This is the capability the integration is for: catching side effects the request never asked for, typed as such.

@adaamko
adaamko merged commit af4059e into main Aug 13, 2026
4 checks passed
@adaamko
adaamko deleted the feat/claude-code-hook branch August 13, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Claude Code hook: flag hallucinations in agent answers

1 participant