Skip to content

feat(evaluation): add bounded LLM analysis policy - #86

Merged
tom-sapletta-com merged 3 commits into
mainfrom
ticket/073-analysis-budget-dsl
Aug 12, 2026
Merged

feat(evaluation): add bounded LLM analysis policy#86
tom-sapletta-com merged 3 commits into
mainfrom
ticket/073-analysis-budget-dsl

Conversation

@tom-sapletta-com

Copy link
Copy Markdown
Contributor

Summary

  • define canonical t2c.analysis-policy/v1 parser and renderer
  • require LLM for selected semantic stages with exact-evidence cache identity
  • bound requests, tokens, and elapsed time globally and per stage
  • add fail-closed validation, selection, fingerprint, usage, and cost tests

Evidence

  • npm run verify: 414 pass, 0 fail, 1 local JDK skip
  • npm run evaluate:gold: all gated metrics 100%
  • governance: 0 findings
  • npm audit --omit=dev: 0 vulnerabilities
  • make docker-smoke: PASS
  • real require-llm pipeline: 43 responses, 651709 tokens, 1.303208672 USD, no semantic fallback
  • deterministic Vallm: 2/2 files PASS

Runtime/CLI/provider wiring remains a separate integration ticket.

@ifuri-validator-agent ifuri-validator-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deterministic Validator approval for exact head 5b61cb8b0cc2a0dd65abf9073f4b60243461a845.

Ticket: ticket-073
Correlation ID: todo2code-pr-86-ticket-073-5b61cb8b0c
Model: openrouter/z-ai/glm-5.2
Reviewed diff chunks: 4
Advisory LLM verdict: APPROVE
Advisory summary: Reviewed all 4 diff chunk(s). Ticket-073 documentation and metadata chunk introducing the analysis-policy DSL work. Changes are documentation-only (TODO, TICKETS index, ticket README/logs/agent notes). No source or test code is present in this chunk. Test results show verify PASS and no failed checks. No security concerns visible. | This chunk introduces a canonical analysis-policy DSL (t2c.analysis-policy/v1) with a parser, renderer, fail-closed validation, deterministic stage selection, and budget enforcement. The code is well-structured with strict schema validation, canonical round-trip enforcement, and proper use of Node.js crypto for fingerprinting. Test results show all checks passing with no failures. Governance and audit findings are reported as clean in the PR body. The visible code demonstrates sound security practices (fail-closed defaults, strict input validation, canonical encoding verification). | This chunk introduces the analysis-policy serialization/deserialization tail (cache key derivation, cost estimation, structured policy reader/parser, validators, fingerprinting) and the beginning of the test file. Code is well-structured with fail-closed validation, canonical format enforcement, budget bounding, and SHA-256 fingerprinting. No security issues observed. Test results show all checks passing. | This chunk contains comprehensive tests for the analysis policy module covering canonical document round-trip parsing/rendering, deterministic stage selection with usage ceiling calculations, trigger validation (unknown, duplicate, non-canonical), cache key fingerprinting, cost estimation, fail-closed policy validation, and parser rejection of non-canonical input. Tests verify that budgets are enforced, stages fail closed, and semantic fallback is rejected. Code quality is high with thorough edge-case coverage.
Advisory findings: none
The LLM output above is advisory and was not used as the approval trust root.
Merge was not requested or performed.

Decision record (recomputable)

DECISION D-073-9404
TICKET ticket-073
HEAD_SHA 5b61cb8b0cc2a0dd65abf9073f4b60243461a845
CORRELATION_ID todo2code-pr-86-ticket-073-5b61cb8b0c
ACTOR agent:ifuri-validator-agent[bot]
APPLIED_RULE P-CORE-015
INPUT author_login = "tom-sapletta-com"
INPUT observed_checks = ["governance=SKIPPING","Live OpenRouter contract (opt-in)=SKIPPING","verify=PASS","Java adapter (JDK 17 required)=PASS","koru / code-review=PASS","Live OpenRouter contract (opt-in)=SKIPPING","governance=SKIPPING","Java adapter (JDK 17 required)=PASS","verify=PASS"]
INPUT required_checks = ["verify","Java adapter (JDK 17 required)","koru / code-review"]
INPUT required_checks_source = "env/request"
INPUT reviewer_login = "ifuri-validator-agent[bot]"
VERDICT APPROVE AUTHORITY DETERMINISTIC
REJECTED REQUEST_CHANGES BECAUSE NO_UNSAFE_CHANGE_REASON_FOUND
ADVISORY llm_verdict = "APPROVE" MODEL "openrouter/z-ai/glm-5.2"
ASSERT VERDICT_AUTHORITY != "ADVISORY"

@tom-sapletta-com
tom-sapletta-com merged commit f0df52c into main Aug 12, 2026
13 checks passed
@tom-sapletta-com
tom-sapletta-com deleted the ticket/073-analysis-budget-dsl branch August 12, 2026 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant