Skip to content

Ask whether changed lines need correction to satisfy rules - #3

Merged
mrubens merged 1 commit into
mainfrom
codex/correction-question
Sep 29, 2026
Merged

mrubens merged 1 commit into
mainfrom
codex/correction-question

Conversation

@mrubens

@mrubens mrubens commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Ask whether changed lines need correction to satisfy a repository rule. The previous question produced low violation probabilities on clear prose violations, forcing consumers to choose between low cutoffs and missed findings. The new question keeps one Noul answer, preserves scope and exception handling, treats source as untrusted evidence, and distinguishes changed lines from removed or unchanged context.

Increment the cache protocol to invalidate answers from the previous question. Prepare version 0.3.1; no publication is performed by this PR.

Validation

  • npm run check: all 52 tests passed; npm pack --dry-run --json passed.
  • Eight prompt variants compared on 59 synthetic capitalization and prose examples, three repetitions each. At a 0.75 cutoff, the correction question caught 61/63 violation runs versus 39/63 for the previous question, with zero false blocks in 114 valid runs. Expected labels stayed outside model input.
  • Fresh confirmation through the full checker: 20 additional examples, five repetitions each, three-second budget. The correction question caught 40/40 violation runs at 0.75 and 0.80, with zero false blocks in 60 valid runs and no operational failures.
  • Full-check regression comparison across all seven consuming rules: 80 existing examples, three repetitions, configured cutoffs with rules 7 and 8 at 0.75. The new prompt caught 81/81 violation runs and passed 159/159 valid runs; the previous prompt caught 75/81 with the same valid passes. Neither had incomplete results or operational failures. Privacy coverage is valid-only.
  • These are bounded synthetic evaluations, not guarantees. One development example still fell below 0.75 twice; consumers should calibrate their own rules and thresholds. Reports remain local.

@mrubens
mrubens merged commit 29c4a8f into main Sep 29, 2026
8 checks passed
@mrubens
mrubens deleted the codex/correction-question branch September 29, 2026 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant