Skip to content

Judge violations with a probability cutoff and expose example presets - #2

Merged
mrubens merged 5 commits into
mainfrom
codex/example-request-presets
Sep 29, 2026
Merged

mrubens merged 5 commits into
mainfrom
codex/example-request-presets

Conversation

@mrubens

@mrubens mrubens commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Judge each change with one boolean question and a violation probability. A finding is emitted when that probability meets the rule's cutoff. Lower scores produce no finding; known missing evidence, partial evidence limitations, and inference errors still produce incomplete checks. Binary answers do not trigger extra model calls based on uncertainty.

Expose prepareExamples so model testers can replay exact initial, expanded, and partial evidence packets from labeled fixtures. Preparation uses disposable Git repositories, makes no inference calls, and keeps expected labels outside model input. Document calibration and model-tester integration for agents. Existing custom outcome/confidence evaluators remain supported; increment the protocol version to invalidate cached judgments.

Validation

  • npm run check: 52 tests passed, covering threshold boundaries, raw probability caching, incomplete evidence/errors, preparation parity, and cancellation cleanup.
  • CI passed on Linux and macOS with Node 22 and 24.
  • npm pack --dry-run --json passed.
  • Live calibration in the consuming repository found that the old confidence cutoffs miss violations when applied to violation probabilities. Recalibrate thresholds with representative valid and violating examples before adopting the new question.

Version prepared as 0.3.0; not published.

@mrubens mrubens changed the title Prepare exact example requests for model testers Judge violations with a probability cutoff and expose example presets Sep 29, 2026
@mrubens
mrubens merged commit badbfcc into main Sep 29, 2026
8 checks passed
@mrubens
mrubens deleted the codex/example-request-presets branch September 29, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant