Add finding previews and calibration feedback tools

README.md

@@ -452,3 +452,84 @@ Use advisory mode on a labeled sample before enforcing a rule.
 - [jev-lint](https://github.com/mizchi/jev-lint): caching, calibration, and hook integration.
 
 This implementation is independently written. MIT licensed.
+
+## Inspect a finding
+
+Text reports show the rule, violation probability, cutoff, and a bounded preview
+of changed lines with before/after line numbers. The model judges an evidence
+packet; the preview does not claim to identify the exact offending line. JSON
+reports include the same preview and indicate omitted or truncated lines.
+
+Checks show delayed progress on stderr, including checked packets and cache hits.
+Use `--no-progress` for quiet output or `--verbose` for individual requests. JSON
+stays on stdout. Progress makes no inference calls and does not extend the hook
+budget. Cached answers are reused only for matching evidence, rule, model/backend
+identity, and protocol. Custom evaluators must supply a stable `cacheIdentity` to
+opt into caching; hosts with mutable inference settings may leave it disabled.
+
+## Turn a mistake into an example
+
+Preview an incorrectly blocked edit (`pass`) or a missed violation (`violation`):
+
+```sh
+judgement capture --rule wording --path docs/guide.md \
+  --name "Quoted UI label" --expected pass
+```
+
+Capture reads the index against HEAD, including unchanged supporting files named
+by the rule's context globs. It does not read unstaged changes or call the model.
+Use `--base <commit> --head <commit>` for committed changes and repeat
+`--context <path>` to include additional unchanged supporting files.
+
+Review and redact the preview, then save a new fixture file:
+
+```sh
+judgement capture --rule wording --path docs/guide.md \
+  --name "Quoted UI label" --expected pass \
+  --output .judgement/examples/quoted-label.json
+judgement test --rule wording --examples .judgement/examples/quoted-label.json
+```
+
+`--output` refuses to overwrite existing files and never stages anything. If the
+preview contains sensitive values, save the preview to a local scratch file and
+replace them with synthetic values before adding it to the repository. The fixture
+can be tested independently with `--examples` or its example merged into the
+rule's main fixture file. Multi-file changes need reduction to a single changed
+file with unchanged supporting context. Binary files, symlinks, and captures above
+256 KB are rejected instead of producing misleading fixtures.
+
+## Compare calibration runs
+
+Keep the fixtures fixed while changing a rule, prompt, model, or cutoff:
+
+```sh
+mkdir -p .judgement/results
+judgement test --format json > .judgement/results/before.json
+# Make the candidate change, then rerun the same examples.
+judgement test --format json > .judgement/results/after.json
+judgement compare --before .judgement/results/before.json \
+  --after .judgement/results/after.json
+```
+
+Comparison lists improvements and regressions per example, so a net gain does not
+hide a new false block. It compares correct-run rates when repetition counts differ
+and retains incomplete results, failures, and raw probability ranges in JSON.
+A regression exits 1; incompatible input exits 2. Fixture hashes and labels must
+match. Reports without fixture provenance must be regenerated. Calibration reports
+with several cutoffs use each report's configured cutoff by default; select another
+recorded cutoff with `--before-threshold` or `--after-threshold`. Comparison is local
+and makes no inference calls. Keep generated reports out of source control.
+
+## Strict CI checks
+
+Run `judgement check --base <merge-base> --head <pr-head> --timeout-ms 120000`
+without `--hook` or `--advisory`. Violations exit 1, invalid configuration exits 2,
+and incomplete coverage or inference failures exit 3. Make that check required
+in the repository's merge rules; a local hook may be skipped or time out.
+
+For public contributions, run a pinned, trusted copy of Judgement with trusted
+workflow code and a read-only checkout of the base revision. Fetch the PR head as
+Git objects and inspect its diff without checking out or executing PR code. Keep
+inference credentials scoped to the checking step, disable package install scripts,
+and never restore caches controlled by untrusted PR code into that job. CI uses
+its configured inference backend; align its model with the one used for calibration.

docs/agents.md

@@ -150,3 +150,24 @@ remain incomplete. The default binary evaluator does not request context expansi
 based on its probability. Use explicit context and compare expanded packets when
 investigating a missed violation. Recalibrate when changing question types; Choice
 confidence and Noul probability use different meanings and scales.
+
+## Collect and compare feedback
+
+Use `judgement capture --rule <id> --path <file> --name <name> --expected pass|violation`
+to preview an index-based example. Review and redact it before saving a new file
+with `--output`; capture never overwrites or stages files. Include unchanged
+supporting evidence with `--context`. Reduce multi-file changes rather than
+silently discarding context that changed. Captured content must satisfy the
+repository's privacy requirements before it is committed.
+
+Save baseline and candidate `test --format json` reports locally, using identical
+fixtures. `judgement compare --before <file> --after <file>` exposes per-example
+regressions and exits 1 when any are found. Do not change expected labels to hide
+regressions. Compare both valid exceptions and violations; repeated observations
+of the same example are not independent coverage.
+
+Finding snippets are changed-line previews, not model-localized accusations.
+Read the supplied evidence before proposing a fix. Progress and cache-hit counts
+are diagnostics; only a complete report establishes check completion. In CI, use
+strict checks without `--hook` or `--advisory`, and require the resulting status
+before merge. Missing inference credentials must not produce a successful check.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@roo-code/judgement",
-  "version": "0.3.1",
+  "version": "0.4.0",
   "description": "Fast Git checks for plain-language business rules, powered by Jev",
   "type": "module",
   "license": "MIT",

src/calibrate.js

@@ -2,7 +2,7 @@ import { mkdtemp, mkdir, readFile, rm, writeFile } from 'node:fs/promises';
 import { tmpdir } from 'node:os';
 import { dirname, join, resolve } from 'node:path';
 import { check } from './check.js';
-import { git, gitEnv } from './git.js';
+import { git, gitEnv, hash } from './git.js';
 import {
   ConfigurationError,
   matches,
@@ -179,6 +179,7 @@ export async function calibrate(options = {}) {
   };
   const report = {
     ruleId: criterion.id,
+    fixtureSha256: hash(JSON.stringify(examples)),
     configuredThreshold: criterion.threshold,
     backend: options.evaluate ? 'custom' : (options.model ?? MODEL),
     protocolVersion: PROTOCOL_VERSION,

src/capture.js

@@ -0,0 +1,116 @@
+import { writeFile } from 'node:fs/promises';
+import { snapshot, inventory, streamGit, git, policyBlob } from './git.js';
+import {
+  parsePolicy,
+  readPolicy,
+  ConfigurationError,
+  matches,
+} from './policy.js';
+import { examplesFrom, safePath } from './calibrate.js';
+
+/** Capture index/commit objects, never unstaged working-tree content. No inference. */
+export async function captureExample(options = {}) {
+  const { ruleId, path, name, expected } = options;
+  if (
+    !ruleId ||
+    !safePath(path) ||
+    !name?.trim() ||
+    !['pass', 'violation'].includes(expected)
+  )
+    throw new ConfigurationError(
+      'Capture requires --rule, a safe --path, --name, and --expected pass|violation.',
+    );
+  const snap = await snapshot(options, options.signal);
+  const text = await policyBlob(snap, snap.base, options.signal);
+  const policy =
+    text === null ? await readPolicy(snap.root) : parsePolicy(text);
+  const rule = policy.criteria.find((item) => item.id === ruleId);
+  if (!rule)
+    throw new ConfigurationError(
+      'The selected rule does not exist in the base policy.',
+    );
+  const change = snap.changes.find((item) => item.path === path);
+  if (!change)
+    throw new ConfigurationError(
+      'The selected path has no change in this snapshot.',
+    );
+  let remaining = 256_000;
+  const read = async (path, oid, mode) => {
+    if (mode === '000000') return null;
+    if (!['100644', '100755'].includes(mode))
+      throw new ConfigurationError('Capture supports regular text files only.');
+    const size = Number(
+      await git(snap.root, ['cat-file', '-s', oid], options.signal, snap.env),
+    );
+    if (!Number.isFinite(size) || size > remaining)
+      throw new ConfigurationError(
+        'Capture exceeds 256 KB. Reduce the example before saving it.',
+      );
+    remaining -= size;
+    const chunks = [];
+    for await (const chunk of streamGit(
+      snap.root,
+      ['cat-file', 'blob', oid],
+      options.signal,
+      snap.env,
+    ))
+      chunks.push(chunk);
+    const content = Buffer.concat(chunks);
+    if (content.includes(0))
+      throw new ConfigurationError('Capture does not support binary files.');
+    try {
+      return new TextDecoder('utf-8', { fatal: true, ignoreBOM: true }).decode(
+        content,
+      );
+    } catch {
+      throw new ConfigurationError('Capture supports UTF-8 text only.');
+    }
+  };
+  const before = await read(path, change.oldOid, change.oldMode);
+  const after = await read(path, change.oid, change.mode);
+  const files = await inventory(snap, options.signal);
+  const contexts = new Set(options.context ?? []);
+  for (const pattern of rule.context ?? []) {
+    const found = [...files.keys()].filter((file) => matches(file, [pattern]));
+    if (!found.length)
+      throw new ConfigurationError(`Required context has no match: ${pattern}`);
+    for (const file of found) contexts.add(file);
+  }
+  const context = {};
+  for (const file of contexts) {
+    if (file === path) continue;
+    if (!safePath(file) || !files.has(file))
+      throw new ConfigurationError(
+        'Context must name an existing safe repository path.',
+      );
+    if (snap.changes.some((item) => item.path === file))
+      throw new ConfigurationError(
+        'Supporting context changed too. Reduce this to a single-file example with unchanged context.',
+      );
+    const entry = files.get(file);
+    context[file] = await read(file, entry.oid, entry.mode);
+  }
+  const fixture = {
+    ruleId,
+    examples: [
+      {
+        name,
+        path,
+        before,
+        after,
+        expected,
+        ...(contexts.size ? { context } : {}),
+      },
+    ],
+  };
+  examplesFrom(fixture, rule);
+  return fixture;
+}
+
+/** Save only to a new file; callers preview/redact first. Never stage or overwrite. */
+export async function saveCapturedExample(fixture, output) {
+  await writeFile(output, JSON.stringify(fixture, null, 2) + '\n', {
+    flag: 'wx',
+    mode: 0o600,
+  });
+}