diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..092695a --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,27 @@ +# Working on Judgement + +Judgement is a small ESM JavaScript library and CLI. Read README.md and +[docs/agents.md](docs/agents.md) for the public workflow and command semantics. + +## Ownership + +Keep generic checks, fixture loading, isolated Git setup, testing, calibration, +threshold selection, reporting, and cancellation in this library. Consuming +projects own their rules, labeled examples, and any custom inference adapter. +Generated calibration reports belong in local output or CI artifacts. + +## Changes and validation + +- Keep `src/index.d.ts` aligned with public JavaScript exports. +- Keep CLI help, README, and the packaged agent guide aligned with actual behavior. +- Run `npm run check` for code changes. Use `npm pack --dry-run --json` when changing + package files or documentation intended for installed-package users. +- Test Git behavior with disposable repositories. Preserve staged/unstaged + separation, base-policy enforcement, cancellation, and bounded concurrency. +- Unit tests use deterministic evaluators. Live model evaluations must be explicit, + use synthetic data, and report incomplete and failed judgments accurately. +- Do not automatically run calibration during normal hooks, overwrite user rules, + or silently drop examples that fail. Runtime and test commands have different + exit semantics; document and test both. +- Preserve the project's pinned dependencies. Do not publish as a side effect of + testing or preparing a PR. diff --git a/README.md b/README.md index 5fc966d..c4b7312 100644 --- a/README.md +++ b/README.md @@ -12,23 +12,30 @@ Check code changes against your team's business rules, written in plain language "context": ["src/audit/**", "src/auth/**"] }, { + "id": "payment-owner", "rule": "Only workspace owners may change payment settings." } ] } ``` -Save this as `JUDGE.json` at the repository root. Run Judgement before a commit +Save this as `.judgement/rules.json` in your repository. Run Judgement before a commit or in CI. It uses [Jev](https://typesafe.ai) to make focused judgments and reads related source files from the exact Git snapshot being checked. +## Agent instructions + +Coding agents: read [the agent guide](docs/agents.md) before adding or tuning rules. +It is included in the npm package and covers fixtures, commands, uncertainty, and +validation. Projects should keep their rules and examples, not copy the runner. + ## Install Node 22.20+ or 24+ and Git are required. Glob matching uses the small `picomatch` dependency. Install the published package from npm. ```sh -npm install --save-dev --save-exact @roo-code/judgement@0.1.5 +npm install --save-dev --save-exact @roo-code/judgement@0.2.0 export TYPESAFE_API_KEY=... ./node_modules/.bin/judgement check --staged ``` @@ -71,12 +78,12 @@ PR, resolve the merge base explicitly and pass its SHA. Checkout full history. Use policy from a trusted base and provide the API key through CI secrets. Run untrusted contributions without write-capable repository tokens. -| Exit | Strict check | `--hook` | -| --- | --- | --- | -| 0 | Completed without a blocking finding | Completed or explicitly incomplete | -| 1 | Confirmed violation | Confirmed violation | -| 2 | Invalid configuration or invocation | Invalid configuration or invocation | -| 3 | Incomplete, uncertain, service/Git failure | Reported, commit allowed | +| Exit | Strict check | `--hook` | +| ---- | ------------------------------------------ | ----------------------------------- | +| 0 | Completed without a blocking finding | Completed or explicitly incomplete | +| 1 | Confirmed violation | Confirmed violation | +| 2 | Invalid configuration or invocation | Invalid configuration or invocation | +| 3 | Incomplete, uncertain, service/Git failure | Reported, commit allowed | `--advisory` reports all outcomes without blocking. `--timeout-ms` changes the budget (strict mode defaults to 120 seconds). Use JSON `status` to distinguish @@ -107,6 +114,129 @@ Existing policy comes from the base tree. If no policy exists there, a newly staged policy can bootstrap checking. Editing or deleting a policy does not weaken the check for the same commit. Judgement never rewrites your policy. +## Project layout + +```text +.judgement/ + rules.json + examples/ + wording.json + billing-audit.json +``` + +Give each rule a stable `id`. Calibration uses that ID to find its examples. +Policy files and `.judgement/examples/` are excluded from normal rule checks, so +labeled counterexamples do not trigger your commit hook. +Save rules in `.judgement/rules.json`. File and context globs are relative to the +repository root. + +## Test all rules + +```sh +judgement test --dry-run +judgement test --rule wording --repeats 3 +judgement test --format json > /tmp/judgement-tests.json +judgement calibrate --all --format json > /tmp/judgement-calibration.json +``` + +`test` uses configured thresholds; a wrong, incomplete, or failed judgment exits + +1. Dry runs and suites whose results all match their labels exit 0. Setup errors + exit 2. `calibrate --all` compares candidates for every rule and exits 3 if any rule + has no recommendation. Both commands require examples for every selected rule; + missing or invalid examples fail before inference. Use `--examples-dir ` for + a separate suite. Rules run sequentially, with bounded fixture concurrency within + each rule. JSON suite reports include hashes of the normalized policy and exact + fixture text used. Keep generated output local or in CI artifacts. + +The library exposes `testRules(options)`, `calibrateRules(options)`, +`formatExampleSuite(report)`, and `exampleSuiteExitCode(report)`. Options include +an optional `ruleId` (omitting it selects all rules), `examplesDirectory`, and the +same custom `evaluate` callback used by `calibrate`. `testRules` always uses the +configured thresholds. This keeps project-specific inference adapters small. + +## Calibration command + +Save labeled examples in `.judgement/examples/wording.json` for a rule whose +`id` is `wording`: + +```json +{ + "ruleId": "wording", + "examples": [ + { + "name": "Mid-sentence capital", + "path": "docs/guide.md", + "before": "Review the remaining checks.\n", + "after": "Review the remaining Session checks.\n", + "expected": "violation" + }, + { + "name": "Sentence beginning", + "path": "docs/guide.md", + "before": "History is available.\n", + "after": "Session history is available.\n", + "expected": "pass" + } + ] +} +``` + +Each example has a unique name and different `before`/`after` text. Use `null` +for the absent side of an addition or deletion. `path` defaults to `example.md` +and must match the rule's `files`. Optional `context` maps relative paths to +unchanged supporting file contents; the rule's `context` globs select which +ones the checker sees. Paths cannot escape the fixture or replace its policy +or Git metadata. Label valid exceptions and inapplicable changes as `pass`. + +```sh +judgement calibrate --rule wording --dry-run +judgement calibrate --rule wording --repeats 3 --thresholds 0.8,0.85,0.9,0.96 +judgement calibrate --rule wording --format json > calibration.json +``` + +The command reads your **working-tree policy**, then commits each candidate +policy in a disposable Git repository and stages its example there. Your real +policy, working tree, and index remain untouched. Checks use the full evaluator, +including context expansion, with caching disabled. Calibration is opt-in and +is never run by a commit hook. Real runs make paid model requests. + +Defaults: three repetitions, two concurrent fixture checks, a three-second +per-check deadline, and thresholds `0.8`, `0.85`, `0.9`, `0.95`, plus the rule's +current threshold. Use `--examples ` to select a different examples file, +`--concurrency <1–8>` to limit parallel checks, `--timeout-ms ` to match your +check budget, and `--verbose` for progress on stderr. JSON remains on stdout. + +The report includes per-example answers and scores, caught violations, false +blocks, incorrect passes, incomplete results, failures, and timing. It recommends +only candidates that caught every labeled violation with zero false blocks, +provided there were no operational failures anywhere in the run. Among those +candidates it prefers more valid passes, then the highest threshold. A dataset +must contain both labels to receive a recommendation. An incomplete valid case +can remain even at the recommended threshold; inspect those rows before adopting +it. Nothing automatically rewrites your policy. + +Calibration exits `0` when a candidate is recommended (or for a dry run), `3` +when none can be recommended, and `2` for invalid configuration or fixture setup. +Unlike `check`, an expected violation is a successful calibration observation. +Use a held-out examples file to verify the selected threshold before saving it. + +Applications with their own inference configuration can use the same library +harness and supply their existing evaluator: + +```js +import { calibrate, formatCalibrationReport } from '@roo-code/judgement'; + +const report = await calibrate({ + cwd: process.cwd(), + ruleId: 'wording', + evaluate: yourExistingEvaluator, +}); +console.log(formatCalibrationReport(report)); +``` + +The [wording example](examples/wording/.judgement/) includes a policy and fixtures. + ## Calibrating a rule's confidence threshold Choose a threshold from labeled examples of your rule. Confidence measures how @@ -122,7 +252,7 @@ for the distinction. The default `0.85` is a starting point, not a universal cut surrounding context, and small edits to large files. Reserve some examples to validate your choice after tuning. 2. **Use an isolated test repository for each candidate policy.** Commit the - candidate `JUDGE.json` as the baseline, then stage a representative change. + candidate `.judgement/rules.json` as the baseline, then stage a representative change. Merely editing or staging a new threshold in an existing repository will still evaluate against its base policy. Do not overwrite your real staged work to run calibration fixtures. Isolating one rule also makes its results @@ -143,6 +273,7 @@ for the distinction. The default `0.85` is a starting point, not a universal cut backend/model version, outcomes, confidence scores, final statuses, and time. Avoid concurrent checks against the same index; use separate fixtures or run their repetitions sequentially. + 4. **Compare candidate thresholds.** Count violations that would block, valid changes that would incorrectly block, and incomplete checks in each group. Also track outright incorrect passes. A valid change reported as incomplete @@ -154,21 +285,6 @@ for the distinction. The default `0.85` is a starting point, not a universal cut examples, and hook deadlines before adopting it. Recalibrate after changing the rule wording, evidence selection, model, or backend. -For example, a wording-rule calibration used 14 labeled changes with three runs -per change. Initial diff-only scores gave this comparison: - -| Threshold | Violation runs that would block | Valid runs that would incorrectly block | -| --- | --- | --- | -| 0.96 | 14/18 | 0/24 | -| 0.90 | 17/18 | 0/24 | -| 0.85 | 18/18 | 0/24 | - -The full checker at `0.85` then blocked all 18 violation runs, with no false -blocks across 24 valid runs. However, 12 valid runs remained incomplete. That -supported lowering the threshold for this wording rule, while also revealing -uncertainty around permitted exceptions. These are observations from a small -sample, not an accuracy guarantee or a recommended threshold for every rule. - The same threshold applies to `pass`, `not_applicable`, and `violation` answers. An answer below the threshold remains incomplete; `unclear` remains incomplete regardless of confidence. Lowering the threshold cannot fix missing credentials, @@ -251,9 +367,15 @@ Judgement's library report: import { fail, warn } from 'danger'; import { check, formatReport } from '@roo-code/judgement'; -if (!process.env.REVIEW_BASE) throw new Error('Set REVIEW_BASE to the trusted base SHA'); -const report = await check({ base: process.env.REVIEW_BASE, head: 'HEAD', cache: false }); -if (report.status === 'violation' || report.status === 'invalid') fail(formatReport(report)); +if (!process.env.REVIEW_BASE) + throw new Error('Set REVIEW_BASE to the trusted base SHA'); +const report = await check({ + base: process.env.REVIEW_BASE, + head: 'HEAD', + cache: false, +}); +if (report.status === 'violation' || report.status === 'invalid') + fail(formatReport(report)); else if (report.status === 'incomplete') warn(formatReport(report)); ``` diff --git a/docs/agents.md b/docs/agents.md new file mode 100644 index 0000000..fd99145 --- /dev/null +++ b/docs/agents.md @@ -0,0 +1,134 @@ +# Using Judgement as a coding agent + +Judgement checks repository diffs against natural-language rules. The project owns +its rules and labeled examples. Judgement owns running checks, testing examples, +comparing thresholds, and reporting results. + +## Find the project setup + +1. Read the repository's agent instructions and `.judgement/rules.json`. +2. Inspect its package scripts and existing inference adapter. Use the project's + configured command and backend when available; a different model may score the + same examples differently. Do not add credentials to files or output. +3. Use the installed, pinned package. Do not install packages inside a commit hook. + The commands below require the release containing the calibration API; check + `judgement --help` rather than assuming an older installation supports them. + +## Write a rule + +Save rules in `.judgement/rules.json`: + +```json +{ + "criteria": [ + { + "id": "billing-audit", + "rule": "Every successful billing mutation must record an audit event for that mutation. Read-only queries do not require audit events.", + "files": ["src/billing/**"], + "context": ["src/audit/**"], + "threshold": 0.85 + } + ] +} +``` + +- Give each rule a stable ID. Use letters, digits and hyphens for easy filenames. +- Describe one observable requirement, its scope, and meaningful exceptions. + Split unrelated requirements when they need different evidence or thresholds. +- Scope `files` to relevant paths. Globs are relative to the repository root and + match hidden paths too. `context` selects unchanged supporting files from the + same Git snapshot. Keep it focused; it is not a request to inspect the whole repo. +- A threshold is a decision cutoff, not proof of accuracy. The default 0.85 is a + starting point. Do not weaken the rule or lower its threshold just to get green. + +## Save labeled examples + +For each rule, write `.judgement/examples/.json`: + +```json +{ + "ruleId": "billing-audit", + "examples": [ + { + "name": "Mutation without audit event", + "path": "src/billing/update.ts", + "before": "export async function update(db, audit, id) { await db.update(id); await audit.record(id); }\n", + "after": "export async function update(db, audit, id) { await db.update(id); }\n", + "expected": "violation" + }, + { + "name": "Read-only query", + "path": "src/billing/read.ts", + "before": "export async function read(db) { return []; }\n", + "after": "export async function read(db) { return db.findMany(); }\n", + "expected": "pass" + } + ] +} +``` + +Label by the intended policy **before** looking at model scores. Include clear +violations, allowed exceptions, fixes, unrelated edits near old violations, and +realistic context. Keep failing examples; investigate them instead of changing the +expected label to match the model. + +`before` and `after` must differ. Use `null` for the absent side of an addition or +deletion. Optional `context` maps unchanged relative paths to file contents; the +rule's context globs still control which files are included. Fixtures are source +text, not programs to execute. Never put real secrets or private data in examples. +If a rule allows synthetic placeholders, those are valid cases, not positive +secret-detection tests. State that coverage limitation explicitly. + +## Test before tuning + +Run with the project's pinned CLI or equivalent adapter command: + +```sh +judgement test --dry-run +judgement test --rule billing-audit --repeats 3 +judgement test --format json > /tmp/judgement-tests.json +judgement calibrate --rule billing-audit --thresholds 0.8,0.85,0.9,0.95 +judgement calibrate --all --format json > /tmp/judgement-calibration.json +``` + +`test` uses each rule's configured threshold and requires every run to match its +label. A valid example must fully pass; an incomplete result fails the test. +`calibrate` compares candidates. A recommendation can still leave valid examples +incomplete, so read the full report. Live runs make paid model requests; dry runs +make none. A recommendation requires both valid and violating examples; a +single-label set can test behavior but cannot select a threshold. Testing and calibration are explicit operations, never part of the +normal commit hook. + +The harness uses disposable Git repositories, repeats without a verdict cache, +and preserves the real working tree and index. It reads working-tree rules so +edited candidate thresholds can be tested. Normal diff checks use the base policy, +so a rule edit cannot weaken the check on its own commit. + +Inspect false blocks, caught violations, incorrect passes, incomplete results, +operational failures, scores, and timing separately. When a case fails: + +1. Check its label, path, rule wording, and selected supporting context. +2. Distinguish an uncertain correct answer from a wrong answer. Lowering a threshold + cannot repair a wrong outcome and may turn uncertainty into a false block. +3. Repeat the run. Never present one score as a guarantee. +4. Validate a candidate against separately labeled held-out examples before adopting + it. Use `--examples` for one rule, or `test --examples-dir` / `calibrate --all +--examples-dir` for a separate suite. Commit deliberate rule and fixture changes; + keep generated reports local or as CI artifacts. + +## Interpret exit codes and report honestly + +- `test`: 0 means all labels matched (or dry run); 1 means wrong, incomplete, or + operationally failed checks; 2 means invalid input/setup; 130 means interrupted. +- `calibrate`: 0 means a candidate was recommended (or dry run); 3 means no candidate + was recommended for at least one selected rule; 2 means invalid input/setup. +- `check --hook` allows incomplete results to keep commits moving. Strict `check` + rejects incomplete results. Neither behavior makes an incomplete result a pass. + +Report what was tested, repeated counts, backend differences, incomplete cases, +and untested categories. Passing small fixtures does not prove that large commits, +missing context, or every possible violation will be handled correctly. + +Applications with existing credentials should supply their evaluator to `testRules` +or `calibrateRules`. Keep that adapter thin: do not copy the runner, fixture loading, +isolated repository setup, threshold logic, or reporting into each application. diff --git a/examples/wording/.judgement/examples/wording.json b/examples/wording/.judgement/examples/wording.json new file mode 100644 index 0000000..03a4a87 --- /dev/null +++ b/examples/wording/.judgement/examples/wording.json @@ -0,0 +1,33 @@ +{ + "ruleId": "wording", + "examples": [ + { + "name": "Mid-sentence capital", + "path": "docs/guide.md", + "before": "Review the remaining checks.\n", + "after": "Review the remaining Session checks.\n", + "expected": "violation" + }, + { + "name": "Lowercase noun", + "path": "docs/guide.md", + "before": "Review the remaining checks.\n", + "after": "Review the remaining session checks.\n", + "expected": "pass" + }, + { + "name": "Sentence beginning", + "path": "docs/guide.md", + "before": "History is available.\n", + "after": "Session history is available.\n", + "expected": "pass" + }, + { + "name": "Quoted UI label", + "path": "docs/guide.md", + "before": "Click the button to continue.\n", + "after": "Click the exact UI label \"New Task\" to continue.\n", + "expected": "pass" + } + ] +} diff --git a/examples/wording/.judgement/rules.json b/examples/wording/.judgement/rules.json new file mode 100644 index 0000000..81974ef --- /dev/null +++ b/examples/wording/.judgement/rules.json @@ -0,0 +1,10 @@ +{ + "criteria": [ + { + "id": "wording", + "rule": "In user-facing prose, use lowercase session and task except at sentence beginnings, in title case, or when quoting an exact UI label.", + "files": ["docs/**"], + "threshold": 0.85 + } + ] +} diff --git a/package-lock.json b/package-lock.json index c363db1..de8586a 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@roo-code/judgement", - "version": "0.1.5", + "version": "0.2.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@roo-code/judgement", - "version": "0.1.5", + "version": "0.2.0", "license": "MIT", "dependencies": { "picomatch": "4.0.7" diff --git a/package.json b/package.json index 6e2582d..a754a93 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@roo-code/judgement", - "version": "0.1.5", + "version": "0.2.0", "description": "Fast Git checks for plain-language business rules, powered by Jev", "type": "module", "license": "MIT", @@ -21,7 +21,8 @@ "README.md", "LICENSE", "examples", - ".pre-commit-hooks.yaml" + ".pre-commit-hooks.yaml", + "docs" ], "scripts": { "test": "node --test test/*.test.js", diff --git a/src/calibrate.js b/src/calibrate.js new file mode 100644 index 0000000..4af1a65 --- /dev/null +++ b/src/calibrate.js @@ -0,0 +1,401 @@ +import { mkdtemp, mkdir, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; +import { check } from './check.js'; +import { git, gitEnv } from './git.js'; +import { + ConfigurationError, + matches, + readPolicy, + isConfigurationPath, +} from './policy.js'; +import { + createJevEvaluator, + MODEL, + PROTOCOL_VERSION, + validateAnswer, +} from './model.js'; + +const object = (value) => + value !== null && typeof value === 'object' && !Array.isArray(value); +const keys = (value, allowed) => + Object.keys(value).every((key) => allowed.includes(key)); +const text = (value) => typeof value === 'string' && !value.includes('\0'); +const fail = (message) => { + throw new ConfigurationError(message); +}; +export const safePath = (path) => { + if ( + !text(path) || + !path || + /[\\:\r\n]/.test(path) || + ['.judgement/rules.json'].some( + (reserved) => + reserved === path.toLowerCase() || + reserved.startsWith(`${path.toLowerCase()}/`) || + path.toLowerCase().startsWith(`${reserved}/`), + ) + ) + return false; + return path + .split('/') + .every( + (part) => + part && + !['.', '..', '.git'].includes(part.toLowerCase()) && + !/[. ]$/.test(part) && + !/^(con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\.|$)/i.test(part), + ); +}; + +export function examplesFrom(value, criterion) { + if ( + !object(value) || + !keys(value, ['ruleId', 'examples']) || + value.ruleId !== criterion.id || + !Array.isArray(value.examples) || + !value.examples.length + ) + fail( + 'Examples must contain the selected ruleId and a non-empty examples array.', + ); + const names = new Set(); + return value.examples.map((example) => { + if ( + !object(example) || + !keys(example, [ + 'name', + 'path', + 'before', + 'after', + 'expected', + 'context', + ]) || + !text(example.name) || + !example.name.trim() || + names.has(example.name) || + !['pass', 'violation'].includes(example.expected) || + !(example.before === null || text(example.before)) || + !(example.after === null || text(example.after)) || + example.before === example.after + ) + fail( + 'Each example needs a unique name, different before/after text (or null), and expected pass or violation.', + ); + names.add(example.name); + const path = example.path ?? 'example.md'; + if ( + !safePath(path) || + isConfigurationPath(path) || + !matches(path, criterion.files) + ) + fail( + `Example ${JSON.stringify(example.name)} needs a safe relative path matching the rule's files.`, + ); + const context = example.context ?? {}; + if ( + !object(context) || + Object.entries(context).some( + ([path, value]) => !safePath(path) || !text(value), + ) + ) + fail('Example context must map safe relative file paths to text.'); + const paths = [path, ...Object.keys(context)].map((path) => + path.toLowerCase(), + ); + if ( + paths.some((path, i) => + paths.some( + (other, j) => + i !== j && (path === other || path.startsWith(`${other}/`)), + ), + ) + ) + fail('Example file paths must not overlap.'); + return { ...example, path, context }; + }); +} + +/** Run real checks in disposable Git repositories, never in the caller's index. */ +export async function calibrate(options = {}) { + const cwd = resolve(options.cwd ?? process.cwd()); + const policy = await readPolicy(cwd); + if (!options.ruleId) fail('Calibration requires --rule .'); + const criterion = policy.criteria.find((rule) => rule.id === options.ruleId); + if (!criterion) fail('The selected rule ID does not exist in the policy.'); + if ( + !options.examplesPath && + (!safePath(`${criterion.id}.json`) || criterion.id.includes('/')) + ) + fail('This rule ID needs an explicit --examples path.'); + let input; + try { + input = JSON.parse( + await readFile( + resolve( + cwd, + options.examplesPath ?? `.judgement/examples/${criterion.id}.json`, + ), + 'utf8', + ), + ); + } catch { + fail('Cannot read examples JSON. Use --examples .'); + } + const examples = examplesFrom(input, criterion); + const repeats = options.repeats ?? 3; + const concurrency = options.concurrency ?? 2; + const deadlineMs = options.deadlineMs ?? 3000; + if (!Number.isInteger(repeats) || repeats < 1 || repeats > 20) + fail('Repeats must be an integer from 1 to 20.'); + if (!Number.isInteger(concurrency) || concurrency < 1 || concurrency > 8) + fail('Concurrency must be an integer from 1 to 8.'); + if (!Number.isFinite(deadlineMs) || deadlineMs <= 0) + fail('The check timeout must be positive.'); + const candidates = options.thresholds ?? [ + 0.8, + 0.85, + 0.9, + 0.95, + criterion.threshold, + ]; + if ( + !Array.isArray(candidates) || + !candidates.length || + candidates.length > 20 || + candidates.some( + (value) => !Number.isFinite(value) || value < 0 || value > 1, + ) + ) + fail('Thresholds must contain 1 to 20 numbers between 0 and 1.'); + const thresholds = [...new Set(candidates)].sort((a, b) => a - b); + const diagnostics = (message) => { + try { + options.onDiagnostic?.(message); + } catch { + /* Optional UI. */ + } + }; + const report = { + ruleId: criterion.id, + configuredThreshold: criterion.threshold, + backend: options.evaluate ? 'custom' : (options.model ?? MODEL), + protocolVersion: PROTOCOL_VERSION, + repeats, + deadlineMs, + dryRun: Boolean(options.dryRun), + examples: examples.length, + plannedChecks: examples.length * thresholds.length * repeats, + results: [], + summaries: [], + recommendation: null, + recommendationReason: '', + }; + diagnostics( + `calibration: ${report.plannedChecks} checks; ${thresholds.length} thresholds; cache disabled${options.dryRun ? '; no model requests' : '; each check may make multiple model requests'}`, + ); + const evaluate = options.evaluate ?? createJevEvaluator(options); + const controller = new AbortController(); + const signal = options.signal + ? AbortSignal.any([options.signal, controller.signal]) + : controller.signal; + const directory = await mkdtemp(join(tmpdir(), 'judgement-calibrate-')); + // Ignore alternate indexes, worktrees, templates, hooks, and user Git config. + const env = Object.fromEntries( + Object.entries(gitEnv()).filter(([key]) => !key.startsWith('GIT_')), + ); + env.HOME = join(directory, 'home'); + env.USERPROFILE = env.HOME; + const template = join(directory, 'template'); + const jobs = examples.flatMap((example) => + thresholds.map((threshold) => ({ example, threshold })), + ); + let next = 0; + try { + await mkdir(env.HOME); + await mkdir(template); + const workers = await Promise.allSettled( + Array.from({ length: concurrency }, () => + (async () => { + while (next < jobs.length) { + const jobIndex = next++; + const { example, threshold } = jobs[jobIndex]; + signal.throwIfAborted(); + const root = join(directory, String(jobIndex)); + await mkdir(root); + const runGit = (args) => git(root, args, signal, env); + const put = async (path, contents) => { + await mkdir(dirname(join(root, path)), { recursive: true }); + await writeFile(join(root, path), contents); + }; + await runGit(['init', '-q', `--template=${template}`]); + await runGit(['config', 'core.hooksPath', template]); + await runGit(['config', 'core.autocrlf', 'false']); + await runGit([ + 'config', + 'core.excludesFile', + join(template, 'exclude'), + ]); + await put( + '.judgement/rules.json', + JSON.stringify({ criteria: [{ ...criterion, threshold }] }), + ); + for (const [path, contents] of Object.entries(example.context)) + await put(path, contents); + if (example.before !== null) + await put(example.path, example.before); + await runGit(['add', '--force', '--all']); + const tree = await runGit(['write-tree']); + const commit = await runGit([ + '-c', + 'user.name=Judgement', + '-c', + 'user.email=judgement@example.invalid', + '-c', + 'commit.gpgSign=false', + 'commit-tree', + tree, + '-m', + 'Calibration baseline', + ]); + await runGit(['update-ref', 'HEAD', commit]); + if (example.after === null) await rm(join(root, example.path)); + else await put(example.path, example.after); + await runGit(['add', '--force', '--all']); + for (let repetition = 1; repetition <= repeats; repetition++) { + signal.throwIfAborted(); + const answers = []; + let backendErrors = 0; + const result = await check({ + cwd: root, + env, + hook: true, + deadlineMs, + signal, + concurrency: 1, + cache: false, + dryRun: options.dryRun, + evaluate: async (request, signal) => { + try { + const answer = validateAnswer( + await evaluate(request, signal), + request, + ); + // A custom evaluator can ignore cancellation; late results must not mutate the report. + signal.throwIfAborted(); + answers.push({ ...answer, complete: request.complete }); + return answer; + } catch { + if (!signal.aborted) backendErrors++; + throw new Error('Calibration evaluator failed.'); + } + }, + }); + signal.throwIfAborted(); + const operationalFailure = + !options.dryRun && + (backendErrors > 0 || + result.status === 'invalid' || + result.coverage.judged === 0 || + result.messages.some((message) => + message.includes('deadline'), + )); + report.results.push({ + example: example.name, + expected: example.expected, + threshold, + repetition, + status: result.status, + elapsedMs: result.elapsedMs, + judgments: result.coverage.judged, + answers, + operationalFailure, + }); + diagnostics( + `${JSON.stringify(example.name)} threshold ${threshold} run ${repetition}: ${result.status}; ${answers.map((answer) => `${answer.outcome} ${answer.confidence.toFixed(2)}`).join(', ') || 'no judgments'}`, + ); + } + } + })().catch((error) => { + controller.abort(); + throw error; + }), + ), + ); + const failed = workers.find((worker) => worker.status === 'rejected'); + if (failed) throw failed.reason; + } finally { + await rm(directory, { recursive: true, force: true }); + } + report.results.sort( + (a, b) => + a.threshold - b.threshold || + a.example.localeCompare(b.example) || + a.repetition - b.repetition, + ); + report.summaries = thresholds.map((threshold) => { + const runs = report.results.filter((run) => run.threshold === threshold); + const count = (expected, status) => + runs.filter((run) => run.expected === expected && run.status === status) + .length; + return { + threshold, + violationRuns: runs.filter((run) => run.expected === 'violation').length, + validRuns: runs.filter((run) => run.expected === 'pass').length, + caughtViolations: count('violation', 'violation'), + falseBlocks: count('pass', 'violation'), + incorrectPasses: count('violation', 'pass'), + validPasses: count('pass', 'pass'), + incompleteViolations: count('violation', 'incomplete'), + incompleteValid: count('pass', 'incomplete'), + operationalFailures: runs.filter((run) => run.operationalFailure).length, + judgments: runs.reduce((total, run) => total + run.judgments, 0), + meanElapsedMs: Math.round( + runs.reduce((total, run) => total + run.elapsedMs, 0) / runs.length, + ), + }; + }); + const eligible = report.summaries.filter( + (row) => + row.violationRuns && + row.validRuns && + row.caughtViolations === row.violationRuns && + row.falseBlocks === 0, + ); + eligible.sort( + (a, b) => b.validPasses - a.validPasses || b.threshold - a.threshold, + ); + if (options.dryRun) + report.recommendationReason = 'Dry run: no model judgments.'; + else if (report.summaries.some((row) => row.operationalFailures)) + report.recommendationReason = + 'Some checks failed or timed out; rerun before choosing a threshold.'; + else if (!eligible.length) + report.recommendationReason = + 'No candidate caught every labeled violation without false blocks. Include both labels; inspect the examples and clarify the rule or context.'; + else { + report.recommendation = eligible[0].threshold; + report.recommendationReason = + 'Caught all labeled violations with no false blocks; preferred more valid passes, then the highest threshold. This is a sample-based recommendation; verify held-out examples.'; + } + return report; +} + +export function formatCalibrationReport(report) { + const lines = [ + `Calibration: ${report.ruleId} (${report.backend}); ${report.examples} examples × ${report.repeats} repeats per threshold`, + 'Threshold | Caught violations | False blocks | Valid passes | Incomplete (violation/valid) | Errors | Mean ms', + ...report.summaries.map( + (row) => + `${row.threshold.toFixed(2)} | ${row.caughtViolations}/${row.violationRuns} | ${row.falseBlocks}/${row.validRuns} | ${row.validPasses}/${row.validRuns} | ${row.incompleteViolations}/${row.incompleteValid} | ${row.operationalFailures} | ${row.meanElapsedMs}`, + ), + `Recommendation: ${report.recommendation ?? 'none'}. ${report.recommendationReason}`, + 'Incomplete checks permit hook commits but fail strict checks. No policy files were changed.', + 'Example results (confidence may include context expansion):', + ...report.results.map( + (run) => + ` ${JSON.stringify(run.example)} @ ${run.threshold}, run ${run.repetition}: ${run.status}; ${run.answers.map((answer) => `${answer.outcome} ${answer.confidence.toFixed(2)}${answer.complete ? '' : ' [partial]'}`).join(', ') || 'no judgments'}`, + ), + ]; + return lines.join('\n'); +} diff --git a/src/check.js b/src/check.js index 12d7e99..92b24ba 100644 --- a/src/check.js +++ b/src/check.js @@ -11,7 +11,13 @@ import { snapshot, textChunks, } from './git.js'; -import { ConfigurationError, matches, parsePolicy } from './policy.js'; +import { + ConfigurationError, + matches, + parsePolicy, + POLICY_PATH, + isConfigurationPath, +} from './policy.js'; import { createJevEvaluator, MAX_EVIDENCE_BYTES, @@ -128,13 +134,19 @@ async function run(options, result, signal, stopped) { result.coverage.files = snap.changes.length; let policyText = await policyBlob(snap, snap.base, signal); result.policySource = 'base'; + // Reject nonregular proposed policies without letting a new policy weaken this check. + const proposedPolicy = + policyText === null || + snap.changes.some((change) => change.path === POLICY_PATH) + ? await policyBlob(snap, snap.tree, signal) + : null; if (policyText === null) { - policyText = await policyBlob(snap, snap.tree, signal); + policyText = proposedPolicy; result.policySource = 'proposed'; } if (policyText === null) { result.policySource = 'absent'; - trace('no JUDGE.json policy; no model requests'); + trace('no Judgement policy; no model requests'); return; } const policy = parsePolicy(policyText); @@ -177,7 +189,8 @@ async function run(options, result, signal, stopped) { v: PROTOCOL_VERSION, backend: cacheIdentity, searchTree, - focusBlobs: snap.changes.filter((change) => request.focusPaths.includes(change.path)) + focusBlobs: snap.changes + .filter((change) => request.focusPaths.includes(change.path)) .map(({ oldOid, oid }) => ({ oldOid, oid })), request, }), @@ -294,7 +307,7 @@ async function run(options, result, signal, stopped) { signal.throwIfAborted(); const changes = snap.changes.filter( (change) => - change.path !== 'JUDGE.json' && + !isConfigurationPath(change.path) && matches(change.path, criterion.files), ); trace( @@ -381,29 +394,49 @@ async function run(options, result, signal, stopped) { }; if (bytes(request) <= MAX_EVIDENCE_BYTES) { let answer = await ask(request, criterion.id); - const uncertain = answer.outcome === 'unclear' || + const uncertain = + answer.outcome === 'unclear' || answer.confidence < criterion.threshold; if (uncertain && !options.dryRun) { // Expand only when needed; keep every hunk in either request. - const parts = change.mode === '000000' ? null : await load(change.path); + const parts = + change.mode === '000000' + ? null + : await load(change.path); let expanded = parts && { ...request, - evidence: [...request.evidence, ...parts.map((part) => ({ ...part, kind: 'after' }))], + evidence: [ + ...request.evidence, + ...parts.map((part) => ({ ...part, kind: 'after' })), + ], }; if (!expanded || bytes(expanded) > MAX_EVIDENCE_BYTES) { const wider = await evidenceFor(change, 80); expanded = wider && { ...request, - evidence: [...wider, ...context.filter((part) => part.path !== change.path)], + evidence: [ + ...wider, + ...context.filter( + (part) => part.path !== change.path, + ), + ], }; } - if (expanded && bytes(expanded) <= MAX_EVIDENCE_BYTES && - JSON.stringify(expanded.evidence) !== JSON.stringify(request.evidence)) { - trace(`expanding context: ${JSON.stringify(criterion.id)} ${JSON.stringify(change.path)}`); + if ( + expanded && + bytes(expanded) <= MAX_EVIDENCE_BYTES && + JSON.stringify(expanded.evidence) !== + JSON.stringify(request.evidence) + ) { + trace( + `expanding context: ${JSON.stringify(criterion.id)} ${JSON.stringify(change.path)}`, + ); answer = await ask(expanded, criterion.id); consume(answer, expanded); } else { - trace(`context expansion cannot fit or adds no evidence: ${JSON.stringify(change.path)}`); + trace( + `context expansion cannot fit or adds no evidence: ${JSON.stringify(change.path)}`, + ); consume(answer, request); } } else { diff --git a/src/cli.js b/src/cli.js index f3a57ef..ef8c7de 100755 --- a/src/cli.js +++ b/src/cli.js @@ -1,11 +1,28 @@ #!/usr/bin/env node import { parseArgs } from 'node:util'; -import { check, exitCode, formatReport } from './index.js'; +import { + check, + exitCode, + formatReport, + calibrate, + formatCalibrationReport, + testRules, + calibrateRules, + exampleSuiteExitCode, + formatExampleSuite, +} from './index.js'; try { const { values, positionals } = parseArgs({ allowPositionals: true, options: { + all: { type: 'boolean' }, + 'examples-dir': { type: 'string' }, + rule: { type: 'string' }, + examples: { type: 'string' }, + repeats: { type: 'string' }, + thresholds: { type: 'string' }, + concurrency: { type: 'string' }, staged: { type: 'boolean' }, verbose: { type: 'boolean', short: 'v' }, hook: { type: 'boolean' }, @@ -23,10 +40,85 @@ try { }); if (values.help) { console.log( - 'judgement check [--staged [--hook] | --base --head ] [--dry-run] [--format json] [--timeout-ms ] [--no-cache] [--advisory] [--verbose]\nReads JUDGE.json from the base tree. Hooks allow incomplete checks after 3 seconds; strict checks exit 3.', + 'judgement check [--staged [--hook] | --base --head ] [--dry-run] [--format json] [--timeout-ms ] [--no-cache] [--advisory] [--verbose]\nReads .judgement/rules.json from the base tree. Hooks allow incomplete checks after 3 seconds; strict checks exit 3.\njudgement test [--rule ] [--examples-dir ] [--repeats 3] [--dry-run] [--format json]\njudgement calibrate (--rule | --all) [--examples ] [--repeats 3] [--thresholds 0.8,0.85,0.9,0.95] [--concurrency 2] [--timeout-ms 3000] [--dry-run] [--format json] [--verbose]\nCalibration reads the working-tree policy and .judgement/examples/.json; it never edits your policy or index.', ); + } else if (['calibrate', 'test'].includes(positionals[0])) { + if ( + positionals.length !== 1 || + values.staged || + values.base || + values.head || + values.hook || + values.advisory || + (values.all && values.rule) || + (values.examples && values['examples-dir']) || + (positionals[0] === 'test' && values.thresholds !== undefined) || + (values.format && !['json', 'text'].includes(values.format)) + ) + throw new Error('Invalid calibration arguments. Use --help.'); + if (values.thresholds?.split(',').some((value) => !value.trim())) + throw new Error('Thresholds must be comma-separated numbers.'); + const controller = new AbortController(); + const interrupt = () => controller.abort(); + process.once('SIGINT', interrupt); + const suite = positionals[0] === 'test' || values.all; + const run = + positionals[0] === 'test' + ? testRules + : values.all + ? calibrateRules + : calibrate; + if (values['examples-dir'] && !suite) + throw new Error('--examples-dir requires test or calibrate --all.'); + let report; + try { + report = await run({ + examplesDirectory: values['examples-dir'], + signal: controller.signal, + cwd: values.cwd, + ruleId: values.rule, + examplesPath: values.examples, + repeats: + values.repeats === undefined ? undefined : Number(values.repeats), + thresholds: values.thresholds?.split(',').map(Number), + concurrency: + values.concurrency === undefined + ? undefined + : Number(values.concurrency), + deadlineMs: + values['timeout-ms'] === undefined + ? undefined + : Number(values['timeout-ms']), + model: values.model, + dryRun: values['dry-run'], + onDiagnostic: values.verbose + ? (message) => console.error(`judgement: ${message}`) + : undefined, + }); + } finally { + process.removeListener('SIGINT', interrupt); + } + console.log( + values.format === 'json' + ? JSON.stringify(report) + : suite + ? formatExampleSuite(report) + : formatCalibrationReport(report), + ); + process.exitCode = suite + ? exampleSuiteExitCode(report) + : report.dryRun || report.recommendation !== null + ? 0 + : 3; } else { if ( + values.all || + values['examples-dir'] || + values.rule || + values.examples || + values.repeats || + values.thresholds || + values.concurrency || positionals.length > 1 || (positionals[0] && positionals[0] !== 'check') || (values.staged && values.base) || @@ -61,5 +153,5 @@ try { } } catch (error) { console.error(error.message); - process.exitCode = 2; + process.exitCode = error.name === 'AbortError' ? 130 : 2; } diff --git a/src/examples.js b/src/examples.js new file mode 100644 index 0000000..0771687 --- /dev/null +++ b/src/examples.js @@ -0,0 +1,121 @@ +import { createHash } from 'node:crypto'; +import { mkdtemp, mkdir, readFile, writeFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join, resolve } from 'node:path'; +import { + calibrate, + examplesFrom, + safePath, + formatCalibrationReport, +} from './calibrate.js'; +import { ConfigurationError, readPolicy, POLICY_PATH } from './policy.js'; + +const hash = (text) => createHash('sha256').update(text).digest('hex'); + +async function runExamples(options, mode) { + if (mode === 'test' && options.thresholds !== undefined) + throw new ConfigurationError( + 'Tests use configured thresholds. Use calibrate to compare candidates.', + ); + const policy = await readPolicy(options.cwd ?? process.cwd()); + const rules = policy.criteria.filter( + (rule) => !options.ruleId || rule.id === options.ruleId, + ); + if (!rules.length) throw new ConfigurationError('No matching rules.'); + if (options.examplesPath && rules.length !== 1) + throw new ConfigurationError( + 'An explicit examples file requires a single rule.', + ); + const directory = resolve( + options.cwd ?? '.', + options.examplesDirectory ?? '.judgement/examples', + ); + // Snapshot and validate every selected fixture before any inference requests. + const fixtures = await Promise.all( + rules.map(async (rule) => { + if ( + !options.examplesPath && + (!safePath(`${rule.id}.json`) || rule.id.includes('/')) + ) + throw new ConfigurationError( + 'This rule ID needs an explicit examples path.', + ); + const path = options.examplesPath + ? resolve(options.cwd ?? '.', options.examplesPath) + : join(directory, `${rule.id}.json`); + let text, value; + try { + text = await readFile(path, 'utf8'); + value = JSON.parse(text); + } catch { + throw new ConfigurationError( + `Cannot read examples JSON for ${rule.id}.`, + ); + } + examplesFrom(value, rule); + return { rule, text }; + }), + ); + const policyText = JSON.stringify(policy); + const cwd = await mkdtemp(join(tmpdir(), 'judgement-examples-')); + const reports = []; + try { + await mkdir(join(cwd, '.judgement')); + await writeFile(join(cwd, POLICY_PATH), policyText); + for (const [index, { rule, text }] of fixtures.entries()) { + options.signal?.throwIfAborted(); + const examplesPath = join(cwd, `fixture-${index}.json`); + await writeFile(examplesPath, text); + reports.push( + await calibrate({ + ...options, + cwd, + ruleId: rule.id, + examplesPath, + thresholds: mode === 'test' ? [rule.threshold] : options.thresholds, + }), + ); + } + } finally { + await rm(cwd, { recursive: true, force: true }); + } + const passed = + mode === 'test' + ? reports.every((report) => + report.results.every( + (run) => !run.operationalFailure && run.status === run.expected, + ), + ) + : reports.every((report) => report.recommendation !== null); + return { + mode, + status: options.dryRun ? 'dry-run' : passed ? 'pass' : 'fail', + policySha256: hash(policyText), + fixtureSha256: Object.fromEntries( + fixtures.map(({ rule, text }) => [rule.id, hash(text)]), + ), + reports, + }; +} + +export const testRules = (options = {}) => runExamples(options, 'test'); +export const calibrateRules = (options = {}) => + runExamples(options, 'calibrate'); +export function exampleSuiteExitCode(report) { + return report.status !== 'fail' ? 0 : report.mode === 'test' ? 1 : 3; +} +export function formatExampleSuite(report) { + return [ + `judgement ${report.mode}: ${report.status} (${report.reports.length} rules)`, + ...report.reports.map((result) => { + const text = formatCalibrationReport(result); + return report.mode === 'test' + ? text + .replace(/^Calibration:/, 'Examples:') + .split('\n') + .filter((line) => !line.startsWith('Recommendation:')) + .join('\n') + : text; + }), + ].join('\n\n'); +} diff --git a/src/git.js b/src/git.js index 7e8bc2b..33f7b0b 100644 --- a/src/git.js +++ b/src/git.js @@ -1,7 +1,7 @@ import { spawn } from 'node:child_process'; import { createHash } from 'node:crypto'; import { resolve } from 'node:path'; -import { ConfigurationError } from './policy.js'; +import { ConfigurationError, POLICY_PATH } from './policy.js'; export function hash(value) { return createHash('sha256').update(value).digest('hex'); @@ -195,14 +195,14 @@ export async function inventory(snap, signal) { export async function policyBlob(snap, tree, signal) { const record = await git( snap.root, - ['ls-tree', tree, '--', 'JUDGE.json'], + ['ls-tree', tree, '--', POLICY_PATH], signal, snap.env, ); if (!record) return null; const [mode, type, oid] = record.split(/[ \t]/); if (!['100644', '100755'].includes(mode) || type !== 'blob') - throw new ConfigurationError('JUDGE.json must be a regular file.'); + throw new ConfigurationError('Judgement policy must be a regular file.'); return git(snap.root, ['cat-file', 'blob', oid], signal, snap.env); } diff --git a/src/index.d.ts b/src/index.d.ts index 375a541..efd85ac 100644 --- a/src/index.d.ts +++ b/src/index.d.ts @@ -99,3 +99,97 @@ export function installGitHook(options: { command: string[]; signal?: AbortSignal; }): Promise<{ directory: string; previousHooksPath: string | null }>; + +export type CalibrationExample = { + name: string; + /** Relative changed-file path, defaults to example.md. */ + path?: string; + before: string | null; + after: string | null; + expected: 'pass' | 'violation'; + /** Unchanged supporting files in both snapshots. */ + context?: Record; +}; +export type CalibrationExamples = { + ruleId: string; + examples: CalibrationExample[]; +}; +export type CalibrationOptions = BackendOptions & { + cwd?: string; + ruleId: string; + examplesPath?: string; + repeats?: number; + thresholds?: number[]; + concurrency?: number; + /** Per-check deadline, defaults to the hook's 3000 ms. */ + deadlineMs?: number; + signal?: AbortSignal; + dryRun?: boolean; + evaluate?: Evaluator; + onDiagnostic?: (message: string) => void; +}; +export type CalibrationRun = { + example: string; + expected: 'pass' | 'violation'; + threshold: number; + repetition: number; + status: Report['status']; + elapsedMs: number; + judgments: number; + answers: (Answer & { complete: boolean })[]; + operationalFailure: boolean; +}; +export type CalibrationSummary = { + threshold: number; + violationRuns: number; + validRuns: number; + caughtViolations: number; + falseBlocks: number; + incorrectPasses: number; + validPasses: number; + incompleteViolations: number; + incompleteValid: number; + operationalFailures: number; + judgments: number; + meanElapsedMs: number; +}; +export type CalibrationReport = { + ruleId: string; + configuredThreshold: number; + backend: string; + protocolVersion: number; + repeats: number; + deadlineMs: number; + dryRun: boolean; + examples: number; + plannedChecks: number; + results: CalibrationRun[]; + summaries: CalibrationSummary[]; + recommendation: number | null; + recommendationReason: string; +}; +export function calibrate( + options: CalibrationOptions, +): Promise; +export function formatCalibrationReport(report: CalibrationReport): string; + +export type ExampleSuiteOptions = Omit & { + ruleId?: string; + examplesDirectory?: string; +}; +export type ExampleSuiteReport = { + mode: 'test' | 'calibrate'; + status: 'pass' | 'fail' | 'dry-run'; + /** SHA-256 of the normalized policy used for all checks. */ + policySha256: string; + fixtureSha256: Record; + reports: CalibrationReport[]; +}; +export function testRules( + options?: Omit, +): Promise; +export function calibrateRules( + options?: ExampleSuiteOptions, +): Promise; +export function exampleSuiteExitCode(report: ExampleSuiteReport): number; +export function formatExampleSuite(report: ExampleSuiteReport): string; diff --git a/src/index.js b/src/index.js index cb20a16..bd2420e 100644 --- a/src/index.js +++ b/src/index.js @@ -10,3 +10,12 @@ export { } from './model.js'; export { parsePolicy, matches, ConfigurationError } from './policy.js'; export { installGitHook } from './hooks.js'; + +export { calibrate, formatCalibrationReport } from './calibrate.js'; + +export { + testRules, + calibrateRules, + exampleSuiteExitCode, + formatExampleSuite, +} from './examples.js'; diff --git a/src/policy.js b/src/policy.js index f54d3c2..e465cde 100644 --- a/src/policy.js +++ b/src/policy.js @@ -1,8 +1,29 @@ +import { lstat, readFile } from 'node:fs/promises'; +import { join } from 'node:path'; import picomatch from 'picomatch'; const matchesGlob = (path, pattern) => picomatch.isMatch(path, pattern, { dot: true }); +export const POLICY_PATH = '.judgement/rules.json'; + +export const isConfigurationPath = (path) => + path === POLICY_PATH || path.startsWith('.judgement/examples/'); + +export async function readPolicy(cwd) { + let stat; + try { + stat = await lstat(join(cwd, POLICY_PATH)); + } catch (error) { + if (error.code === 'ENOENT') + throw new ConfigurationError(`No ${POLICY_PATH} policy found.`); + throw error; + } + if (!stat.isFile()) + throw new ConfigurationError(`${POLICY_PATH} must be a regular file.`); + return parsePolicy(await readFile(join(cwd, POLICY_PATH), 'utf8')); +} + export class ConfigurationError extends Error {} const keys = (value, allowed) => Object.keys(value).every((key) => allowed.includes(key)); @@ -27,7 +48,7 @@ export function parsePolicy(text) { try { value = JSON.parse(text); } catch { - throw new ConfigurationError('JUDGE.json is not valid JSON.'); + throw new ConfigurationError('Judgement policy is not valid JSON.'); } if ( !object(value) || @@ -36,7 +57,7 @@ export function parsePolicy(text) { !value.criteria.length ) throw new ConfigurationError( - 'JUDGE.json must contain a non-empty criteria array.', + 'Judgement policy must contain a non-empty criteria array.', ); const ids = new Set(); return { diff --git a/test/benchmark.js b/test/benchmark.js index 5c5979a..80e5a2e 100644 --- a/test/benchmark.js +++ b/test/benchmark.js @@ -1,4 +1,4 @@ -import { mkdtemp, writeFile, rm } from 'node:fs/promises'; +import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { execFile } from 'node:child_process'; @@ -11,8 +11,9 @@ try { await git('init', '-q'); await git('config', 'user.name', 'Benchmark'); await git('config', 'user.email', 'benchmark@example.com'); + await mkdir(join(cwd, '.judgement')); await writeFile( - join(cwd, 'JUDGE.json'), + join(cwd, '.judgement/rules.json'), JSON.stringify({ criteria: [{ rule: 'Use sentence case.' }] }), ); await writeFile(join(cwd, 'file.js'), 'before\n'); diff --git a/test/calibrate.test.js b/test/calibrate.test.js new file mode 100644 index 0000000..72be019 --- /dev/null +++ b/test/calibrate.test.js @@ -0,0 +1,335 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { + mkdtemp, + mkdir, + readFile, + readdir, + rm, + writeFile, +} from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; +import { calibrate } from '../src/index.js'; +const exec = promisify(execFile); +const cli = fileURLToPath(new URL('../src/cli.js', import.meta.url)); +const examples = [ + { + name: 'capital', + before: 'remaining checks\n', + after: 'remaining Session checks\n', + expected: 'violation', + }, + { + name: 'lowercase', + before: 'remaining checks\n', + after: 'remaining session checks\n', + expected: 'pass', + }, +]; +async function fixture(t, cases = examples, rule = {}) { + const cwd = await mkdtemp(join(tmpdir(), 'calibration-input-')); + t.after(() => rm(cwd, { recursive: true, force: true })); + await mkdir(join(cwd, '.judgement/examples'), { recursive: true }); + await writeFile( + join(cwd, '.judgement/rules.json'), + JSON.stringify({ + criteria: [ + { + id: 'wording', + rule: 'Use lowercase session.', + threshold: 0.96, + ...rule, + }, + ], + }), + ); + const path = join(cwd, '.judgement/examples/wording.json'); + await writeFile(path, JSON.stringify({ ruleId: 'wording', examples: cases })); + return { + cwd, + path, + run: (options = {}) => + calibrate({ + cwd, + ruleId: 'wording', + thresholds: [0.85, 0.96], + repeats: 2, + ...options, + }), + }; +} +const judge = async (request) => ({ + outcome: request.evidence.some((part) => + part.text.includes('+remaining Session'), + ) + ? 'violation' + : 'pass', + confidence: 0.91, +}); + +test('compares complete checks at each threshold, repeats without caching, and preserves the caller index', async (t) => { + const f = await fixture(t); + await exec('git', ['init', '-q'], { cwd: f.cwd }); + await exec('git', ['add', '.'], { cwd: f.cwd }); + const before = await readFile(join(f.cwd, '.git/index')); + const policy = await readFile(join(f.cwd, '.judgement/rules.json')); + let calls = 0; + const report = await f.run({ + evaluate: async (request) => { + calls++; + return judge(request); + }, + }); + assert.equal(report.recommendation, 0.85); + assert.equal(report.results.length, 8); + assert.equal(report.summaries[0].caughtViolations, 2); + assert.equal(report.summaries[0].validPasses, 2); + assert.equal(report.summaries[1].incompleteViolations, 2); + assert.equal(calls, 12); // Four higher-threshold checks expand context. + assert.deepEqual(await readFile(join(f.cwd, '.git/index')), before); + assert.deepEqual( + await readFile(join(f.cwd, '.judgement/rules.json')), + policy, + ); +}); + +test('recommends nothing when every threshold falsely blocks valid changes', async (t) => { + const f = await fixture(t); + const report = await f.run({ + evaluate: async () => ({ outcome: 'violation', confidence: 1 }), + }); + assert.equal(report.recommendation, null); + assert(report.summaries.every((row) => row.falseBlocks === 2)); +}); + +test('errors, missing labels and ignored cancellation do not yield a recommendation or late mutations', async (t) => { + const f = await fixture(t, [examples[0]]); + const oneLabel = await f.run({ evaluate: judge }); + assert.equal(oneLabel.recommendation, null); + const failed = await f.run({ + evaluate: async () => { + throw new Error('private backend error'); + }, + }); + assert.equal(failed.recommendation, null); + assert.equal(failed.summaries[0].operationalFailures, 2); + assert(!JSON.stringify(failed).includes('private backend error')); + const late = await f.run({ + thresholds: [0.85], + repeats: 1, + deadlineMs: 100, + evaluate: async () => { + await new Promise((resolve) => setTimeout(resolve, 300)); + return { outcome: 'violation', confidence: 1 }; + }, + }); + const snapshot = JSON.stringify(late); + await new Promise((resolve) => setTimeout(resolve, 350)); + assert.equal(JSON.stringify(late), snapshot); + assert.equal(late.recommendation, null); +}); + +test('uses explicit context, handles addition/deletion, and bounds model concurrency', async (t) => { + const cases = [ + { + name: 'addition', + path: 'src/file.ts', + before: null, + after: 'operation();', + expected: 'violation', + context: { 'shared/helper.ts': 'guard();' }, + }, + { + name: 'deletion', + path: 'src/file.ts', + before: 'operation();', + after: null, + expected: 'pass', + context: { 'shared/helper.ts': 'guard();' }, + }, + ]; + const f = await fixture(t, cases, { + files: ['src/**'], + context: ['shared/**'], + }); + let active = 0, + peak = 0; + const report = await f.run({ + concurrency: 1, + thresholds: [0.85], + evaluate: async (request) => { + active++; + peak = Math.max(peak, active); + try { + assert( + request.evidence.some( + (part) => part.kind === 'context' && part.text === 'guard();', + ), + ); + return { + outcome: request.evidence.some((part) => + part.text.includes('new file mode'), + ) + ? 'violation' + : 'pass', + confidence: 1, + }; + } finally { + active--; + } + }, + }); + assert.equal(peak, 1); + assert.equal(report.recommendation, 0.85); +}); + +test('rejects unsafe or unrepresentative fixtures before making model requests', async (t) => { + const f = await fixture(t); + let calls = 0; + const evaluate = async () => { + calls++; + return judge({ evidence: [] }); + }; + for (const path of [ + '../escape', + '/absolute', + '.git/config', + '.GIT/config', + '.judgement/rules.json', + 'a\\b', + 'CON', + 'a/../b', + ]) { + await writeFile( + f.path, + JSON.stringify({ + ruleId: 'wording', + examples: [{ ...examples[0], path }], + }), + ); + await assert.rejects(f.run({ evaluate }), /safe relative path/); + } + await writeFile( + f.path, + JSON.stringify({ + ruleId: 'wording', + examples: [{ ...examples[0], context: { 'example.md': 'collision' } }], + }), + ); + await assert.rejects(f.run({ evaluate }), /overlap/); + await writeFile( + f.path, + JSON.stringify({ + ruleId: 'wording', + examples: [{ ...examples[0], before: examples[0].after }], + }), + ); + await assert.rejects(f.run({ evaluate }), /different before/); + assert.equal(calls, 0); +}); + +test('CLI plans canonical fixtures without credentials and separates diagnostics from JSON', async (t) => { + const f = await fixture(t); + const result = await exec( + process.execPath, + [ + cli, + 'calibrate', + '--cwd', + f.cwd, + '--rule', + 'wording', + '--thresholds', + '0.85', + '--repeats', + '1', + '--dry-run', + '--verbose', + '--format', + 'json', + ], + { env: { PATH: process.env.PATH } }, + ); + const report = JSON.parse(result.stdout); + assert.equal(report.results.length, 2); + assert.equal(report.recommendation, null); + assert(report.results.every((run) => run.judgments === 0)); + assert.match(result.stderr, /cache disabled/); + assert(!result.stdout.includes('remaining Session')); +}); + +test('ignores inherited Git index and worktree overrides', async (t) => { + const f = await fixture(t); + const index = join(f.cwd, 'sentinel'); + await writeFile(index, 'unchanged'); + await exec( + process.execPath, + [ + cli, + 'calibrate', + '--cwd', + f.cwd, + '--rule', + 'wording', + '--thresholds', + '0.85', + '--repeats', + '1', + '--dry-run', + '--format', + 'json', + ], + { + env: { + PATH: process.env.PATH, + GIT_INDEX_FILE: index, + GIT_DIR: join(f.cwd, 'missing'), + GIT_WORK_TREE: f.cwd, + }, + }, + ); + assert.equal(await readFile(index, 'utf8'), 'unchanged'); +}); + +test('chooses the highest threshold when detection and valid passes tie', async (t) => { + const f = await fixture(t); + const report = await f.run({ + repeats: 1, + evaluate: async (request) => ({ ...(await judge(request)), confidence: 1 }), + }); + assert.equal(report.recommendation, 0.96); +}); + +test('cancellation waits for workers and removes disposable repositories', async (t) => { + const f = await fixture(t); + const before = new Set( + (await readdir(tmpdir())).filter((name) => + name.startsWith('judgement-calibrate-'), + ), + ); + const controller = new AbortController(); + await assert.rejects( + f.run({ + signal: controller.signal, + evaluate: async () => { + controller.abort(); + throw new Error('stop'); + }, + }), + ); + const after = (await readdir(tmpdir())).filter( + (name) => name.startsWith('judgement-calibrate-') && !before.has(name), + ); + assert.deepEqual(after, []); +}); + +test('missing policies and unmatched files fail before inference', async (t) => { + const f = await fixture(t, examples, { files: ['src/**'] }); + await assert.rejects(f.run({ evaluate: judge }), /matching the rule/); + await rm(join(f.cwd, '.judgement/rules.json')); + await assert.rejects(f.run(), /No \.judgement\/rules.json policy found/); +}); diff --git a/test/check.test.js b/test/check.test.js index d7d68bd..5057908 100644 --- a/test/check.test.js +++ b/test/check.test.js @@ -28,7 +28,8 @@ async function repo(t, criteria = [rule], initial = true) { await writeFile(join(root, path), text); }; const git = (...args) => exec('git', args, { cwd: root }); - if (criteria) await put('JUDGE.json', JSON.stringify({ criteria })); + if (criteria) + await put('.judgement/rules.json', JSON.stringify({ criteria })); await put('billing.js', 'original\n'); if (initial) { await git('add', '.'); @@ -113,7 +114,7 @@ test('each changed file receives explicitly selected supporting context', async test('policy edits do not weaken their own check and are never restored', async (t) => { const r = await repo(t); const updated = JSON.stringify({ criteria: [{ rule: 'A weaker rule.' }] }); - await r.put('JUDGE.json', updated); + await r.put('.judgement/rules.json', updated); await r.put('billing.js', 'changed\n'); await r.git('add', '.'); await r.run({ @@ -122,7 +123,10 @@ test('policy edits do not weaken their own check and are never restored', async return pass(); }, }); - assert.equal(await readFile(join(r.root, 'JUDGE.json'), 'utf8'), updated); + assert.equal( + await readFile(join(r.root, '.judgement/rules.json'), 'utf8'), + updated, + ); }); test('bootstrap policy and initial commits work', async (t) => { @@ -136,7 +140,7 @@ test('bootstrap policy and initial commits work', async (t) => { test('absent and invalid policies have distinct outcomes', async (t) => { const r = await repo(t, null); assert.equal((await r.run()).status, 'pass'); - await r.put('JUDGE.json', '{oops'); + await r.put('.judgement/rules.json', '{oops'); await r.git('add', '.'); const report = await r.run(); assert.equal(report.status, 'invalid'); @@ -473,16 +477,18 @@ test('small edit to a large file uses all diff hunks without the whole file', as await r.put('billing.js', lines.join('')); await r.git('add', '.'); let calls = 0; - const report = await r.run({ evaluate: async (request) => { - calls++; - assert(request.complete); - assert(request.evidence.every((part) => part.kind === 'patch')); - const text = JSON.stringify(request); - assert.match(text, /first edit/); - assert.match(text, /second edit/); - assert(Buffer.byteLength(text) < 5000); - return pass(); - } }); + const report = await r.run({ + evaluate: async (request) => { + calls++; + assert(request.complete); + assert(request.evidence.every((part) => part.kind === 'patch')); + const text = JSON.stringify(request); + assert.match(text, /first edit/); + assert.match(text, /second edit/); + assert(Buffer.byteLength(text) < 5000); + return pass(); + }, + }); assert.equal(report.status, 'pass'); assert.equal(calls, 1); }); @@ -492,20 +498,29 @@ test('uncertain diff expands to the staged file and can complete', async (t) => await r.put('billing.js', 'guard();\n' + 'context\n'.repeat(60) + 'old();\n'); await r.git('add', '.'); await r.git('commit', '-qm', 'baseline'); - await r.put('billing.js', 'guard();\n' + 'context\n'.repeat(60) + 'newOperation();\n'); + await r.put( + 'billing.js', + 'guard();\n' + 'context\n'.repeat(60) + 'newOperation();\n', + ); await r.git('add', '.'); await r.put('billing.js', 'unstaged content must not be used'); let calls = 0; - const report = await r.run({ evaluate: async (request) => { - calls++; - if (calls === 1) { - assert.doesNotMatch(JSON.stringify(request), /guard\(\)/); - return { outcome: 'unclear', confidence: 0.99 }; - } - assert(request.evidence.some((part) => part.kind === 'after' && part.text.includes('guard();'))); - assert.doesNotMatch(JSON.stringify(request), /unstaged content/); - return pass(); - } }); + const report = await r.run({ + evaluate: async (request) => { + calls++; + if (calls === 1) { + assert.doesNotMatch(JSON.stringify(request), /guard\(\)/); + return { outcome: 'unclear', confidence: 0.99 }; + } + assert( + request.evidence.some( + (part) => part.kind === 'after' && part.text.includes('guard();'), + ), + ); + assert.doesNotMatch(JSON.stringify(request), /unstaged content/); + return pass(); + }, + }); assert.equal(calls, 2); assert.equal(report.status, 'pass'); }); @@ -520,13 +535,66 @@ test('large-file uncertainty widens context once and remains incomplete if unres await r.put('billing.js', lines.join('')); await r.git('add', '.'); const sizes = []; - const report = await r.run({ evaluate: async (request) => { - sizes.push(JSON.stringify(request).length); - assert(request.complete); - assert(request.evidence.every((part) => part.kind === 'patch')); - return { outcome: 'unclear', confidence: 0.99 }; - } }); + const report = await r.run({ + evaluate: async (request) => { + sizes.push(JSON.stringify(request).length); + assert(request.complete); + assert(request.evidence.every((part) => part.kind === 'patch')); + return { outcome: 'unclear', confidence: 0.99 }; + }, + }); assert.equal(sizes.length, 2); assert(sizes[1] > sizes[0]); assert.equal(report.status, 'incomplete'); }); + +test('policy edits still enforce the base rule', async (t) => { + const r = await repo(t); + await r.put( + '.judgement/rules.json', + JSON.stringify({ criteria: [{ rule: 'weaker replacement' }] }), + ); + await r.put('billing.js', 'changed'); + await r.git('add', '.'); + const edited = await r.run({ + evaluate: async (request) => { + assert.equal(request.rule, rule.rule); + assert.deepEqual(request.focusPaths, ['billing.js']); + return bad(); + }, + }); + assert.equal(edited.status, 'violation'); + await r.git('commit', '-qm', 'update policy'); + await r.put('billing.js', 'changed again'); + await r.git('add', '.'); + const canonical = await r.run({ + evaluate: async (request) => { + assert.equal(request.rule, 'weaker replacement'); + return pass(); + }, + }); + assert.equal(canonical.status, 'pass'); +}); + +test('nonregular policies are invalid', async (t) => { + const r = await repo(t); + await r.git('rm', '.judgement/rules.json'); + await r.put('.judgement/rules.json/nested', 'not a policy'); + await r.git('add', '.'); + assert.equal((await r.run()).status, 'invalid'); +}); + +test('saved counterexamples do not trigger normal rule checks', async (t) => { + const r = await repo(t); + await r.put('.judgement/examples/wording.json', 'intentional violation'); + await r.git('add', '.'); + let calls = 0; + const report = await r.run({ + evaluate: async () => { + calls++; + return bad(); + }, + }); + assert.equal(report.status, 'pass'); + assert.equal(calls, 0); +}); diff --git a/test/examples.test.js b/test/examples.test.js new file mode 100644 index 0000000..e018d5d --- /dev/null +++ b/test/examples.test.js @@ -0,0 +1,138 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; +import { + testRules, + calibrateRules, + exampleSuiteExitCode, +} from '../src/index.js'; +const exec = promisify(execFile); +const cli = fileURLToPath(new URL('../src/cli.js', import.meta.url)); +async function fixture(t) { + const cwd = await mkdtemp(join(tmpdir(), 'example-suite-test-')); + t.after(() => rm(cwd, { recursive: true, force: true })); + await mkdir(join(cwd, '.judgement/examples'), { recursive: true }); + await writeFile( + join(cwd, '.judgement/rules.json'), + JSON.stringify({ + criteria: [ + { id: 'first', rule: 'Reject forbidden.', threshold: 0.8 }, + { id: 'second', rule: 'Reject forbidden.', threshold: 0.95 }, + ], + }), + ); + for (const ruleId of ['first', 'second']) + await writeFile( + join(cwd, `.judgement/examples/${ruleId}.json`), + JSON.stringify({ + ruleId, + examples: [ + { + name: 'bad', + before: 'old', + after: 'forbidden', + expected: 'violation', + }, + { name: 'good', before: 'old', after: 'allowed', expected: 'pass' }, + ], + }), + ); + return cwd; +} +const evaluate = async (request) => ({ + outcome: request.evidence.some((part) => part.text.includes('forbidden')) + ? 'violation' + : 'pass', + confidence: 0.9, +}); + +test('test uses each configured threshold; calibration compares candidates across rules', async (t) => { + const cwd = await fixture(t); + const report = await testRules({ cwd, repeats: 1, evaluate }); + assert.equal(report.reports.length, 2); + assert.equal(report.status, 'fail'); + assert.equal(exampleSuiteExitCode(report), 1); + assert.equal(report.reports[0].results[0].status, 'violation'); + assert(report.reports[1].results.every((run) => run.status === 'incomplete')); + assert.match(report.policySha256, /^[a-f0-9]{64}$/); + const sweep = await calibrateRules({ + cwd, + repeats: 1, + thresholds: [0.8, 0.95], + evaluate, + }); + assert.equal(sweep.status, 'pass'); + assert.equal(exampleSuiteExitCode(sweep), 0); + assert(sweep.reports.every((report) => report.recommendation === 0.8)); + await assert.rejects( + testRules({ cwd, thresholds: [0.8] }), + /configured thresholds/, + ); +}); + +test('missing or invalid later fixtures fail before any model requests', async (t) => { + const cwd = await fixture(t); + await writeFile(join(cwd, '.judgement/examples/second.json'), '{}'); + let calls = 0; + await assert.rejects( + testRules({ + cwd, + evaluate: async () => { + calls++; + return { outcome: 'pass', confidence: 1 }; + }, + }), + ); + assert.equal(calls, 0); +}); + +test('suite snapshots later fixtures before starting inference', async (t) => { + const cwd = await fixture(t); + let changed = false; + const report = await testRules({ + cwd, + repeats: 1, + evaluate: async (request) => { + if (!changed) { + changed = true; + await writeFile( + join(cwd, '.judgement/examples/second.json'), + 'invalid replacement', + ); + } + return evaluate(request); + }, + }); + assert.equal(report.reports.length, 2); + assert.equal(report.reports[1].examples, 2); +}); + +test('CLI test and calibration --all support JSON dry runs without credentials', async (t) => { + const cwd = await fixture(t); + for (const args of [['test'], ['calibrate', '--all']]) { + const { stdout } = await exec(process.execPath, [ + cli, + ...args, + '--cwd', + cwd, + '--dry-run', + '--repeats', + '1', + '--format', + 'json', + ]); + const report = JSON.parse(stdout); + assert.equal(report.status, 'dry-run'); + assert.equal(report.reports.length, 2); + assert( + report.reports.every((report) => + report.results.every((run) => run.judgments === 0), + ), + ); + } +});