Skip to content

Integrate Judgement feedback tools and strict PR checks - #3310

Merged
mrubens merged 4 commits into
developfrom
codex/judgement-feedback-tools
Sep 30, 2026
Merged

mrubens merged 4 commits into
developfrom
codex/judgement-feedback-tools

Conversation

@mrubens

@mrubens mrubens commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Roomote uses Judgement's shared CLI for local checks, example capture, and calibration report comparison. Findings include a bounded changed-line preview and the rule cutoff; longer checks report progress. The inference adapter continues to use Roomote model settings and keeps caching disabled while backend configuration can change.

  • pnpm judgement capture --rule <id> --path <file> --name <name> --expected pass|violation previews a fixture from the index. Saving requires --output <new-file> and refuses to overwrite existing files.
  • pnpm judgement compare --before <report.json> --after <report.json> identifies per-example regressions on matching fixtures without inference.
  • Adds a strict Judgement PR status: violations, incomplete evidence, timeouts, and inference failures fail the check. The workflow collects snapshot objects in a read-only job without inference credentials. A separate job installs the pinned checker and imports the objects into an empty repository without checking out PR files. Git configuration, hooks, and repository history are excluded. Status writes run in separate jobs; the API credential is available only to the inference step.

Rollout

Pins published @roo-code/judgement@0.4.0 in the root, worker, cloud-agents, and CI workflow. The lockfile uses the registry integrity hash, and tester presets are regenerated against the installed release.

The TYPESAFE_API_KEY repository secret is configured; its presence has been verified, but inference with it has not yet run in this workflow. JUDGEMENT_MODEL defaults to jev-1.13.0 and should match the calibrated model. After the workflow lands and a successful run is verified, add Judgement as a required status check on protected branches. The new workflow has not run live yet.

Rules, cutoffs, and model questions remain unchanged. No generated calibration reports are committed.

Validation

  • 13 focused cloud-agents tests passed (local CLI, inference adapter, tester preset provenance).
  • pnpm judgement:presets --check passed; 47 synthetic examples prepared without inference.
  • pnpm lint:fast, pnpm check-types:fast, and pnpm knip passed.
  • Actionlint passed for both changed workflows.
  • Four CI isolation tests passed: permission boundaries, identical judgments after transferring objects without checkout/config/hooks, corrupt-pack rejection, and failure status for unsuccessful or skipped inference. These tests now run in CI.
  • Targeted oxlint passed for the new tests.
  • Root CI tests declare Judgement as a root dev dependency and import it directly; frozen install, all four isolation tests, and Knip pass after this correction.
  • Shared library: 60 tests passed, including cache reuse, capture safety and encoding, comparison provenance, progress, and strict exit semantics.

@roomote-community

roomote-community Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

No new code issues found. See task

  • scripts/judgement/__tests__/workflow.test.mjs:26 Root-level test creates an unlisted Knip dependency.

Reviewed d300c82

Comment thread .github/workflows/judgement.yml Fixed
Comment thread scripts/judgement/__tests__/workflow.test.mjs Outdated
@mrubens
mrubens marked this pull request as ready for review September 30, 2026 15:06
@mrubens
mrubens merged commit 87d1421 into develop Sep 30, 2026
17 checks passed
@mrubens
mrubens deleted the codex/judgement-feedback-tools branch September 30, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants