Skip to content

test(research): harden candidate oracle validation - #116

Merged
stacknil merged 7 commits into
mainfrom
stacknil/v07-candidate-oracle-validation
Aug 27, 2026
Merged

test(research): harden candidate oracle validation#116
stacknil merged 7 commits into
mainfrom
stacknil/v07-candidate-oracle-validation

Conversation

@stacknil

@stacknil stacknil commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Summary - install and pin Draft 2020-12 JSON Schema validation for the research fixture gate - run the complete baseline/core/oracle/negative-validation suite inside CTest and cross-platform CI - preserve sanitized CLI failure behavior while adapting validation to #117's explicit baseline context ## Why #115 materializes the bounded oracle. This stacked PR makes schema and malformed-input/cross-reference checks mandatory in CI rather than relying on a developer's optional environment. It remains separate so dependency/CI policy can be reviewed and rolled back independently. ## How to validate text python -m pip install -r requirements-test.txt python -m ruff check scripts/episode_baseline_contract.py scripts/episode_candidate_core.py scripts/evaluate_episode_candidate.py tests/test_episode_candidate_baseline_contract.py tests/test_episode_candidate_core.py tests/test_episode_candidate.py tests/test_episode_candidate_validation.py python -B -m unittest discover -s tests -p "test_episode_candidate*.py" -v cmake -S . -B build -D CMAKE_BUILD_TYPE=Release -D BUILD_TESTING=ON -D LOGLENS_WARNINGS_AS_ERRORS=ON cmake --build build --config Release ctest --test-dir build --build-config Release --output-on-failure Expected: Ruff passes, 22 research tests pass without schema skips, and all 7 CTest entries pass. ## Design decision Keep jsonschema==4.26.0 test-only. The evaluator and baseline contract remain standard-library tools; CI owns schema availability. Semantic validation requires fixture, baseline, and oracle together, while schema validation covers the serialized shape. ## Main risk A pinned test dependency adds update maintenance. The pin is isolated to requirements-test.txt; removing this PR restores the prior dependency-free runtime and optional schema check. ## Compatibility impact No runtime, detector, report, CLI, configuration, or release behavior change. The workflow adds one test-only installation step. ## Rollback path Revert the two validation commits plus the small compatibility-test commit. #117 and #115 remain independently usable without the CI dependency. ## Review policy

Merged after #117 and #115, retargeting to main, fresh exact-head checks (11/11), a 22-test run with no schema skip, and final review of the three-file CI/validation diff.

@stacknil

Copy link
Copy Markdown
Owner Author

Post-CI diff review completed at 63a436783fa4dc808ff6cd901e1af3eb05d93898.

  • Design decision: keep JSON Schema outside the standard-library evaluator and require it only in the test gate.
  • Main risk: a pinned test-only dependency adds limited CI supply-chain surface.
  • Compatibility impact: the base oracle bytes and all runtime/CLI/report behavior are unchanged.
  • Rollback path: revert these two commits; the base PR retains its semantic validator and golden-oracle check.

Evidence: 14/14 candidate tests with no schema skip, sanitized malformed-input regression, 7/7 CTest, CI YAML parse, git diff --check, Gitleaks with no findings, and all GitHub checks passing on both push and PR events. No review/bot comment requires action.

This PR remains draft until #115 completes its delayed review; then retarget it to main.

@stacknil
stacknil changed the base branch from stacknil/v07-candidate-oracle to main August 27, 2026 03:17
@stacknil

Copy link
Copy Markdown
Owner Author

Final current-head review for 00e390374e3c4a0a39a7e27f420101341930180b is complete.

Fresh evidence on this exact head:

  • GitHub checks: 11/11 successful
  • python -m pip install -r requirements-test.txt: satisfied the exact pin
  • Ruff: passed
  • research suite: 22/22 passed with no schema skip
  • strict MSVC /WX build: passed
  • CTest: 7/7 passed
  • git diff --check origin/main...HEAD: passed
  • Gitleaks over origin/main..HEAD: no leaks
  • final 3-file diff review: no unresolved findings

No review or bot comment requires action.

@stacknil
stacknil marked this pull request as ready for review August 27, 2026 03:20
@stacknil
stacknil merged commit 2fbf9bf into main Aug 27, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant