Skip to content

Run the pinned GNN conformance checks in CI - #8

Merged
docxology merged 1 commit into
mainfrom
ci/gnn-conformance-20261102
Nov 3, 2026
Merged

docxology merged 1 commit into
mainfrom
ci/gnn-conformance-20261102

Conversation

@docxology

Copy link
Copy Markdown
Collaborator

Why

tests/test_gnn_conformance.py is the only check that ties gnn/gaussian_agent.gnn.md to the parser revision pinned in gnn/upstream.json, but CI never ran it with a GNN checkout: the pinned-parser half skipped and the green run proved nothing about the pin.

This adds a gnn conformance (pinned v5.0.3) job that provisions the pinned revision and runs both the strict validator and the conformance tests. The fixture now fails instead of skipping when GNN_ROOT selects a missing or unusable checkout, so the job cannot pass vacuously.

Modelled on ActInf_RxInfer and ActiveBlockference (a separate gnn-conformance job with its own pinned GNN runtime) and on GEN24 / active_inference_power (fetch the pinned tree at the recorded SHA and validate against it).

What changed

  • .github/workflows/ci.yml — new gnn-conformance job. It fetches 8f293d9b… (tag v5.0.3) from the public upstream repository at that exact SHA, checks out detached, creates .venv inside the checkout with numpy and defusedxml (what the pinned extractor needs), then runs python scripts/validate_gnn_spec.py and python -m pytest tests/test_gnn_conformance.py -q. A pin that no longer resolves fails at git fetch instead of skipping. Actions are pinned by SHA, reusing the SHAs the repository already uses.
  • tests/test_gnn_conformance.py — the extracted_model fixture now fails (pytest.fail) when GNN_ROOT is set but the pinned parser is unavailable, and skips only when GNN_ROOT is unset. Docstring updated.
  • AGENTS.md — notes that CI runs both commands against the pinned revision.

Negative tests: the new check can fail

Every mutation was made in a scratch copy and reverted; nothing is committed.

Baseline against a checkout of the pinned revision: scripts/validate_gnn_spec.py exit 0 (printing the four conformance checks) and 10 passed.

  1. Pin moved to a resolvable but wrong commit — gnn/upstream.json commit set to 8b864ae68dd20527479f2901174c9864d56f3c04 (upstream main tip) and fetched into a scratch checkout, so the fetch itself succeeds:
FAIL: pinned tag v5.0.3 resolves to 8f293d9ba810e6eda5c09842d9c7ed9ffa5f92c1, not 8b864ae68dd20527479f2901174c9864d56f3c04
SCRIPT_EXIT=1

FAILED tests/test_gnn_conformance.py::test_pin_record_is_canonical
ERROR tests/test_gnn_conformance.py::test_extraction_uses_schema_major_one
ERROR tests/test_gnn_conformance.py::test_extracted_spec_conforms_to_the_agent
1 failed, 7 passed, 2 errors in 0.53s
exit 1
  1. Specification drift — the prior covariance Q diagonal 0.5 → 0.25 in gnn/gaussian_agent.gnn.md:
FAIL: Q must be (1 / prior_precision) I
SCRIPT_EXIT=1

FAILED tests/test_gnn_conformance.py::test_extracted_spec_conforms_to_the_agent
1 failed, 9 passed in 1.37s
exit 1
  1. Explicit selection, checkout absent — GNN_ROOT pointing at a missing directory now errors instead of skipping:
ERROR tests/test_gnn_conformance.py::test_extraction_uses_schema_major_one
ERROR tests/test_gnn_conformance.py::test_extracted_spec_conforms_to_the_agent
8 passed, 2 errors in 32.31s
exit 1
  1. Dead pin — a commit SHA that does not exist upstream: the provisioning step's git fetch --depth 1 origin <sha> exits 128 (upload-pack: not our ref), so the job stops before the validator runs.

Verification

  • actionlint clean on the edited workflow.
  • The job's steps were replayed locally against a scratch checkout of the pinned revision: validator exit 0, 10 passed.
  • No repository code or dependency changes outside the conformance file's skip/fail guard.

I will not merge this until the new job is green here.

Add a gnn-conformance job that fetches the GNN revision recorded in
gnn/upstream.json from the public upstream repository at that exact SHA,
provisions its interpreter, runs the strict scripts/validate_gnn_spec.py
gate, and runs tests/test_gnn_conformance.py. The conformance tests now
fail instead of skipping when GNN_ROOT selects a missing or unusable
checkout, so the job cannot pass vacuously.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@docxology
docxology merged commit 07d62b4 into main Nov 3, 2026
4 checks passed
@docxology
docxology deleted the ci/gnn-conformance-20261102 branch November 3, 2026 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant