Finish the gate: keep private specs and baselines out of the repository

.github/workflows/gate.yml

@@ -0,0 +1,114 @@
+# The merge gate. Every push to main and every pull request runs the tests and
+# then replays the three folders of ci-corpus, the committed corpus, which
+# ci-baseline.json names. The job is CI's report on a branch and nothing
+# merges on it: the workbench merges with `gh pr merge` as soon as its own
+# check passes, and that check replays the private corpus, whose folders a
+# runner cannot read, on the machine that holds it.
+name: gate
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  gate:
+    name: gate
+    # Pinned rather than ubuntu-latest: the runner's TeX Live decides what
+    # xelatex writes, and the PDF's bytes are the OCR cache's key.
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+
+      # Pandoc reads every source document, so its version decides the blocks
+      # a spec selects. ubuntu-24.04 packages pandoc 3.1.3, under which every
+      # ci-corpus document faults on a block no selector reaches, and the
+      # tests fail with it. This release is the one the baselines were
+      # recorded under.
+      - name: Install pandoc
+        run: |
+          curl -fsSL -o "${RUNNER_TEMP}/pandoc.deb" \
+            https://github.com/jgm/pandoc/releases/download/3.9.0.2/pandoc-3.9.0.2-1-amd64.deb
+          sudo dpkg -i "${RUNNER_TEMP}/pandoc.deb"
+          pandoc --version | head -1
+
+      # The Dockerfile's package list, which is the one record of what the PDF
+      # generator's template.latex loads.
+      - name: Install poppler and TeX Live
+        run: |
+          sudo apt-get update
+          sudo apt-get install --no-install-recommends -y \
+            poppler-utils \
+            texlive-xetex \
+            texlive-latex-recommended \
+            texlive-latex-extra \
+            texlive-science \
+            texlive-lang-chinese \
+            texlive-lang-arabic \
+            texlive-bibtex-extra \
+            texlive-plain-generic \
+            texlive-fonts-recommended \
+            lmodern \
+            fonts-noto-core \
+            fonts-noto-cjk
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install poetry
+        run: pipx install poetry
+
+      - name: Install the package
+        run: poetry install --with dev
+
+      # SOURCE_DATE_EPOCH fixes the timestamp xelatex writes into the PDF, so
+      # the same tex source compiles to the same bytes and the OCR cache key
+      # does not change between runs.
+      - name: Compile the CI corpus PDF
+        run: |
+          cd ci-corpus/tex
+          SOURCE_DATE_EPOCH=0 FORCE_SOURCE_DATE=1 \
+            xelatex -interaction=nonstopmode -output-directory=../pdf sheet-1.tex
+          rm -f ../pdf/sheet-1.aux ../pdf/sheet-1.log
+
+      # Keyed by the PDF's bytes, which is also how ocr_pdf names its entry
+      # inside the directory. A restored entry for another PDF is unused.
+      - name: Restore the OCR cache
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/in2lambda-agent
+          key: ocr-${{ hashFiles('ci-corpus/pdf/*.pdf') }}
+          restore-keys: ocr-
+
+      - name: Tests
+        run: poetry run pytest -q
+
+      - name: Gate
+        env:
+          MATHPIX_APP_ID: ${{ secrets.MATHPIX_APP_ID }}
+          MATHPIX_API_KEY: ${{ secrets.MATHPIX_API_KEY }}
+        run: poetry run in2lambda-agent gate ci-baseline.json
+
+      # What this run would record, so that a baseline change is committed from
+      # what CI saw. Written whether or not the gate passed. The file is
+      # recorded where it is checked out, because a baseline names its specs
+      # relative to its own directory and a copy under RUNNER_TEMP would find
+      # none; the job throws the checkout away.
+      - name: Record what this run did
+        if: always()
+        env:
+          MATHPIX_APP_ID: ${{ secrets.MATHPIX_APP_ID }}
+          MATHPIX_API_KEY: ${{ secrets.MATHPIX_API_KEY }}
+        run: poetry run in2lambda-agent gate --record ci-baseline.json
+
+      - name: Upload it
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: recorded-baseline
+          path: ci-baseline.json

.gitignore

@@ -5,5 +5,12 @@ __pycache__/
 *.egg-info/
 .venv/
 dist/
+# The specs for ExampleContents quote the headings of private documents, and
+# gate-baseline.json beside them records those documents' file names and the
+# absolute path of the corpus on one machine.
+corpus-specs/
 # in2lambda's KaTeX converter writes this into the working directory on import.
 log
+# The gate's CI corpus: xelatex compiles this PDF from ci-corpus/tex/sheet-1.tex
+# in the workflow, so the repository holds the tex source and not the output.
+ci-corpus/pdf/*.pdf

README.md

@@ -252,7 +252,7 @@ poetry run in2lambda-agent corpus ExampleContents --suffix tex --suffix md
 In full:
 
 ```sh
-poetry run in2lambda-agent corpus ROOT [PATH ...] [--suffix S] [--replay] [--rounds N] [--results FILE] [--work DIR] [--specs DIR]
+poetry run in2lambda-agent corpus ROOT [PATH ...] [--suffix S] [--replay] [--rounds N] [--results FILE] [--work DIR] [--specs DIR] [--cache DIR]
 ```
 
 `ROOT` is the corpus directory and each `PATH` a folder under it to run, defaulting to
@@ -285,6 +285,117 @@ review, rejections
 where its value comes from. One document that fails is one row and not the end of the
 sweep, and a set whose folder cannot be copied is a row for each of its documents.
 
+`--cache` is where the OCR of each PDF is kept. It defaults to `./.in2lambda-agent`,
+the directory `run` caches into, so a sweep over PDFs that `run` has already converted
+makes no Mathpix call and needs no Mathpix credentials.
+
+## Gate
+
+Nothing merges without a replay over real documents. `gate` reruns the saved specs
+over the folders a baseline file names, and compares what each folder did this run
+with what the baseline records:
+
+```sh
+poetry run in2lambda-agent gate BASELINE [--record] [--cache DIR] [--work DIR]
+```
+
+Every run is `corpus --replay`, so no model call is made. The command prints one line
+per folder, and exits 1 when a folder builds fewer documents than the baseline records
+or when a single document does worse than the baseline records it doing. The second
+check is what a baseline of no builds rests on: a corpus where every document faults
+still reports the document that stops being read.
+
+```
+work      /tmp/in2lambda-agent-gate-3f1a
+tex                  built 2  faulted 0  build refused 0  skipped 0  no spec 2   (baseline built 4)
+  worse  tex/sheet-3.tex  built -> no spec: replay: no model call is allowed
+```
+
+The gate reads `specs`, and a folder's `root` where `root` is relative, from the
+directory `BASELINE` is in, so the command gives the same run from any directory.
+
+There are two corpora and a baseline for each:
+
+| Baseline | Corpus | Run by |
+| --- | --- | --- |
+| `corpus-specs/gate-baseline.json` | the three folders of `ExampleContents` | the workbench check |
+| `ci-baseline.json` | the three folders of `ci-corpus` | `.github/workflows/gate.yml` |
+
+The repository holds `ci-baseline.json` and the specs it names, under
+`ci-corpus/specs/`, because `ci-corpus` is synthetic. The repository holds neither the
+specs for `ExampleContents` nor the baseline that names them: the specs quote the
+headings of private documents, and the baseline records those documents' file names and
+the path of the corpus on one machine. `.gitignore` lists `corpus-specs/`, and
+`gate-baseline.json` sits in that directory beside the specs it reads, with `"specs":
+"."`.
+
+`ci-corpus` is synthetic, so the repository holds its documents — every one but the
+PDF, which xelatex compiles from `ci-corpus/tex/sheet-1.tex`. Run the command the
+workflow runs before `gate ci-baseline.json`, because the PDF's bytes are the key the
+OCR cache reads under:
+
+```sh
+cd ci-corpus/tex
+SOURCE_DATE_EPOCH=0 FORCE_SOURCE_DATE=1 \
+  xelatex -interaction=nonstopmode -output-directory=../pdf sheet-1.tex
+rm -f ../pdf/sheet-1.aux ../pdf/sheet-1.log
+```
+
+Without the PDF the `pdf` folder holds no document, builds 0 against a recorded 1,
+and the gate exits 1.
+
+`ExampleContents` is a set of private documents and is never in the repository: the
+gate reads it at the absolute `root` that `gate-baseline.json` gives, which is a path
+on the machine the check runs on.
+
+Each folder's `root` and `suffixes` are written by hand. `built`, the count of documents
+that built, and `documents`, the outcome of each single document, are what `--record`
+writes. A folder the file records no `built` for passes on any count, and its line reads
+`(not recorded)`. A change to a recorded count or outcome belongs in a pull request that
+says why the count or the outcome changed.
+
+Over `ExampleContents` today, every document replays to `faulted` and the baseline
+records 0 built for all three folders: pandoc's line wrapping is reported as a math
+delimiter error, and each document needs a fixing round that a replay does not run. The
+recorded outcomes are what the gate defends until a later ticket raises the count.
+
+`--cache` defaults to `~/.cache/in2lambda-agent`, which is outside every worktree,
+because the gate runs in a worktree of its own: a PDF converted on one branch is
+converted again on the next if the cache sits in the branch's directory. `--work`
+defaults to a new directory under the system temp
+directory, which the gate does not delete: read the
+drafts of a folder that failed there. The gate also copies the spec tree into the work
+directory and replays the copy, because a sweep appends a record of each run beside the
+spec it reads. Neither directory is inside the repository, so `git status` after a gate
+run reports no new file.
+
+`.github/workflows/gate.yml` runs pytest and then `gate ci-baseline.json` on every push
+to `main` and every pull request. Mathpix reads `ci-corpus/pdf/sheet-1.pdf` once and the
+workflow stores the markdown in the Actions cache under the PDF's hash. The job needs
+two repository secrets, `MATHPIX_APP_ID` and `MATHPIX_API_KEY`.
+
+A pull request from a fork is given neither secret. `actions/cache` restores the cache
+of the base branch for a fork, and the cached markdown is what the `pdf` folder then
+replays, so the job passes without the secrets. If the cache is empty — the PDF's bytes
+changed, or GitHub evicted the entry — Mathpix cannot be called and the `pdf` folder
+builds 0 against a recorded 1, so the job fails. Push that branch to a branch of this
+repository, where the secrets are read, and the job runs Mathpix once.
+
+The job is CI's report on a branch and no merge waits for it. The workbench merges with
+`gh pr merge` as soon as its own check passes, and `gh pr merge` cannot wait for a
+GitHub check, so requiring the job on `main` would refuse every merge the workbench
+makes.
+
+The workbench check runs the gate over `ExampleContents`, which is the larger corpus of
+the two:
+
+```sh
+poetry install -q --with dev && poetry run pytest -q && poetry run in2lambda-agent gate /Users/peterbjohnson/code/lambdafeedback/in2lambda-agent/corpus-specs/gate-baseline.json
+```
+
+The path is absolute because the check runs in a worktree and the worktree holds
+neither the baseline nor the specs it names.
+
 ## Docker
 
 The image carries pandoc, a TeX Live whose xelatex runs the PDF generator's

ci-baseline.json

@@ -0,0 +1,37 @@
+{
+  "specs": "ci-corpus/specs",
+  "folders": {
+    "tex": {
+      "root": "ci-corpus",
+      "suffixes": [
+        "tex"
+      ],
+      "built": 3,
+      "documents": {
+        "tex/sheet-1.tex": "built",
+        "tex/sheet-2.tex": "built",
+        "tex/sheet-3.tex": "built"
+      }
+    },
+    "docx": {
+      "root": "ci-corpus",
+      "suffixes": [
+        "docx"
+      ],
+      "built": 1,
+      "documents": {
+        "docx/sheet.docx": "built"
+      }
+    },
+    "pdf": {
+      "root": "ci-corpus",
+      "suffixes": [
+        "pdf"
+      ],
+      "built": 1,
+      "documents": {
+        "pdf/sheet-1.pdf": "built"
+      }
+    }
+  }
+}

ci-corpus/specs/docx/in2lambda-spec.yaml

@@ -0,0 +1,6 @@
+ignore:   Header
+question: Para text~'^[A-Z]'
+part:     ListItem
+solution: after Header text=Solutions, Para
+strip:    ['^\([a-z]\) ', '^\d+\([a-z]\) ']
+layout:   PartsSepSol

ci-corpus/specs/pdf/in2lambda-spec.yaml

@@ -0,0 +1,10 @@
+# The spec for the pdf set, which is sheet-1.tex compiled by xelatex and read
+# back by Mathpix. The OCR writes the sheet's shape back: the question as a
+# paragraph, its parts as a numbered list, and the solutions under the
+# `Solutions` heading. The selectors are the tex set's for that reason.
+ignore:   Header
+question: Para text~'^[A-Z]'
+part:     ListItem
+solution: after Header text=Solutions, Para
+strip:    ['^\d+\([a-z]\) ']
+layout:   PartsSepSol

ci-corpus/specs/tex/in2lambda-spec.yaml

@@ -0,0 +1,10 @@
+# The spec for the tex set: each question a paragraph, its parts the items of
+# the list under it, and its solutions under a `Solutions` heading — in the
+# same file for sheet-1 and sheet-3, and in sheet-2-solutions.tex for sheet-2.
+# A solution begins with its label, `1(a)`, so no solution matches `question`.
+ignore:   Header
+question: Para text~'^[A-Z]'
+part:     ListItem
+solution: after Header text=Solutions, Para
+strip:    ['^\d+\([a-z]\) ']
+layout:   PartsSepSol

ci-corpus/tex/sheet-1.tex

@@ -0,0 +1,36 @@
+% Synthetic, in the shape of the corpus's problem sheets: a starred section
+% heading, each question a paragraph with its parts as a lettered list, and a
+% solutions section at the end. Nothing here is copied from ExampleContents.
+\documentclass[12pt]{article}
+\usepackage{amsmath}
+\usepackage{graphicx}
+
+\begin{document}
+
+\section*{Problem Sheet 1: Kinematics}
+
+A ball is thrown straight up at $20\,\mathrm{m/s}$.
+
+\begin{enumerate}
+\item Find the greatest height it reaches.
+\item Find its time of flight.
+\end{enumerate}
+
+A block of mass $m$ rests on a slope of angle $\theta$.
+
+\begin{enumerate}
+\item Name the three forces acting on the block.
+\item Find the least coefficient of friction that holds it still.
+\end{enumerate}
+
+\section*{Solutions}
+
+1(a) $h = v^2 / 2g = 20.4\,\mathrm{m}$
+
+1(b) $t = 2v/g = 4.08\,\mathrm{s}$
+
+2(a) Weight, the normal reaction, and friction along the slope.
+
+2(b) $\mu = \tan\theta$
+
+\end{document}

ci-corpus/tex/sheet-2-solutions.tex

@@ -0,0 +1,24 @@
+% Synthetic: the solutions to sheet-2.tex in a file of their own, which is the
+% shape that first failed a sweep. The name is what pairs the two: `pair` reads
+% the stem before the `-solutions` ending, finds sheet-2.tex beside it, and the
+% run freezes the questions first and these second. The spec reads every block
+% under the `Solutions` heading as a solution, so this file adds no question.
+\documentclass[12pt]{article}
+\usepackage{amsmath}
+\usepackage{graphicx}
+
+\begin{document}
+
+\section*{Problem Sheet 2: Answers}
+
+\subsection*{Solutions}
+
+1(a) Take the divergence term by term; each pair cancels.
+
+1(b) $B = \mu_0 m / 2\pi z^3$
+
+2(a) $r = mv / qB$
+
+2(b) $T = 2\pi m / qB$
+
+\end{document}