Gate every merge on a replay sweep over the corpus

.github/branch-protection.json

@@ -0,0 +1,9 @@
+{
+  "required_status_checks": {
+    "strict": false,
+    "contexts": ["gate"]
+  },
+  "enforce_admins": false,
+  "required_pull_request_reviews": null,
+  "restrictions": null
+}

.github/workflows/gate.yml

@@ -0,0 +1,102 @@
+# The merge gate. Every push to main and every pull request runs the tests and
+# then replays the three folders of ci-corpus, the committed corpus, which
+# ci-baseline.json names. gate-baseline.json names the folders of the private
+# corpus, which a runner cannot read, and is replayed by the workbench check.
+# The job is a required status check on main, so a branch that breaks a build
+# cannot be merged.
+name: gate
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  gate:
+    name: gate
+    # Pinned rather than ubuntu-latest: the runner's TeX Live decides what
+    # xelatex writes, and the PDF's bytes are the OCR cache's key.
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+
+      # The Dockerfile's package list, which is the one record of what the PDF
+      # generator's template.latex loads.
+      - name: Install pandoc, poppler and TeX Live
+        run: |
+          sudo apt-get update
+          sudo apt-get install --no-install-recommends -y \
+            pandoc \
+            poppler-utils \
+            texlive-xetex \
+            texlive-latex-recommended \
+            texlive-latex-extra \
+            texlive-science \
+            texlive-lang-chinese \
+            texlive-lang-arabic \
+            texlive-bibtex-extra \
+            texlive-plain-generic \
+            texlive-fonts-recommended \
+            lmodern \
+            fonts-noto-core \
+            fonts-noto-cjk
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install poetry
+        run: pipx install poetry
+
+      - name: Install the package
+        run: poetry install --with dev
+
+      # SOURCE_DATE_EPOCH fixes the timestamp xelatex writes into the PDF, so
+      # the same tex source compiles to the same bytes and the OCR cache key
+      # does not change between runs.
+      - name: Compile the CI corpus PDF
+        run: |
+          cd ci-corpus/tex
+          SOURCE_DATE_EPOCH=0 FORCE_SOURCE_DATE=1 \
+            xelatex -interaction=nonstopmode -output-directory=../pdf sheet-1.tex
+          rm -f ../pdf/sheet-1.aux ../pdf/sheet-1.log
+
+      # Keyed by the PDF's bytes, which is also how ocr_pdf names its entry
+      # inside the directory. A restored entry for another PDF is unused.
+      - name: Restore the OCR cache
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/in2lambda-agent
+          key: ocr-${{ hashFiles('ci-corpus/pdf/*.pdf') }}
+          restore-keys: ocr-
+
+      - name: Tests
+        run: poetry run pytest -q
+
+      - name: Gate
+        env:
+          MATHPIX_APP_ID: ${{ secrets.MATHPIX_APP_ID }}
+          MATHPIX_API_KEY: ${{ secrets.MATHPIX_API_KEY }}
+        run: poetry run in2lambda-agent gate ci-baseline.json
+
+      # What this run would record, so that a baseline change is committed from
+      # what CI saw. Written whether or not the gate passed.
+      - name: Record what this run did
+        if: always()
+        env:
+          MATHPIX_APP_ID: ${{ secrets.MATHPIX_APP_ID }}
+          MATHPIX_API_KEY: ${{ secrets.MATHPIX_API_KEY }}
+        run: |
+          cp ci-baseline.json "${RUNNER_TEMP}/ci-baseline.json"
+          poetry run in2lambda-agent gate --record "${RUNNER_TEMP}/ci-baseline.json"
+
+      - name: Upload it
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: recorded-baseline
+          path: ${{ runner.temp }}/ci-baseline.json

.gitignore

@@ -7,3 +7,6 @@ __pycache__/
 dist/
 # in2lambda's KaTeX converter writes this into the working directory on import.
 log
+# The gate's CI corpus: xelatex compiles this PDF from ci-corpus/tex/sheet-1.tex
+# in the workflow, so the repository holds the tex source and not the output.
+ci-corpus/pdf/*.pdf

README.md

@@ -252,7 +252,7 @@ poetry run in2lambda-agent corpus ExampleContents --suffix tex --suffix md
 In full:
 
 ```sh
-poetry run in2lambda-agent corpus ROOT [PATH ...] [--suffix S] [--replay] [--rounds N] [--results FILE] [--work DIR] [--specs DIR]
+poetry run in2lambda-agent corpus ROOT [PATH ...] [--suffix S] [--replay] [--rounds N] [--results FILE] [--work DIR] [--specs DIR] [--cache DIR]
 ```
 
 `ROOT` is the corpus directory and each `PATH` a folder under it to run, defaulting to
@@ -285,6 +285,82 @@ review, rejections
 where its value comes from. One document that fails is one row and not the end of the
 sweep, and a set whose folder cannot be copied is a row for each of its documents.
 
+`--cache` (default `./.in2lambda-agent`) is where the OCR of each PDF is kept. A sweep
+pointed at a cache that an earlier run filled makes no Mathpix call, and needs no
+Mathpix credentials.
+
+## Gate
+
+Nothing merges without a replay over real documents. `gate` reruns the saved specs
+over the folders a baseline file names, and compares what each folder did this run
+with what the baseline records:
+
+```sh
+poetry run in2lambda-agent gate BASELINE [--record] [--cache DIR] [--work DIR]
+```
+
+Every run is `corpus --replay`, so no model call is made. The command prints one line
+per folder, and exits 1 when a folder builds fewer documents than the baseline records
+or when a single document does worse than the baseline records it doing. The second
+check is what a baseline of no builds rests on: a corpus where every document faults
+still reports the document that stops being read.
+
+```
+work      /tmp/in2lambda-agent-gate-3f1a
+UCL_MechEng          built 0  faulted 0  build refused 0  skipped 1  no spec 2   (baseline built 0)
+  worse  UCL_MechEng/Worksheet_2.pdf  faulted -> no spec: replay: no model call is allowed
+```
+
+Run the command from the repository root. The gate reads `specs`, and a folder's `root`
+where `root` is relative, from the directory the command runs in.
+
+The repository holds two baselines, because a clone holds the second corpus and not the
+first:
+
+| File | Corpus | Run by |
+| --- | --- | --- |
+| `gate-baseline.json` | the three folders of `ExampleContents`, which is private | the workbench check |
+| `ci-baseline.json` | the three folders of `ci-corpus`, which is committed | `.github/workflows/gate.yml` |
+
+Each folder's `root` and `suffixes` are written by hand. `built`, the count of documents
+that built, and `documents`, the outcome of each single document, are what `--record`
+writes. A folder the file records no `built` for passes on any count, and its line reads
+`(not recorded)`. A change to a recorded count or outcome belongs in a pull request that
+says why the count or the outcome changed.
+
+`gate-baseline.json` records 0 built for all three folders. Every document of
+`ExampleContents` replays to `faulted`, because pandoc's line wrapping is reported as a
+math delimiter error and each document needs a fixing round that a replay does not run.
+The recorded outcomes are what the gate defends until a later ticket raises the count.
+
+`--cache` defaults to `~/.cache/in2lambda-agent`, outside any worktree, so that a PDF
+converted on one branch is not converted again on the next. `--work` defaults to a new
+directory under the system temp directory, which the gate does not delete: read the
+drafts of a folder that failed there. The gate also copies the spec tree into the work
+directory and replays the copy, because a sweep appends a record of each run beside the
+spec it reads. Neither directory is inside the repository, so `git status` after a gate
+run reports no new file.
+
+`.github/workflows/gate.yml` runs pytest and then `gate ci-baseline.json` on every push
+to `main` and every pull request. Mathpix reads `ci-corpus/pdf/sheet-1.pdf` once and the
+workflow stores the markdown in the Actions cache under the PDF's hash. The job needs
+two repository secrets, `MATHPIX_APP_ID` and `MATHPIX_API_KEY`.
+
+A pull request from a fork is given neither secret. `actions/cache` restores the cache
+of the base branch for a fork, and the cached markdown is what `ci-corpus/pdf` then
+replays, so the job passes without the secrets. If the cache is empty — the PDF's bytes
+changed, or GitHub evicted the entry — Mathpix cannot be called, `ci-corpus/pdf` builds
+0 against a recorded 1, the job fails and the pull request cannot be merged. A
+maintainer merges that branch by pushing it to a branch of this repository, where the
+secrets are read.
+
+The `gate` job is a required status check on `main`:
+
+```sh
+gh api -X PUT repos/{owner}/{repo}/branches/main/protection \
+  --input .github/branch-protection.json
+```
+
 ## Docker
 
 The image carries pandoc, a TeX Live whose xelatex runs the PDF generator's

ci-baseline.json

@@ -0,0 +1,38 @@
+{
+  "specs": "corpus-specs",
+  "folders": {
+    "ci-corpus/tex": {
+      "root": ".",
+      "suffixes": [
+        "tex"
+      ],
+      "built": 4,
+      "documents": {
+        "ci-corpus/tex/sheet-1.tex": "built",
+        "ci-corpus/tex/sheet-2.tex": "built",
+        "ci-corpus/tex/sheet-3.tex": "built",
+        "ci-corpus/tex/solutions-2.tex": "built"
+      }
+    },
+    "ci-corpus/docx": {
+      "root": ".",
+      "suffixes": [
+        "docx"
+      ],
+      "built": 1,
+      "documents": {
+        "ci-corpus/docx/sheet.docx": "built"
+      }
+    },
+    "ci-corpus/pdf": {
+      "root": ".",
+      "suffixes": [
+        "pdf"
+      ],
+      "built": 1,
+      "documents": {
+        "ci-corpus/pdf/sheet-1.pdf": "built"
+      }
+    }
+  }
+}

ci-corpus/tex/sheet-1.tex

@@ -0,0 +1,46 @@
+% Synthetic, in the shape of the corpus's problem sheets: a starred section
+% heading, an enumerate of questions with lettered parts, and a solutions
+% section at the end. Nothing here is copied from ExampleContents.
+\documentclass[12pt]{article}
+\usepackage{amsmath}
+\usepackage{graphicx}
+
+\begin{document}
+
+\section*{Problem Sheet 1: Kinematics}
+
+\begin{enumerate}
+
+\item A ball is thrown straight up at $20\,\mathrm{m/s}$.
+  \begin{enumerate}
+  \item Find the greatest height it reaches.
+  \item Find its time of flight.
+  \end{enumerate}
+
+\item A block of mass $m$ rests on a slope of angle $\theta$.
+  \begin{enumerate}
+  \item Name the three forces acting on the block.
+  \item Find the least coefficient of friction that holds it still.
+  \end{enumerate}
+
+\end{enumerate}
+
+\section*{Solutions}
+
+\begin{enumerate}
+
+\item
+  \begin{enumerate}
+  \item $h = v^2 / 2g = 20.4\,\mathrm{m}$
+  \item $t = 2v/g = 4.08\,\mathrm{s}$
+  \end{enumerate}
+
+\item
+  \begin{enumerate}
+  \item Weight, the normal reaction, and friction along the slope.
+  \item $\mu = \tan\theta$
+  \end{enumerate}
+
+\end{enumerate}
+
+\end{document}

ci-corpus/tex/sheet-2.tex

@@ -0,0 +1,28 @@
+% Synthetic: questions with lettered parts and no solutions on the sheet. The
+% solutions are in solutions-2.tex, which is a document of its own, as the
+% corpus's problem sheets are. Nothing here is copied from ExampleContents.
+\documentclass[12pt]{article}
+\usepackage{amsmath}
+\usepackage{graphicx}
+
+\begin{document}
+
+\section*{Problem Sheet 2: Fields}
+
+\begin{enumerate}
+
+\item A dipole sits at the origin.
+  \begin{enumerate}
+  \item Show that $\nabla \cdot \mathbf{B} = 0$ for its field.
+  \item Find the field on the axis at a distance $z$.
+  \end{enumerate}
+
+\item A charge $q$ moves at speed $v$ through a uniform field $\mathbf{B}$.
+  \begin{enumerate}
+  \item Find the radius of its circular path.
+  \item Find the period of the motion.
+  \end{enumerate}
+
+\end{enumerate}
+
+\end{document}

ci-corpus/tex/sheet-3.tex

@@ -0,0 +1,50 @@
+% Synthetic: a sheet whose first question includes a figure, which is the shape
+% that first failed a sweep on the path of an image. Nothing here is copied
+% from ExampleContents.
+\documentclass[12pt]{article}
+\usepackage{amsmath}
+\usepackage{graphicx}
+
+\begin{document}
+
+\section*{Problem Sheet 3: Statics}
+
+\begin{enumerate}
+
+\item The beam below carries a load $W$ at its midpoint.
+
+  \includegraphics[width=0.2\textwidth]{figures/ball.png}
+
+  \begin{enumerate}
+  \item Find the reaction at each support.
+  \item Find the bending moment at the midpoint.
+  \end{enumerate}
+
+\item A ladder of mass $m$ leans against a smooth wall.
+  \begin{enumerate}
+  \item Draw the forces acting on the ladder.
+  \item Find the least angle at which it does not slip.
+  \end{enumerate}
+
+\end{enumerate}
+
+\section*{Solutions}
+
+\begin{enumerate}
+
+\item
+  \begin{enumerate}
+  \item $W/2$ at each support.
+  \item $M = WL/4$
+  \end{enumerate}
+
+\item
+  \begin{enumerate}
+  \item The weight at the centre, the normal reaction at the wall, and the
+    normal reaction and friction at the floor.
+  \item $\tan\alpha = 1/2\mu$
+  \end{enumerate}
+
+\end{enumerate}
+
+\end{document}