Skip to content

Run the corpus sweep through the two-route conversion #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
peterbjohnson merged 6 commits into from
Sep 23, 2026
Merged

Run the corpus sweep through the two-route conversion #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fa29ba1
implement: Run the corpus sweep through the two-route conversion (t39)
peterbjohnson Sep 23, 2026
54ce102
Merge commit 'bc9be9c339cf96e1f80a6f170cda4f2470c9cb98' into wb/t39
peterbjohnson Sep 23, 2026
aaeb97f
implement: Run the corpus sweep through the two-route conversion (t39)
peterbjohnson Sep 23, 2026
a7ce219
verify: Run the corpus sweep through the two-route conversion (t39)
peterbjohnson Sep 23, 2026
25aa4be
implement: Run the corpus sweep through the two-route conversion (t39)
peterbjohnson Sep 23, 2026
fd93ea5
implement: Run the corpus sweep through the two-route conversion (t39)
peterbjohnson Sep 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 35 additions & 36 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,8 +255,10 @@ spec in2lambda refuses, a review command naming a question that is not under rev
and a draft command in2lambda refuses each print `in2lambda-agent: MESSAGE` on stderr
and exit 1.

`corpus` exits 0 where it ran at least one document and every row is `built` or
`skipped`. Any other outcome on any row exits 1.
`corpus` exits 0 where it ran at least one sheet and every sheet built a set. It exits 1
where it found no sheet, and where any sheet built no set — a row whose reason begins
`no set:`. A sheet route B failed on built its set from route A and does not change the
exit code.

`compare` exits 1 where Mathpix, the model or `pdftoppm` failed, and 0 otherwise. A
difference it reports does not change the exit code.
Expand Down Expand Up @@ -352,8 +354,8 @@ lists the differences `compare` found over three corpus documents and judges eac

## Corpus

The design spec's test plan is the agent run over a corpus of real documents, with one
row of a table recorded for each document:
The design spec's test plan is the two-route conversion run over a corpus of real
documents, with one row of a table recorded for each sheet:

```sh
poetry run in2lambda-agent corpus ExampleContents --suffix tex --suffix md
Expand All @@ -362,52 +364,48 @@ poetry run in2lambda-agent corpus ExampleContents --suffix tex --suffix md
In full:

```sh
poetry run in2lambda-agent corpus ROOT [PATH ...] [--suffix S] [--replay] [--rounds N] [--tries N] [--results FILE] [--work DIR] [--specs DIR] [--cache DIR]
poetry run in2lambda-agent corpus ROOT [PATH ...] [--suffix S] [--results FILE] [--work DIR] [--cache DIR]
```

`ROOT` is the corpus directory and each `PATH` a folder under it to run, defaulting to
all of it. `--suffix` is repeatable and defaults to `tex`, `md` and `docx`; `--suffix
pdf` runs the PDFs too, which needs Mathpix credentials and one call per PDF. Every run
is review mode `none`.
pdf` runs the PDFs too, which needs Mathpix credentials and one call per PDF.

A sheet and the solutions file beside it are one run and one row, named after the
questions file. A solutions file with no questions file beside it is converted on its
own, and is a row like any other document.
A set is a folder holding at least one questions document. Each set converts as a
folder run of `convert` does: one model call writes the set's filter from the first
sheet of it pandoc can read, and each sheet of the set then runs through route A and
that filter. A sheet and the solutions file beside it are one conversion and one row,
named after the questions file. A folder of figures, a tex drawing with no
`\begin{document}` among them, and a folder holding a solutions file alone, are not
sets and have no row.

The sweep never writes to the corpus. It copies each set's folder into `--work`
(default `./.in2lambda-agent/corpus`), empties that copy first, and runs the documents
there. The copy holds everything under the folder — the figures a sheet names among it —
less what an earlier run left there: a spec, a draft, an `.in2lambda-agent` directory.
It keeps three kinds of file in `--specs` (default `./corpus-specs`), in a tree
mirroring the corpus: the set's spec, each document's log of the commands its fixing
rounds ran, and the `in2lambda-agent-runs.jsonl` every run appends a line to. Set `A/B`
keeps its spec at `corpus-specs/A/B/in2lambda-spec.yaml` and the log of `A/B/sheet.tex`
at `corpus-specs/A/B/sheet.tex.commands.json`. A log entry holds the block ids, field
keys and line ranges its command named, and the wording a `field replace` or a typed
field spells out. It does not hold the draft's fields, which hold every field's
captured text.
The sweep never writes to the corpus. It writes each set's filter to
`WORK/SET/filter.lua` and each sheet's set folder and zip to `WORK/SET/SHEET/`, where
`--work` defaults to `./.in2lambda-agent/corpus`.

So the copies are throwaway and the specs and the logs are worth keeping. `--replay`
runs the set's spec, then the document's log, and nothing from the model, which turns
a document set into a deterministic test: a document a fixing round repaired replays
to the set the sweep built.

`--results` (default `./results.csv`) holds one row per document, sorted by path, with
these columns:
`--results` (default `./results.csv`) holds one row per sheet, in path order, with these
twelve columns:

```
source, set, outcome, reason, spec, layout, blocks, fields, layer1..layer4, edited,
unassigned, rounds, input_tokens, output_tokens, model_seconds, wall_seconds,
review, rejections
set, sheet, questions, parts, fields, agreed, adjudicated, flagged, not_verbatim,
tokens, seconds, reason
```

[docs/how-it-works.md](docs/how-it-works.md#the-corpus-table) names each column and
where its value comes from. One document that fails is one row and not the end of the
sweep, and a set whose folder cannot be copied is a row for each of its documents.
where its value comes from. `reason` is empty where the sheet ran through both routes
and built its set. One sheet whose conversion raises is one row, with `no set:` and the
error as its reason, and the sheets after it still run. A set with no filter — its
filter call did not finish, or every sheet of it is a PDF, which pandoc cannot read —
converts every sheet through route A alone, and each of those rows reads `no filter:`
and why. A PDF among sheets pandoc reads leaves the rest of the set on both routes: the
filter is written from one of those sheets, and the PDF alone fails route B. The command
exits 1 where a sheet built no set.

`--cache` is where the OCR of each PDF is kept. It defaults to `./.in2lambda-agent`,
the directory `run` caches into, so a sweep over PDFs that `run` has already converted
makes no Mathpix call and needs no Mathpix credentials.
makes no Mathpix call. It still reads `MATHPIX_APP_ID` and `MATHPIX_API_KEY`: `convert`
builds the Mathpix client before it asks the cache, and refuses a PDF where either
variable is unset, whether or not the cache holds that PDF.

## Targets

Expand Down Expand Up @@ -493,7 +491,8 @@ with what the baseline records:
poetry run in2lambda-agent gate BASELINE [--record] [--cache DIR] [--work DIR]
```

Every run is `corpus --replay`, so no model call is made. The command prints one line
Every run replays the spec route's saved specs and command logs, so no model call is
made. The command prints one line
per folder, and exits 1 when a folder builds fewer documents than the baseline records
or when a single document does worse than the baseline records it doing. The second
check is what a baseline of no builds rests on: a corpus where every document faults
Expand Down
55 changes: 23 additions & 32 deletions docs/how-it-works.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,9 +163,9 @@ the draft the earlier run's rounds left. The message has one form:
```

in2lambda refusing one of the commands ends the run, and the refusal names which
command it was: `command 3 of 5, question add: b7b is in a field already`. The corpus
sweep writes these files and `corpus --replay` reads them; a `run` is given one through
`pipeline.run`.
command it was: `command 3 of 5, question add: b7b is in a field already`. The spec
route's sweep, `corpus.sweep`, writes these files, and the gate's replay reads them; a
`run` is given one through `pipeline.run`.

### `set`

Expand Down Expand Up @@ -353,9 +353,8 @@ makes.

`field replace` writes no layer. in2lambda leaves the field at the layer that wrote it,
leaves it quoting the lines it was copied from, and sets the field's `edited` flag. So a
round that replaces a layer 1 field records `layer1=1, layer4=0, edited=1`, and the
corpus table's `layer1` to `layer4` columns count each field under the layer in2lambda
recorded. `package.questions` is the one reader that counts an edited field as layer 4,
round that replaces a layer 1 field records `layer1=1, layer4=0, edited=1`.
`package.questions` is the one reader that counts an edited field as layer 4,
and it does so to sort the questions a `sample` review shows.

## Review modes
Expand Down Expand Up @@ -412,35 +411,27 @@ run from another directory.

## The corpus table

`in2lambda-agent corpus` writes one row per document to `--results`. The columns are
these 21, in this order:
`in2lambda-agent corpus` writes one row per sheet to `--results`. The columns are these
12, in this order:

| Column | Where it comes from |
| --- | --- |
| `source` | the document, relative to the corpus root |
| `set` | the folder the document is in |
| `outcome` | `built`, `build refused`, `faulted`, `skipped`, `no spec`, `replay refused` where in2lambda would not run one of the saved commands, `no model`, `spec failed` and `fix failed` where the model call did not finish, `spec rejected`, `bad spec`, or `error: <exception>` |
| `reason` | the build's refusal, the first error the checks still found, the warnings a build proceeded past, or what an exception said |
| `spec` | `wrote`, `reused`, or `rewritten` where the spec rewrite ran |
| `layout` | the coverage's layout |
| `blocks` | the coverage's block count, before any fixing round |
| `fields` | how many fields the finished draft holds |
| `layer1` | `package.layers`: how many fields the spec wrote |
| `layer2` | `package.layers`: how many a predicate wrote, so 0, because the agent's spec prompt asks for no predicate |
| `layer3` | `package.layers`: how many a round quoted out of the source |
| `layer4` | `package.layers`: how many a round typed out |
| `edited` | `package.layers`: how many fields carry the `edited` flag |
| `unassigned` | how many blocks the coverage left in no field, before any round |
| `rounds` | how many fixing rounds ran |
| `input_tokens` | what the run's model calls read |
| `output_tokens` | what they wrote |
| `model_seconds` | how long they took |
| `wall_seconds` | how long the whole document took |
| `review` | the review mode the run was given, which the sweep sets to `none` |
| `rejections` | how many questions a reviewer turned down, so 0 under mode `none` |

`blocks` and `unassigned` report the spec run alone, so a `built` row can still name
blocks the spec left unassigned and a later round covered.
| `set` | the sheet's folder, relative to the corpus root |
| `sheet` | the questions document, relative to the corpus root; the solutions document beside it is read into the same row |
| `questions` | how many questions the conversion returned |
| `parts` | how many parts those questions hold |
| `fields` | `routes.fields`: how many text fields the questions and parts hold, the options of a multiple-choice part among them |
| `agreed` | how many fields the two routes returned the same text for, after `routes.fold` folds the whitespace and the notation that renders the same |
| `adjudicated` | how many fields the adjudication call decided |
| `flagged` | how many fields a person is asked to read |
| `not_verbatim` | how many of the flagged fields are not quotes of the source |
| `tokens` | what the sheet's model calls read and wrote, the direct call and the adjudication call; the set's filter call is counted on the set's first sheet |
| `seconds` | how long the sheet took, the filter call included on the set's first sheet |
| `reason` | empty where both routes ran and the sheet built its set; `no set: <error>` where the conversion raised, `route B failed: <pandoc's message>` where the set is route A's alone, and `no filter: <why>` where no sheet of the set ran route B, because its filter call did not finish or because pandoc reads no sheet of it |

A field only one route filled is neither agreed nor adjudicated: the count of those is
`fields - agreed - adjudicated`. `fields` counts what the conversion returned, so a
sheet route B did not run on still reports its fields.

## Backends and settings

Expand Down
72 changes: 21 additions & 51 deletions in2lambda_agent/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from pathlib import Path
from typing import Optional, Sequence

from in2lambda_agent import compare, corpus, gate, pair, pipeline, routes, targets
from in2lambda_agent import compare, gate, pair, pipeline, routes, sweep, targets
from in2lambda_agent.mathpix import MathpixClient, MathpixError
from in2lambda_agent.model import ModelError, ModelUnavailable, choose_backend
from in2lambda_agent.ocr import ocr_pdf
Expand Down Expand Up @@ -247,63 +247,38 @@ def build_parser() -> argparse.ArgumentParser:
help="Where the run left the review.",
)

sweep = subcommands.add_parser(
"corpus", help="Run every document of a corpus and record what each did."
corpus_command = subcommands.add_parser(
"corpus", help="Convert every set of a corpus and record what each sheet did."
)
sweep.add_argument("root", type=Path, help="The corpus directory.")
sweep.add_argument(
corpus_command.add_argument("root", type=Path, help="The corpus directory.")
corpus_command.add_argument(
"paths",
nargs="*",
type=Path,
help="Folders under ROOT to run, defaulting to all of it.",
)
sweep.add_argument(
corpus_command.add_argument(
"--suffix",
action="append",
dest="suffixes",
metavar="SUFFIX",
help="A file suffix to run, repeatable. Default: "
f"{', '.join(corpus.DEFAULT_SUFFIXES)}.",
f"{', '.join(sweep.DEFAULT_SUFFIXES)}.",
)
sweep.add_argument(
"--replay",
action="store_true",
help="Run the saved specs and the documents' saved logs, making no "
"model call.",
)
sweep.add_argument(
"--rounds",
type=int,
default=3,
help="How many times the agent may try to fix validation errors.",
)
sweep.add_argument(
"--tries",
type=try_count,
default=3,
help="How many specs the agent may write before keeping the best.",
)
sweep.add_argument(
corpus_command.add_argument(
"--results",
type=Path,
default=corpus.DEFAULT_RESULTS,
help="Where to write the table, one row per document.",
default=sweep.DEFAULT_RESULTS,
help="Where to write the table, one row per sheet.",
)
sweep.add_argument(
corpus_command.add_argument(
"--work",
type=Path,
default=corpus.DEFAULT_WORK_DIR,
help="Where each set's folder is copied to be run; the corpus itself "
"is never written to.",
)
sweep.add_argument(
"--specs",
type=Path,
default=corpus.DEFAULT_SPEC_DIR,
help="The tree the sets' specs are kept in, mirroring the corpus, with "
"each document's log of its fixing rounds beside its set's spec.",
default=sweep.DEFAULT_WORK_DIR,
help="Where each set's filter and each sheet's zip are written; the "
"corpus itself is never written to.",
)
sweep.add_argument(
corpus_command.add_argument(
"--cache",
type=Path,
default=pipeline.DEFAULT_CACHE_DIR,
Expand Down Expand Up @@ -543,26 +518,21 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
return convert_command(args)

if args.command == "corpus":
rows = corpus.sweep(
rows = sweep.sweep(
args.root,
paths=args.paths,
# Appended to, so the default cannot be the parser's: that would be
# the default and whatever was named.
suffixes=args.suffixes or corpus.DEFAULT_SUFFIXES,
suffixes=args.suffixes or sweep.DEFAULT_SUFFIXES,
results=args.results,
work=args.work,
specs=args.specs,
replay=args.replay,
rounds=args.rounds,
tries=args.tries,
cache=args.cache,
settings=load_settings(),
)
print(f"{len(rows)} documents, written to {args.results}")
# A file that is not a document is not a document that failed, so a
# figure's tex source among the rows does not make the sweep one.
succeeded = {"built", "skipped"}
return 0 if rows and all(row.outcome in succeeded for row in rows) else 1
print(f"{len(rows)} sheets, written to {args.results}")
# A sheet route B failed on built its set from route A, so the sheets
# that built no set are what the exit code reports.
return 0 if rows and all(row.built for row in rows) else 1

if args.command == "targets":
results = targets.run(
Expand Down
4 changes: 4 additions & 0 deletions in2lambda_agent/corpus.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
"""The test plan's sweep: the agent over a corpus, one row per document.

This is the spec route's sweep, and `gate.run` is its one caller: the gate
replays the saved specs, which the two-route conversion has nothing to replay.
The `corpus` command runs the two-route sweep of `sweep.py`.

The design spec asks, per document, what share of the fields each layer wrote,
how many rounds the checks took to come clean or that they never did, what the
model calls cost in tokens and time, and whether the set's spec was reused. That
Expand Down
Loading
Loading