Finish the spec iteration: stream its lines and document the set stage

.gitignore

@@ -13,6 +13,9 @@ dist/
 corpus-specs/
 # in2lambda's KaTeX converter writes this into the working directory on import.
 log
+# The default `--cache`, which a run and the tests write into the directory
+# they are run from.
+.in2lambda-agent/
 # The gate's CI corpus: xelatex compiles this PDF from ci-corpus/tex/sheet-1.tex
 # in the workflow, so the repository holds the tex source and not the output.
 ci-corpus/pdf/*.pdf

README.md

@@ -56,12 +56,13 @@ poetry run in2lambda-agent run sheet.pdf
 In full:
 
 ```sh
-poetry run in2lambda-agent run SOURCE [--spec FILE] [--review none|sample|per-question] [--rounds N] [--sample N] [--cache DIR] [--fresh-ocr] [--out DIR]
+poetry run in2lambda-agent run SOURCE [--spec FILE] [--review none|sample|per-question] [--rounds N] [--tries N] [--sample N] [--cache DIR] [--fresh-ocr] [--out DIR]
 ```
 
 `SOURCE` is a PDF, markdown, tex or docx file. Mathpix converts a PDF first, and the
 agent keeps the markdown and the images under the PDF's hash in `--cache` (default
-`./.in2lambda-agent`), so a second run over the same PDF makes no Mathpix call.
+`./.in2lambda-agent`), so a second run over the same PDF makes no Mathpix call. The
+cache also holds the copy of the set's other sheet each spec is run over.
 `--fresh-ocr` converts the PDF again and restarts the run from the new markdown.
 `--out` defaults to `./out`, where in2lambda writes the set's JSON folder and its zip.
 
@@ -74,25 +75,46 @@ you name that file or the solutions one. A solutions file with no questions file
 beside it stops the run, which exits 1 saying `solutions without questions`.
 
 The run writes a spec — the YAML selectors naming which blocks of the source are
-questions, parts and solutions — in one model call, and saves it as
-`in2lambda-spec.yaml` beside `SOURCE`. A folder of sheets is one document set and
-shares one spec, so the second sheet in that folder runs with no model call. `--spec`
-keeps the set's spec elsewhere; the agent reads that file if it exists and writes it if
-it does not. A run appends a line to `in2lambda-agent-runs.jsonl` beside the spec. The
-line records the layout, the blocks and the fields of the spec run, the tokens and the
-seconds of the model calls, and the commands of each fixing round. A run that stops for
-a review appends its line at the last approval.
+questions, parts and solutions — and saves it as `in2lambda-spec.yaml` beside `SOURCE`.
+A folder of sheets is one document set and shares one spec, so the second sheet in that
+folder runs with no model call. `--spec` keeps the set's spec elsewhere; the agent reads
+that file if it exists and writes it if it does not.
+
+Every sheet of the set reuses that spec, so the run writes it in up to `--tries` model
+calls, three by default, and keeps the best of them. Each call after the first reads the
+spec before it, what running that spec covered, the errors the checks found, and the
+blocks that spec left in no field in another document of the folder. The run keeps the
+spec that left the fewest blocks unassigned and the fewest errors, and makes no further
+call once a spec leaves neither.
+
+Each spec is run over a copy of that other document, kept in `--cache`, so the run
+writes nothing beside the set's own sheets: the draft an earlier run left beside a
+sheet, and the fields and the commands in it, stay as that run wrote them. The sheet's
+own solutions file is not another document, since it is in this run's draft already. A
+PDF beside a PDF source is passed over, because converting it takes a Mathpix call. The
+record's `second` names the document the specs were run over, or names the one the run
+passed over and says why, and a `set` line says the same.
+
+A run appends a line to `in2lambda-agent-runs.jsonl` beside the spec. The line records
+the layout, the blocks and the fields of the spec run, the tokens and the seconds of the
+model calls, one `iterations` entry per spec the run wrote, and the commands of each
+fixing round. A run that stops for a review appends its line at the last approval.
 
 Each stage prints a line:
 
 ```
 ocr       fresh pass, restarting from /home/me/sheets/.in2lambda-agent/9f2c…/source.md
 freeze    /home/me/sheets/.in2lambda-agent/9f2c…/source.draft.json
-spec      wrote /home/me/sheets/in2lambda-spec.yaml via anthropic, 1883 tokens, 6.4s
-coverage  PartsSepSol: 14 blocks, 9 fields at layer 1, 4 ignored, b13 unassigned
-validate  b13 (lines 21-21) is in no field and not marked ignore.
-fix       round 1: 1 command (question solution q2), 2604 tokens, 4.1s
+spec      wrote /home/me/sheets/in2lambda-spec.yaml via anthropic, 1883 tokens, 6.4s (try 1 of 3)
+coverage  PartsSepSol: 14 blocks, 8 fields at layer 1, 4 ignored, b12, b13 unassigned
+validate  b12 (lines 19-19) is in no field and not marked ignore.; b13 (lines 21-21) is in no field and not marked ignore.
+set       sheet-2.md: PartsSepSol: 11 blocks, 7 fields at layer 1, 3 ignored, b9 unassigned
+freeze    /home/me/sheets/.in2lambda-agent/9f2c…/source.draft.json
+spec      wrote /home/me/sheets/in2lambda-spec.yaml via anthropic, 2410 tokens, 7.1s (try 2 of 3)
+coverage  PartsSepSol: 14 blocks, 10 fields at layer 1, 4 ignored, none unassigned
 validate  nothing to report
+set       sheet-2.md: PartsSepSol: 11 blocks, 8 fields at layer 1, 3 ignored, none unassigned
+spec      kept try 2 of 3
 review    not asked for (mode none)
 build     /home/me/sheets/out/set.zip
 ```
@@ -103,6 +125,7 @@ build     /home/me/sheets/out/set.zip
 | `freeze` | the draft in2lambda wrote from the source |
 | `spec` | the spec file, with the backend and the tokens where the model wrote it |
 | `coverage` | the layout, the blocks, the fields per layer, and the blocks in no field |
+| `set` | what the spec covered of the set's other document, or why none was run over |
 | `validate` | what the checks found, or `nothing to report` |
 | `fix` | the round's number, the commands the model ran, and the tokens |
 | `render` | the question PDFs a reviewer reads, or why there are none |
@@ -134,8 +157,9 @@ that tries to write one ends the run, and the last line names the field and what
 checks say about it, for a person or a later command to quote the source range into.
 Where the run
 reused a saved spec and the checks fault the draft, the run writes the spec again with
-the report in the prompt, if `--rounds` is 1 or more: a spec that covers the whole set
-repairs every sheet in it. That rewrite is not one of the rounds.
+what that spec covered in the prompt, if `--rounds` is 1 or more: a spec that covers the
+whole set repairs every sheet in it. That rewrite takes `--tries` calls like any other
+spec, and is not one of the rounds.
 
 ### Exit codes
 
@@ -251,7 +275,7 @@ poetry run in2lambda-agent corpus ExampleContents --suffix tex --suffix md
 In full:
 
 ```sh
-poetry run in2lambda-agent corpus ROOT [PATH ...] [--suffix S] [--replay] [--rounds N] [--results FILE] [--work DIR] [--specs DIR] [--cache DIR]
+poetry run in2lambda-agent corpus ROOT [PATH ...] [--suffix S] [--replay] [--rounds N] [--tries N] [--results FILE] [--work DIR] [--specs DIR] [--cache DIR]
 ```
 
 `ROOT` is the corpus directory and each `PATH` a folder under it to run, defaulting to

docs/how-it-works.md

@@ -6,8 +6,9 @@ writes and every message the stage prints. [README.md](../README.md) gives the
 commands and their options.
 
 The agent makes three kinds of model call and no others: one writes the set's spec,
-one rewrites a saved spec the checks fault, and one answers a validation report.
-in2lambda performs every other step and every write.
+one rewrites a saved spec the checks fault, and one answers a validation report. A run
+makes the spec call up to `--tries` times, three by default, and saves one of the specs
+it wrote. in2lambda performs every other step and every write.
 
 ## The stages
 
@@ -17,25 +18,36 @@ then the message:
 ```
 ocr       fresh pass, restarting from /home/me/sheets/.in2lambda-agent/9f2c…/source.md
 freeze    /home/me/sheets/.in2lambda-agent/9f2c…/source.draft.json
-spec      wrote /home/me/sheets/in2lambda-spec.yaml via anthropic, 1883 tokens, 6.4s
-coverage  PartsSepSol: 14 blocks, 9 fields at layer 1, 4 ignored, b13 unassigned
-validate  b13 (lines 21-21) is in no field and not marked ignore.
-fix       round 1: 1 command (question solution q2), 2604 tokens, 4.1s
+spec      wrote /home/me/sheets/in2lambda-spec.yaml via anthropic, 1883 tokens, 6.4s (try 1 of 3)
+coverage  PartsSepSol: 14 blocks, 8 fields at layer 1, 4 ignored, b12, b13 unassigned
+validate  b12 (lines 19-19) is in no field and not marked ignore.; b13 (lines 21-21) is in no field and not marked ignore.
+set       sheet-2.md: PartsSepSol: 11 blocks, 7 fields at layer 1, 3 ignored, b9 unassigned
+freeze    /home/me/sheets/.in2lambda-agent/9f2c…/source.draft.json
+spec      wrote /home/me/sheets/in2lambda-spec.yaml via anthropic, 2410 tokens, 7.1s (try 2 of 3)
+coverage  PartsSepSol: 14 blocks, 10 fields at layer 1, 4 ignored, none unassigned
 validate  nothing to report
+set       sheet-2.md: PartsSepSol: 11 blocks, 8 fields at layer 1, 3 ignored, none unassigned
+spec      kept try 2 of 3
 review    not asked for (mode none)
 build     /home/me/sheets/out/set.zip
 ```
 
-There are nine stage names: `ocr`, `freeze`, `spec`, `coverage`, `validate`, `fix`,
-`render`, `review` and `build`. A run prints `validate` once per check and `fix` once
-per fixing round, so those two names repeat.
+There are ten stage names: `ocr`, `freeze`, `spec`, `coverage`, `validate`, `set`,
+`fix`, `render`, `review` and `build`. The spec loop prints `freeze`, `spec`,
+`coverage`, `validate` and `set` once per try, and a run prints `validate` once per
+check and `fix` once per fixing round, so those names repeat. Each line is printed as
+the run makes it, so a `--tries 3` run prints seven lines before its second model
+call: `ocr`, the five lines of the first try, and the `freeze` of the second. The run
+above made two of its three tries, because the second spec left no block unassigned
+and no error behind.
 
 | Stage | in2lambda function | What the stage writes |
 | --- | --- | --- |
 | `ocr` | none: Mathpix converts the PDF | `CACHE/HASH/source.md` and `CACHE/HASH/media/` |
 | `freeze` | `in2lambda.source.add` | `SOURCE.draft.json`, beside the frozen source |
 | `spec` | `in2lambda.source.show`, for the model's prompt | `in2lambda-spec.yaml`, beside `SOURCE` or at `--spec` |
 | `coverage` | `in2lambda.draft.execute` with `in2lambda.draft.spec_command` | the layer 1 fields of the draft |
+| `set` | `in2lambda.source.add`, then `in2lambda.draft.execute` with `in2lambda.draft.spec_command` | the draft of the copy in `CACHE/second/`, and nothing beside the set's own sheets |
 | `validate` | `in2lambda.draft.report.validate` | the report inside the draft |
 | `fix` | `in2lambda.source.show`, then `in2lambda.draft.execute` once per command | the fields and the log of the draft |
 | `render` | `in2lambda.draft.export.render` | `OUT/render/question_000_Question_1.pdf`, one PDF per question |
@@ -75,15 +87,20 @@ run wrote.
 
 ### `spec`
 
-The stage prints one of two messages:
+The stage prints one of three messages:
 
 * `reused /home/me/sheets/in2lambda-spec.yaml` — the spec file exists, and the stage
   makes no model call. A run that reuses a spec the checks then fault prints this stage
   a second time in its `wrote` form.
-* `wrote /home/me/sheets/in2lambda-spec.yaml via anthropic, 1883 tokens, 6.4s` — the
-  model wrote the spec. The backend is `anthropic`, `openrouter` or `agent-sdk`. The
-  token count is the call's input and output tokens added together, and the time is
-  the wall time of the call to one decimal place.
+* `wrote /home/me/sheets/in2lambda-spec.yaml via anthropic, 1883 tokens, 6.4s (try 1 of
+  3)` — the model wrote the spec. The backend is `anthropic`, `openrouter` or
+  `agent-sdk`. The token count is the call's input and output tokens added together,
+  and the time is the wall time of the call to one decimal place. The try number counts
+  from 1 to `--tries`.
+* `kept try 2 of 3` — the loop wrote more than one spec, and this names the try saved
+  for the set: the one that left the fewest blocks unassigned and the fewest errors,
+  over this source and over the set's other document. The loop writes one spec where
+  the first leaves neither, and prints no `kept` line.
 
 in2lambda refuses a spec it cannot run, and the run raises `SpecRejected`. A spec this
 run wrote is deleted before that refusal reaches the user, and a spec this run wrote
@@ -105,6 +122,33 @@ stage prints `no fields` where the spec wrote none. `4 ignored` is the number of
 blocks the spec's `ignore` selector matched. The unassigned blocks are listed by id,
 and the stage prints `none unassigned` where every block reached a field.
 
+### `set`
+
+The stage runs the try's spec over another document of the set, so that a spec is
+judged on the set it is saved for rather than on this one sheet. The document is the
+first other document of the source's folder, by name, whose suffix is the source's. The
+run copies it under `CACHE/second/` and freezes that copy, so the draft an earlier run
+left beside that sheet stays as that run wrote it. The stage prints one of four
+messages:
+
+* `sheet-2.md: PartsSepSol: 11 blocks, 7 fields at layer 1, 3 ignored, b9 unassigned` —
+  the spec ran over the other document, in the `coverage` line's own form. The blocks
+  it left in no field there count toward the try's score, beside the blocks and the
+  errors this source left.
+* `sheet-2.md: in2lambda refused the spec: ERROR` — in2lambda ran the spec over this
+  source and refused it over the other document. The try scores as leaving every block
+  of that document in no field. The next try is run over the document again.
+* `sheet-2.md cannot be read: ERROR` — in2lambda refuses the document itself, a Word
+  lock file beside a docx among them. The run continues, judges its tries on this
+  source alone, and runs no later try over the document.
+* `sheet-2.pdf passed over: converting it takes an OCR call, and the spec loop makes no
+  call but the model's` — the run passed the document over before running a spec over
+  it. A PDF beside a PDF source is passed over for the reason the message gives, and a
+  document the run cannot copy is passed over saying so.
+
+A folder holding one sheet prints no `set` line. The sheet's own solutions file is no
+other document of the set: it is a second source of this run's own draft already.
+
 ### `validate`
 
 `in2lambda.draft.report.validate` checks the draft and writes its report into the
@@ -213,18 +257,25 @@ Each backend limits a call differently:
 
 | Call | What it is given | What it may write |
 | --- | --- | --- |
-| Spec | the spec system prompt, and the frozen source as `in2lambda.source.show` prints it | `in2lambda-spec.yaml`, and nothing else |
-| Spec rewrite | the same, with the errors of the last report appended to the prompt | `in2lambda-spec.yaml`, and nothing else |
+| Spec | the spec system prompt, the frozen source as `in2lambda.source.show` prints it, and, from the second call on, the spec before it, that spec's coverage line, the errors the report holds and the blocks that spec left in no field in the set's other document | `in2lambda-spec.yaml`, and nothing else |
+| Spec rewrite | the same, with the saved spec and what running it covered as the first call's try 0 | `in2lambda-spec.yaml`, and nothing else |
 | Fixing round | the fixing system prompt, the frozen source, every finding of the report, and a reviewer's note where there is one | the eight draft commands, and nothing else |
 
 The spec call has no tools. Its reply is the YAML of a spec, past a code fence where
 the model wrote one. The agent refuses a reply that is not YAML, a reply that is not a
 mapping, and a reply naming a `layout` outside `PartsSepSol`, `PartsOneSol`,
 `PartSolPartSol` and `PartPartSolSol`.
 
-The spec rewrite is the same call with the report's errors in the prompt. It runs once
-per run, before any fixing round, where the run reused a saved spec and the checks
-fault the draft. It writes layer 1 fields, and it is not one of the `--rounds`.
+A run makes the spec call up to `--tries` times, three by default. Each call after the
+first is asked for a spec that leaves fewer blocks unassigned and fewer errors behind
+than the one before it, over this source and over the set's other document. The run
+makes no further call once a spec leaves no block unassigned and no error behind, and
+saves the try that left the fewest of both.
+
+The spec rewrite is the same loop, with the saved spec and what running it covered as
+try 0. It runs where the run reused a saved spec and the checks fault the draft, before
+any fixing round, and takes `--tries` calls like any other spec. It writes layer 1
+fields, and it is not one of the `--rounds`.
 
 The fixing round's tools are the eight in2lambda draft commands: `mark ignore`,
 `question add`, `part add`, `question solution`, `part solution`, `field replace`,

in2lambda_agent/cli.py

@@ -41,6 +41,27 @@ def sample_count(given: str) -> int:
     return count
 
 
+def try_count(given: str) -> int:
+    """How many specs the agent may write, which is at least one.
+
+    Args:
+        given: What was typed after `--tries`.
+
+    Returns:
+        The count.
+
+    Raises:
+        ArgumentTypeError: it is below one. A run that may write no spec has
+            none to run, and a set with no saved spec has nothing to reuse.
+    """
+    count = int(given)
+    if count < 1:
+        raise argparse.ArgumentTypeError(
+            f"a run writes at least one spec, not {count}"
+        )
+    return count
+
+
 def reviewer_name(given: Optional[str]) -> str:
     """Who the draft's log records an edit as being by.
 
@@ -102,6 +123,12 @@ def build_parser() -> argparse.ArgumentParser:
         default=3,
         help="How many times the agent may try to fix validation errors.",
     )
+    run.add_argument(
+        "--tries",
+        type=try_count,
+        default=3,
+        help="How many specs the agent may write before keeping the best.",
+    )
     run.add_argument(
         "--sample",
         type=sample_count,
@@ -188,6 +215,12 @@ def build_parser() -> argparse.ArgumentParser:
         default=3,
         help="How many times the agent may try to fix validation errors.",
     )
+    sweep.add_argument(
+        "--tries",
+        type=try_count,
+        default=3,
+        help="How many specs the agent may write before keeping the best.",
+    )
     sweep.add_argument(
         "--results",
         type=Path,
@@ -297,6 +330,7 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
             specs=args.specs,
             replay=args.replay,
             rounds=args.rounds,
+            tries=args.tries,
             cache=args.cache,
             settings=load_settings(),
         )
@@ -390,6 +424,7 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
                 spec=args.spec,
                 review=args.review,
                 rounds=args.rounds,
+                tries=args.tries,
                 sample=args.sample,
                 cache_dir=args.cache,
                 fresh_ocr=args.fresh_ocr,