Replay each document's draft log in the corpus sweep

README.md

@@ -290,11 +290,19 @@ own, and is a row like any other document.
 
 The sweep never writes to the corpus. It copies each set's folder into `--work`
 (default `./.in2lambda-agent/corpus`), empties that copy first, and runs the documents
-there. It keeps the sets' specs in `--specs` (default `./corpus-specs`) in a tree
-mirroring the corpus: set `A/B` keeps its spec at
-`corpus-specs/A/B/in2lambda-spec.yaml`. So the copies are throwaway and the specs are
-worth keeping — `--replay` reruns the saved specs and nothing else, making no model
-call, which turns a document set into a deterministic test.
+there. It keeps three kinds of file in `--specs` (default `./corpus-specs`), in a tree
+mirroring the corpus: the set's spec, each document's log of the commands its fixing
+rounds ran, and the `in2lambda-agent-runs.jsonl` every run appends a line to. Set `A/B`
+keeps its spec at `corpus-specs/A/B/in2lambda-spec.yaml` and the log of `A/B/sheet.tex`
+at `corpus-specs/A/B/sheet.tex.commands.json`. A log entry holds the block ids, field
+keys and line ranges its command named, and the wording a `field replace` or a typed
+field spells out. It does not hold the draft's fields, which hold every field's
+captured text.
+
+So the copies are throwaway and the specs and the logs are worth keeping. `--replay`
+runs the set's spec, then the document's log, and nothing from the model, which turns
+a document set into a deterministic test: a document a fixing round repaired replays
+to the set the sweep built.
 
 `--results` (default `./results.csv`) holds one row per document, sorted by path, with
 these columns:

docs/how-it-works.md

@@ -32,15 +32,16 @@ review    not asked for (mode none)
 build     /home/me/sheets/out/set.zip
 ```
 
-There are eleven stage names: `pair`, `ocr`, `freeze`, `spec`, `coverage`, `validate`,
-`set`, `fix`, `render`, `review` and `build`. A run prints `pair` only for a solutions
-file with no questions file beside it. The spec loop prints `freeze`, `spec`,
-`coverage`, `validate` and `set` once per try, and a run prints `validate` once per
-check and `fix` once per fixing round, so those names repeat. Each line is printed as
-the run makes it, so a `--tries 3` run prints seven lines before its second model
-call: `ocr`, the five lines of the first try, and the `freeze` of the second. The run
-above made two of its three tries, because the second spec left no block unassigned
-and no error behind.
+There are twelve stage names: `pair`, `ocr`, `freeze`, `spec`, `coverage`, `replay`,
+`validate`, `set`, `fix`, `render`, `review` and `build`. A run prints `pair` only for
+a solutions file with no questions file beside it, and `replay` only when it was given
+a saved log to run, which is what a corpus replay is given. The spec loop prints
+`freeze`, `spec`, `coverage`, `validate` and `set` once per try, and a run prints
+`validate` once per check and `fix` once per fixing round, so those names repeat. Each
+line is printed as the run makes it, so a `--tries 3` run prints seven lines before its
+second model call: `ocr`, the five lines of the first try, and the `freeze` of the
+second. The run above made two of its three tries, because the second spec left no
+block unassigned and no error behind.
 
 | Stage | in2lambda function | What the stage writes |
 | --- | --- | --- |
@@ -49,6 +50,7 @@ and no error behind.
 | `freeze` | `in2lambda.source.add` | `SOURCE.draft.json`, beside the frozen source |
 | `spec` | `in2lambda.source.show`, for the model's prompt | `in2lambda-spec.yaml`, beside `SOURCE` or at `--spec` |
 | `coverage` | `in2lambda.draft.execute` with `in2lambda.draft.spec_command` | the layer 1 fields of the draft |
+| `replay` | `in2lambda.draft.execute` once per saved command | the fields and the log of the draft |
 | `set` | `in2lambda.source.add`, then `in2lambda.draft.execute` with `in2lambda.draft.spec_command` | the draft of the copy in `CACHE/second/`, and nothing beside the set's own sheets |
 | `validate` | `in2lambda.draft.report.validate` | the report inside the draft |
 | `fix` | `in2lambda.source.show`, then `in2lambda.draft.execute` once per command | the fields and the log of the draft |
@@ -135,6 +137,22 @@ stage prints `no fields` where the spec wrote none. `4 ignored` is the number of
 blocks the spec's `ignore` selector matched. The unassigned blocks are listed by id,
 and the stage prints `none unassigned` where every block reached a field.
 
+### `replay`
+
+The stage runs the commands an earlier run's fixing rounds ran, read from the file the
+run was given, in the order they were saved. It calls no model: the commands name the
+blocks, field keys and line ranges each one wrote, so the draft the checks then see is
+the draft the earlier run's rounds left. The message has one form:
+
+```
+5 commands from /home/me/corpus-specs/sheets/sheet.md.commands.json
+```
+
+in2lambda refusing one of the commands ends the run, and the refusal names which
+command it was: `command 3 of 5, question add: b7b is in a field already`. The corpus
+sweep writes these files and `corpus --replay` reads them; a `run` is given one through
+`pipeline.run`.
+
 ### `set`
 
 The stage runs the try's spec over another document of the set, so that a spec is
@@ -384,7 +402,7 @@ these 21, in this order:
 | --- | --- |
 | `source` | the document, relative to the corpus root |
 | `set` | the folder the document is in |
-| `outcome` | `built`, `build refused`, `faulted`, `skipped`, `no spec`, `no model`, `spec failed` and `fix failed` where the model call did not finish, `spec rejected`, `bad spec`, or `error: <exception>` |
+| `outcome` | `built`, `build refused`, `faulted`, `skipped`, `no spec`, `replay refused` where in2lambda would not run one of the saved commands, `no model`, `spec failed` and `fix failed` where the model call did not finish, `spec rejected`, `bad spec`, or `error: <exception>` |
 | `reason` | the build's refusal, the first error the checks still found, the warnings a build proceeded past, or what an exception said |
 | `spec` | `wrote`, `reused`, or `rewritten` where the spec rewrite ran |
 | `layout` | the coverage's layout |

in2lambda_agent/cli.py

@@ -206,7 +206,8 @@ def build_parser() -> argparse.ArgumentParser:
     sweep.add_argument(
         "--replay",
         action="store_true",
-        help="Run the saved specs and nothing else, making no model call.",
+        help="Run the saved specs and the documents' saved logs, making no "
+        "model call.",
     )
     sweep.add_argument(
         "--rounds",
@@ -237,7 +238,8 @@ def build_parser() -> argparse.ArgumentParser:
         "--specs",
         type=Path,
         default=corpus.DEFAULT_SPEC_DIR,
-        help="The tree the sets' specs are kept in, mirroring the corpus.",
+        help="The tree the sets' specs are kept in, mirroring the corpus, with "
+        "each document's log of its fixing rounds beside its set's spec.",
     )
     sweep.add_argument(
         "--cache",

in2lambda_agent/corpus.py

@@ -12,12 +12,15 @@
 Nothing here writes into the corpus. A run leaves a draft beside its source,
 and a spec and a record beside that, so each set's folder is copied into
 a work directory and run there, and the specs are kept in a tree of their own
-mirroring the corpus. The copy is thrown away and made again every run; the spec
-is what survives, and is what makes a document replayable — a saved spec plus
-its source rebuilt with no model call at all.
+mirroring the corpus. The work directory is deleted and written again every
+sweep. A sweep keeps in the spec tree the set's spec, each document's log of
+the commands its fixing rounds ran, and the record every run appends a line to.
+A replay runs the spec, then the document's log, and makes no model call, so a
+document a round repaired replays to the set the sweep built.
 """
 
 import csv
+import json
 import shutil
 import time
 from dataclasses import asdict, dataclass, fields
@@ -44,6 +47,13 @@
 """The mirror tree the sets' specs are kept in, which the work directory being
 wiped does not touch."""
 
+COMMANDS_SUFFIX = ".commands.json"
+"""What a document's saved log is named with, beside its set's spec: the
+commands the fixing rounds ran, each with the block ids, field keys and line
+ranges it named, and the wording a `field replace` or a typed field spells out.
+The draft's `fields`, which hold every field's captured text, stay in the work
+directory."""
+
 ROOT_SET = "_root"
 """What the corpus root's own documents are staged under. They are a set like
 any other, but the set's folder under the work directory would be the work
@@ -62,8 +72,10 @@ class Row:
             the checks still fault and no zip, `skipped` for a file that is not
             a document and for a solutions document with no questions document
             beside it, `no spec` for a replay with nothing saved to replay,
-            `no model`, `spec failed` and `fix failed` where a model call did
-            not finish, `spec rejected`, `bad spec`, or `error: <exception>`.
+            `replay refused` where in2lambda would not run one of the saved
+            commands, `no model`, `spec failed` and `fix failed` where a model
+            call did not finish, `spec rejected`, `bad spec`, or
+            `error: <exception>`.
         reason: What the run had to say for itself, in the words of whatever
             said it: the refusal, the first error the checks were still finding,
             or what the exception said. On a `built` row it holds the warnings
@@ -255,11 +267,13 @@ def run_one(
         name: What to call it in the table, relative to the corpus root.
         set_name: Its set, relative to the corpus root.
         spec: The set's spec, in the mirror tree: read if it is there, written
-            if it is not.
+            if it is not. The document's log is written beside it, named after
+            the document.
         settings: The environment the run has available.
         rounds: The round limit, ignored in a replay, which can run none.
         tries: How many specs the run may write before keeping the best.
-        replay: Run the saved spec and nothing else, making no model call.
+        replay: Run the saved spec and the document's saved log, making no
+            model call.
         cache: Where the OCR of each PDF is kept.
         backend: The backend to write a spec with, chosen from the settings if
             absent.
@@ -268,6 +282,9 @@ def run_one(
         The document's row.
     """
     existed = Path(spec).is_file()
+    # The log is one document's and the spec is the set's, so the log is named
+    # after the document, beside the spec.
+    commands = Path(spec).parent / f"{source.name}{COMMANDS_SUFFIX}"
     row = Row(source=name, set=set_name)
     started = time.monotonic()
     try:
@@ -276,6 +293,9 @@ def run_one(
             out_dir=source.parent / "out",
             settings=settings,
             spec=spec,
+            # The log of an earlier sweep's fixing rounds, which is what a
+            # replay has in place of the rounds. A sweep writes it below.
+            commands=commands if replay else None,
             review=row.review,
             # A replay has no model to run a round with, so it stops at the
             # report: the row then says what the saved spec left rather than
@@ -288,6 +308,9 @@ def run_one(
     except ModelUnavailable as error:
         row.outcome = "no model" if existed else "no spec"
         row.reason = _one_line(str(error))
+    except package.CommandRefused as error:
+        row.outcome = "replay refused"
+        row.reason = _one_line(str(error))
     except ModelError as error:
         # Which call did not finish, and what the provider said it stopped on.
         # A row reading `error: ResultError` says neither.
@@ -322,6 +345,14 @@ def run_one(
             row.blocks = result.coverage.blocks
             row.unassigned = len(result.coverage.unassigned)
         if result.draft is not None:
+            if not replay:
+                # What a later replay runs in place of the fixing rounds. A run
+                # that took no round writes an empty list, so that a replay
+                # reading no file there knows the document was never swept.
+                commands.write_text(
+                    json.dumps(package.fix_log(result.draft), indent=1),
+                    encoding="utf-8",
+                )
             # The counts are keyed by the column names they fill, and `fields`
             # is their total, since a round writes fields the spec run's own
             # count knows nothing about.
@@ -378,8 +409,10 @@ def sweep(
         suffixes: The file suffixes that are documents.
         results: Where to write the table.
         work: Where each set's folder is copied to be run.
-        specs: The tree the sets' specs are kept in, mirroring the corpus.
-        replay: Run the saved specs and nothing else, making no model call.
+        specs: The tree the sets' specs are kept in, mirroring the corpus, with
+            each document's log of its fixing rounds beside its set's spec.
+        replay: Run the saved specs and logs and nothing else, making no model
+            call.
         rounds: The round limit each run is given.
         tries: How many specs each run may write before keeping the best.
         cache: Where the OCR of each PDF is kept, so that a sweep pointed at a

in2lambda_agent/package.py

@@ -364,6 +364,50 @@ def command_log(draft: Path) -> list[dict[str, Any]]:
     return _frozen(draft)["log"]
 
 
+def fix_log(draft: Path) -> list[dict[str, Any]]:
+    """The commands the fixing rounds ran, without the spec run before them.
+
+    A replay runs the set's spec itself and then these, so the `spec run` entry
+    is left out: it names the spec file relative to the draft, and the replay's
+    draft is in the work directory rather than where the sweep's draft was.
+
+    Args:
+        draft: The draft file.
+
+    Returns:
+        One entry per command, each `{"command", "args", "by"}`, in the order
+        they ran. An entry's arguments are the block ids, field keys and line
+        ranges the command named, together with the wording a `field replace`
+        or a typed field spells out. The draft's `fields`, which hold every
+        field's captured text, are not here.
+    """
+    return [one for one in command_log(draft) if one["command"] != "spec run"]
+
+
+def replay(draft: Path, commands: list[dict[str, Any]]) -> int:
+    """Runs a saved list of commands over a draft, in the order they were saved.
+
+    Args:
+        draft: The draft file, with the set's spec already run over it.
+        commands: The entries `fix_log` wrote.
+
+    Returns:
+        How many commands ran.
+
+    Raises:
+        CommandRefused: in2lambda would not run one of them. The message names
+            which command it was, and the commands before it have been applied.
+    """
+    for index, entry in enumerate(commands, start=1):
+        try:
+            command(draft, entry["command"], entry["args"], entry.get("by", BY))
+        except CommandRefused as error:
+            raise CommandRefused(
+                f"command {index} of {len(commands)}, {entry['command']}: {error}"
+            ) from None
+    return len(commands)
+
+
 def questions(draft: Path) -> dict[str, QuestionInfo]:
     """The questions a draft holds, in the order they are numbered.
 

in2lambda_agent/pipeline.py

@@ -26,6 +26,7 @@
 reviewer was shown has been approved.
 """
 
+import json
 import random
 import re
 import shutil
@@ -135,6 +136,7 @@ def run(
     out_dir: Path,
     settings: Settings,
     spec: Optional[Path] = None,
+    commands: Optional[Path] = None,
     review: str = "none",
     rounds: int = 3,
     tries: int = 3,
@@ -154,6 +156,10 @@ def run(
         out_dir: Where in2lambda writes the set's JSON folder and zip.
         settings: The environment the run has available.
         spec: The set's spec file, when it is not the one beside the source.
+        commands: A file holding the commands an earlier run's fixing rounds
+            ran, as `package.fix_log` wrote them. They are run after the spec,
+            which is how a replay rebuilds a draft the rounds repaired without
+            calling a model. A path that is not a file runs the spec alone.
         review: One of REVIEW_MODES.
         rounds: The round limit, N in the design spec: how many model calls may
             answer what the checks found before the run stops without a zip.
@@ -186,6 +192,7 @@ def run(
             spec call or a fixing round.
         BadSpec: If what the model answers with is not a spec.
         SpecRejected: If in2lambda will not run the spec.
+        CommandRefused: If in2lambda will not run one of the saved commands.
         SourceError: If in2lambda cannot freeze or check the source.
     """
     # A relative --out means the directory the user ran from, whatever in2lambda
@@ -259,6 +266,12 @@ def run(
         result.add_stage("spec", f"reused {saved}")
         result.coverage = package.spec_run(draft, saved)
         result.add_stage("coverage", str(result.coverage))
+
+        if commands is not None and Path(commands).is_file():
+            saved_commands = json.loads(Path(commands).read_text(encoding="utf-8"))
+            ran = package.replay(draft, saved_commands)
+            result.add_stage("replay", f"{ran} commands from {commands}")
+
         report = package.validate(draft)
         if report.clean or rounds < 1:
             result.add_stage(