Compare each target document with the set Lambda Feedback exported from it

.gitignore

@@ -11,6 +11,9 @@ dist/
 # gate-baseline.json beside them records those documents' file names and the
 # absolute path of the corpus on one machine.
 corpus-specs/
+# The default `--filters`: a target's filter is written from a private
+# document's structure, and its reply.json holds that document's text.
+targets/
 # in2lambda's KaTeX converter writes this into the working directory on import.
 log
 # The default `--cache`, which a run and the tests write into the directory

README.md

@@ -409,6 +409,80 @@ sweep, and a set whose folder cannot be copied is a row for each of its document
 the directory `run` caches into, so a sweep over PDFs that `run` has already converted
 makes no Mathpix call and needs no Mathpix credentials.
 
+## Targets
+
+A target is a folder holding one set: a questions document, a solutions document where
+the set has one, and the folder Lambda Feedback exported for that set, named
+`set_<Name>`. The export is what the conversion is trying to reproduce, so a target is
+the one place the agent can be told right from wrong rather than merely flagged:
+
+```sh
+poetry run in2lambda-agent targets ExampleContents/targets
+```
+
+In full:
+
+```sh
+poetry run in2lambda-agent targets ROOT [PATH ...] [--filters DIR] [--out DIR] [--cache DIR] [--fresh]
+```
+
+`ROOT` is the directory the targets are under and each `PATH` a folder under it to run,
+defaulting to all of them. A target is found by the `set_*` folder it holds, either
+directly under `ROOT` or grouped by course a folder down:
+`targets/ME2_Fluids_introduction/` and `targets/EART40013_Mathematical_Methods_II/CW1/`
+are both targets. The folder's two documents are read by role rather than by name: the
+one whose name ends in `_solutions` is the solutions document, and the other is the
+questions document, which is how a sheet pairs with a solutions document the platform
+printed months later under a name of its own. A folder holding two of either is
+reported and not run.
+
+Each target is converted, and the zip it wrote is compared question by question with
+the export. The run prints one line per difference and one line of counts per target:
+
+```
+differs   ME2_Fluids_introduction: Question 2 "", part (a), text: the agent says … and the export says …
+known     ME2_Fluids_introduction: Question 1 "", main text: the agent says … and the export says …
+agrees    ME2_Fluids_introduction: q3.p2.worked_solution now agrees, remove the line
+ME2_Fluids_introduction: 4 differ, 3 known, 1 new, 2 flagged
+```
+
+A difference you have read and accepted goes into `differs.txt` beside that target's
+filter. A line of that file names the field the difference is in, and states after a `#`
+why the field differs:
+
+```
+q1.main_text           # the export keeps the spacing the platform wrote around display maths
+q2.p1.worked_solution  # Mathpix reads the separator line under the working as a minus sign
+```
+
+The file records a field rather than a sentence because the report quotes a model's
+wording. Route A reads the document on every run, and a model writes the same field
+differently each time it is asked. A difference in a field the file names is reported as
+`known` whatever its wording; a difference in any other field is reported as `differs`
+and is new; a field the file names that no longer differs is reported as `agrees`, which
+is a line to delete, and does not fail the run. The command exits 0 when every target
+ran and reported no new difference, and 1 otherwise, so a target set is a check as well
+as a report.
+
+`--filters` (default `./targets`) is the tree of saved filters, mirroring the targets:
+target `A/B` keeps its filter at `targets/A/B/filter.lua`, route A's reply at
+`targets/A/B/reply.json` and its accepted fields at `targets/A/B/differs.txt`. The first
+run over a target makes one model call for the filter and one for route A's reply, and
+writes both. Every run after that reads the two files and makes neither of those two
+calls, so the second run over a target differs from the export in the same fields as the
+first. `--fresh` reads the documents again and writes a new reply, which changes the
+wording of the report and the number of fields flagged.
+
+Those two are the only calls a saved target spares. A target with a filter runs route B
+on every run, and a model adjudicates every field the two routes word differently. A
+verdict can go the other way on a later run, so the wording of a difference and the
+`flagged` count move from run to run while the fields `differs.txt` accepts stay
+accepted. A target whose questions document is a PDF has no filter:
+pandoc cannot read a PDF, so route B does not run and route A converts the pages'
+markdown alone. `--out` (default `./out`) is where each target's set is written, under
+the target's own name, and `--cache` (default `./.in2lambda-agent`) is where the OCR of
+each PDF is kept.
+
 ## Gate
 
 Nothing merges without a replay over real documents. `gate` reruns the saved specs

in2lambda_agent/cli.py

@@ -8,7 +8,7 @@
 from pathlib import Path
 from typing import Optional, Sequence
 
-from in2lambda_agent import compare, corpus, gate, pair, pipeline, routes
+from in2lambda_agent import compare, corpus, gate, pair, pipeline, routes, targets
 from in2lambda_agent.mathpix import MathpixClient, MathpixError
 from in2lambda_agent.model import ModelError, ModelUnavailable, choose_backend
 from in2lambda_agent.ocr import ocr_pdf
@@ -119,8 +119,8 @@ def build_parser() -> argparse.ArgumentParser:
     """The command line as the design spec describes it.
 
     Returns:
-        A parser with the `convert`, `run`, `review`, `corpus`, `gate`,
-        `compare` and `ui` subcommands.
+        A parser with the `convert`, `run`, `review`, `corpus`, `targets`,
+        `gate`, `compare` and `ui` subcommands.
     """
     parser = argparse.ArgumentParser(
         prog="in2lambda-agent",
@@ -311,6 +311,46 @@ def build_parser() -> argparse.ArgumentParser:
         "another run filled converts nothing.",
     )
 
+    against_export = subcommands.add_parser(
+        "targets",
+        help="Convert each target under ROOT and compare it with its export.",
+    )
+    against_export.add_argument(
+        "root", type=Path, help="The directory the targets are under."
+    )
+    against_export.add_argument(
+        "paths",
+        nargs="*",
+        type=Path,
+        help="Folders under ROOT to run, defaulting to all of them.",
+    )
+    against_export.add_argument(
+        "--filters",
+        type=Path,
+        default=targets.DEFAULT_FILTER_DIR,
+        help="The tree each target's filter and saved reply are kept in, "
+        "mirroring the targets, with the fields the maintainer accepts a "
+        f"difference in written in {targets.DIFFERS_NAME} beside them.",
+    )
+    against_export.add_argument(
+        "--fresh",
+        action="store_true",
+        help="Read each document again rather than converting the saved reply, "
+        "which is how a target is given a new reading of its pages.",
+    )
+    against_export.add_argument(
+        "--out",
+        type=Path,
+        default=Path("out"),
+        help="Where to write each target's set, under the target's own name.",
+    )
+    against_export.add_argument(
+        "--cache",
+        type=Path,
+        default=pipeline.DEFAULT_CACHE_DIR,
+        help="Where the OCR of each PDF is kept.",
+    )
+
     check = subcommands.add_parser(
         "gate", help="Replay the corpus the baseline names and check it against it."
     )
@@ -524,6 +564,27 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
         succeeded = {"built", "skipped"}
         return 0 if rows and all(row.outcome in succeeded for row in rows) else 1
 
+    if args.command == "targets":
+        results = targets.run(
+            args.root,
+            paths=args.paths,
+            filters=args.filters,
+            out_dir=args.out,
+            cache_dir=args.cache,
+            settings=load_settings(),
+            fresh=args.fresh,
+        )
+        new = sum(len(one.new) for one in results)
+        failed = [one for one in results if one.error]
+        print(
+            f"{len(results)} target{'' if len(results) == 1 else 's'}, "
+            f"{new} new difference{'' if new == 1 else 's'}"
+            + (f", {len(failed)} did not run" if failed else "")
+        )
+        # A root with no target under it is a mistyped path rather than a clean
+        # run, so an empty run fails like a new difference does.
+        return 0 if results and not new and not failed else 1
+
     if args.command == "gate":
         baseline = gate.read_baseline(args.baseline)
         # The directory is printed and is not deleted, so that the drafts of a

in2lambda_agent/pair.py

@@ -13,6 +13,10 @@
 
 `pairs_in` reads a whole folder that way: every sheet in it with its solutions
 document, which is what a run over a folder converts.
+
+`target_documents` reads the other kind of folder, the one holding a single set.
+There is nothing to pair there, so the roles alone decide: the document whose
+name ends in `solutions` answers the other one, whatever either is called.
 """
 
 import re
@@ -134,6 +138,52 @@ def pairs_in(folder: Path) -> list[tuple[Path, Optional[Path]]]:
     return [(path, solutions_beside(path)) for path in sheets.values()]
 
 
+def target_documents(folder: Path) -> tuple[Path, Optional[Path]]:
+    """The two documents of a target folder, paired by role rather than by name.
+
+    A target folder holds one set, so there is nothing to pair: the document
+    whose name ends in `solutions` or `sol` is the solutions document and the
+    other one is the questions document, whatever the two are called. The sets
+    are often exported months apart from the sheet, so their names share only
+    the course.
+
+    Args:
+        folder: A target's folder: its two documents, and the folder Lambda
+            Feedback exported from them.
+
+    Returns:
+        The questions document, and the solutions document or None where the
+        folder holds none. Subfolders are not looked into.
+
+    Raises:
+        ValueError: Where the folder holds no questions document, or more than
+            one document of either role, naming what it holds. Which of two
+            sheets is the target's is not this function's to guess.
+    """
+    folder = Path(folder)
+    documents = [one for one in _files_in(folder) if one.suffix.lower() in DOCUMENTS]
+    by_role = {
+        "questions": [one for one in documents if questions_stem(one) is None],
+        "solutions": [one for one in documents if questions_stem(one) is not None],
+    }
+    for role, held in by_role.items():
+        if len(held) > 1:
+            named = ", ".join(one.name for one in held)
+            raise ValueError(
+                f"{folder} holds {len(held)} {role} documents: {named}. A target "
+                "folder holds one set: one questions document, and a solutions "
+                "document whose name ends in `_solutions`."
+            )
+    if not by_role["questions"]:
+        held = ", ".join(one.name for one in _files_in(folder)) or "nothing"
+        raise ValueError(
+            f"{folder} holds no questions document, only {held}. A target "
+            f"folder holds one file whose suffix is one of {' '.join(DOCUMENTS)} "
+            "and whose name does not end in `_solutions`."
+        )
+    return by_role["questions"][0], (by_role["solutions"] or [None])[0]
+
+
 def of(source: Path) -> tuple[Path, Optional[Path]]:
     """The two documents a run freezes, whichever of them the user named.
 

in2lambda_agent/routes.py

@@ -399,6 +399,10 @@ class Converted:
     zip_path: Optional[Path]
     flags: list[Flag]
     reply: Reply_
+    # Route A's reply, before reconciling. A caller that needs the same answer
+    # twice saves this reply and passes it back as `convert`'s `route_a`; a
+    # second call to the model returns different wording.
+    route_a: Reply_ = field(default_factory=list)
     tokens: int = 0
     # The counts of the reconciliation, zero where route B did not run.
     fields: int = 0
@@ -494,6 +498,7 @@ def convert(
     lua: Optional[Path] = None,
     name: str = "set",
     on_stage: Optional[Callable[[str, str], None]] = None,
+    route_a: Optional[Reply_] = None,
 ) -> Converted:
     """Route A, route B where a filter is given, reconcile, verify, write.
 
@@ -502,6 +507,10 @@ def convert(
     result and `route_b_error` holds pandoc's message, so that one sheet of a folder does
     not stop the other eight.
 
+    `route_a`, where it is given, is a reply from an earlier run of this document, kept
+    by a caller who needs the same answer twice: route A is not called, and the result's
+    `route_a` is what was given.
+
     `on_stage`, where it is given, is called with a name and a message as each step
     finishes - `ocr`, `route A`, `route B`, `fields`, `build` - so that a caller watching
     a run shows each line as the step ends rather than the report at the end of it.
@@ -518,9 +527,14 @@ def said(stage: str, message: str) -> None:
     solutions_md = markdown_of(solutions, cache_dir, settings)[0] if solutions else None
     said("ocr", "; ".join(read))
     source = markdown + ("\n" + solutions_md if solutions_md else "")
-    reply, usage = direct(markdown, solutions_md, backend)
-    tokens = usage.usage.input_tokens + usage.usage.output_tokens
-    said("route A", f"{tokens} tokens")
+    if route_a is None:
+        route_a, usage = direct(markdown, solutions_md, backend)
+        tokens = usage.usage.input_tokens + usage.usage.output_tokens
+        said("route A", f"{tokens} tokens")
+    else:
+        tokens = 0
+        said("route A", "the reply given, no call made")
+    reply = route_a
     counts, error = (0, 0, 0, 0), None
     flags = [Flag(k, fields(reply)[k], "", "not a quote of the source") for k in not_verbatim(reply, source)]
     if lua is None:
@@ -547,7 +561,7 @@ def said(stage: str, message: str) -> None:
             flags.append(Flag(k, fields(reply)[k], "", STRAY_MINUS))
     result = Converted(
         set=to_set(reply, name=name, directory=images), zip_path=None, flags=flags,
-        reply=reply, tokens=tokens,
+        reply=reply, route_a=route_a, tokens=tokens,
         fields=counts[0], agreed=counts[1], defaulted=counts[2], adjudicated=counts[3],
         route_b_error=error,
     )