Add a convert command to the command line

README.md

@@ -4,14 +4,21 @@ Turns a PDF, docx, tex or md file into a validated Lambda Feedback set.
 [in2lambda](https://github.com/lambda-feedback/in2lambda) performs every deterministic
 step and every write; this agent performs the OCR, the model calls and the loop control.
 
-One model call per document set writes a spec of selectors, which is layer 1, and
-in2lambda runs that spec over the frozen source. A draft the checks fault returns to
-the model as a fixing round, which repairs the draft one field at a time at layers 3
-and 4, until the checks report no error or the round limit is reached.
-
-[docs/how-it-works.md](docs/how-it-works.md) describes a run stage by stage: the line
-each stage prints, the file each stage writes, and what each of the three model calls
-is given and may write.
+`convert` reads the document by two routes and compares them. Route A is one model call
+that returns the set as JSON. Route B is a Lua filter, written by one model call per
+folder, that pandoc runs with no further call. A field the two routes read the same way
+is taken as it stands; a field they read differently goes to a small adjudicating call;
+a field that call cannot settle is flagged for a person to read. Every field of either
+route must be a quote of the document. [docs/plan.md](docs/plan.md) describes each step
+and what it detects.
+
+`run --route spec` is the earlier route. One model call per document set writes a spec
+of selectors, which is layer 1, and in2lambda runs that spec over the frozen source. A
+draft the checks fault returns to the model as a fixing round, which repairs the draft
+one field at a time at layers 3 and 4, until the checks report no error or the round
+limit is reached. [docs/how-it-works.md](docs/how-it-works.md) describes that route
+stage by stage: the line each stage prints, the file each stage writes, and what each of
+the three model calls is given and may write.
 
 ## Install
 
@@ -47,16 +54,92 @@ installed and `claude login` run. A run over a set whose spec is saved makes no
 call, and needs no key at all. A stage that needs a variable names that variable and
 the run exits 1.
 
-## Run
+## Convert
+
+```sh
+poetry run in2lambda-agent convert sheet.pdf
+```
+
+In full:
+
+```sh
+poetry run in2lambda-agent convert DOCUMENT [--solutions FILE] [--filter FILE | --write-filter] [--out DIR] [--cache DIR]
+```
+
+`DOCUMENT` is a PDF, markdown, tex or docx file. Mathpix converts a PDF first and the
+agent keeps the markdown and the images under the PDF's hash in `--cache` (default
+`./.in2lambda-agent`), so a second conversion of the same PDF makes no Mathpix call.
+pandoc converts a tex or docx file. `--out` defaults to `./out`, where in2lambda writes
+the set's JSON folder and its zip.
+
+`--solutions` names the document holding the solutions. Without it the agent takes the
+file beside `DOCUMENT` whose name is the document's with `_solutions`, `-solutions` or
+` Solutions` after it, in any case, and whose suffix is the same: `Worksheet_1.pdf` and
+`Worksheet_1_solutions.pdf`. Naming the solutions document converts the pair too, and
+the set is named after the questions document whichever of the two you name; a solutions
+document with no questions document beside it converts on its own. Route A reads both
+documents in one call, and route B reads each under its own role. The `solutions` line
+of the report names the document the
+conversion read. Where the two names share no stem, as they do where the platform has
+put the time of the download in each, the agent finds no solutions document and the
+line names none:
+
+```
+solutions none found beside ME2_Fluids_2024-03-11.pdf; pass --solutions FILE
+```
+
+A conversion that reads that line and goes on writes an empty answer and an empty
+worked solution for every question.
+
+`--filter` names the Lua filter route B runs, which is the file `--write-filter` wrote
+for another sheet of the same set. `--write-filter` writes one for this document with a
+model call and keeps it at `OUT/filter.lua`. The two options together are refused: a
+conversion runs one filter. With neither option route A converts the document alone, no
+field is compared, and the counts line says so.
+
+The command prints the solutions document, one line per flagged field, the counts of
+the comparison, and the zip:
+
+```
+solutions /home/me/sheets/sheet_solutions.pdf
+flag      q2.p1.worked_solution: a stray minus sign inside or beside a display maths; Mathpix reads a separator line as one
+flag      q4.p2.content: two readings of the source
+  A: Find the drag force on the plate.
+  B: Find the drag force on the plate, in newtons.
+fields    60 fields, agreed 54, defaulted 4, adjudicated 2, flagged 2
+build     /home/me/out/sheet.zip
+```
+
+A flag names the field, the reason, and each route's text where both routes filled the
+field. `fields` counts the fields the two routes agreed on, the fields one route alone
+filled, the fields the adjudicating call settled, and the fields flagged. With no
+filter there is no comparison to count, so the line is `60 fields, route B did not run`:
+the fields are route A's, and each one is flagged or is route A's word for it. A filter
+run that fails counts route A's fields in the same way, and adds a `route B failed` line
+naming pandoc's message; the set is route A's reading alone.
+
+`convert` exits 1 where a named file is not there, and where Mathpix, the model or
+pandoc failed, and 0 otherwise. A flagged
+field does not change the exit code: the zip is written whatever the flags say, and a
+person reads the flags after it.
+
+## Run: the spec route
+
+`run DOCUMENT` converts the document as `convert` does and takes the same options.
+`--route spec` runs the earlier route instead, which the rest of this section describes.
+The two routes' options do not mix: `--review` without `--route spec`, or
+`--write-filter` with it, is refused naming the route the option belongs to, because a
+run that read it and ignored it would be a spec written twice or a review never stopped
+for.
 
 ```sh
-poetry run in2lambda-agent run sheet.pdf
+poetry run in2lambda-agent run sheet.pdf --route spec
 ```
 
 In full:
 
 ```sh
-poetry run in2lambda-agent run SOURCE [--spec FILE] [--review none|sample|per-question] [--rounds N] [--tries N] [--sample N] [--cache DIR] [--fresh-ocr] [--out DIR]
+poetry run in2lambda-agent run SOURCE --route spec [--spec FILE] [--review none|sample|per-question] [--rounds N] [--tries N] [--sample N] [--cache DIR] [--fresh-ocr] [--out DIR]
 ```
 
 `SOURCE` is a PDF, markdown, tex or docx file. Mathpix converts a PDF first, and the
@@ -164,7 +247,7 @@ spec, and is not one of the rounds.
 
 ### Exit codes
 
-`run` and `review` exit 0 where they wrote a zip, and where a review is still waiting
+`run --route spec` and `review` exit 0 where they wrote a zip, and where a review is still waiting
 for a verdict. They exit 1 where the checks still fault the draft, where in2lambda
 refused the build, and where a review has no question left to answer and no zip was
 written. A missing credential, an unavailable backend, a reply that is not a spec, a

in2lambda_agent/cli.py

@@ -2,12 +2,13 @@
 
 import argparse
 import getpass
+import subprocess
 import sys
 import tempfile
 from pathlib import Path
 from typing import Optional, Sequence
 
-from in2lambda_agent import compare, corpus, gate, pipeline
+from in2lambda_agent import compare, corpus, gate, pair, pipeline, routes
 from in2lambda_agent.mathpix import MathpixClient, MathpixError
 from in2lambda_agent.model import ModelError, ModelUnavailable, choose_backend
 from in2lambda_agent.ocr import ocr_pdf
@@ -85,26 +86,87 @@ def reviewer_name(given: Optional[str]) -> str:
         return "reviewer"
 
 
+def _conversion_options(parser: argparse.ArgumentParser) -> None:
+    """Adds the options of a conversion, which `convert` and `run` both take.
+
+    Args:
+        parser: The subcommand's parser.
+    """
+    parser.add_argument(
+        "--solutions",
+        type=Path,
+        default=None,
+        help="The solutions document. Default: the file beside the document "
+        "whose name is the document's with `_solutions` after it.",
+    )
+    filter_ = parser.add_mutually_exclusive_group()
+    filter_.add_argument(
+        "--filter",
+        type=Path,
+        default=None,
+        help="The Lua filter route B runs. Without one, route A converts the "
+        "document alone and no field is compared.",
+    )
+    filter_.add_argument(
+        "--write-filter",
+        action="store_true",
+        help="Write route B's filter for this document with a model call, and "
+        "keep it at `OUT/filter.lua`.",
+    )
+
+
 def build_parser() -> argparse.ArgumentParser:
     """The command line as the design spec describes it.
 
     Returns:
-        A parser with the `run`, `review`, `corpus`, `gate`, `compare` and
-        `ui` subcommands.
+        A parser with the `convert`, `run`, `review`, `corpus`, `gate`,
+        `compare` and `ui` subcommands.
     """
     parser = argparse.ArgumentParser(
         prog="in2lambda-agent",
         description="Turns a source file into a Lambda Feedback set.",
     )
     subcommands = parser.add_subparsers(dest="command", required=True)
 
-    run = subcommands.add_parser("run", help="Convert SOURCE into a set.")
+    convert = subcommands.add_parser(
+        "convert", help="Convert DOCUMENT into a set through both routes."
+    )
+    convert.add_argument(
+        "document",
+        type=Path,
+        help="The question file to convert: a PDF, markdown, tex or docx file.",
+    )
+    _conversion_options(convert)
+    convert.add_argument(
+        "--out",
+        type=Path,
+        default=Path("out"),
+        help="Where to write the set's JSON folder and zip.",
+    )
+    convert.add_argument(
+        "--cache",
+        type=Path,
+        default=pipeline.DEFAULT_CACHE_DIR,
+        help="Where the OCR of each PDF is kept.",
+    )
+
+    run = subcommands.add_parser(
+        "run", help="Convert SOURCE into a set; `convert` under the default route."
+    )
     run.add_argument(
         "source",
         type=Path,
         help="The question file to convert. A solutions file beside it, named "
         "after it, is frozen with it.",
     )
+    run.add_argument(
+        "--route",
+        choices=("direct", "spec"),
+        default="direct",
+        help="Which route converts the document: `direct` is the `convert` "
+        "command, and `spec` writes a spec of selectors and runs it.",
+    )
+    _conversion_options(run)
     run.add_argument(
         "--spec",
         type=Path,
@@ -308,6 +370,118 @@ def build_parser() -> argparse.ArgumentParser:
     return parser
 
 
+# Each route's own options. `run` takes both sets, because argparse cannot know
+# the route until it has parsed the line, so the run refuses an option of the
+# route it is not taking rather than reading it and throwing it away.
+_SPEC_ROUTE_OPTIONS = {
+    "spec": "--spec",
+    "review": "--review",
+    "rounds": "--rounds",
+    "tries": "--tries",
+    "sample": "--sample",
+    "fresh_ocr": "--fresh-ocr",
+}
+_DIRECT_ROUTE_OPTIONS = {
+    "solutions": "--solutions",
+    "filter": "--filter",
+    "write_filter": "--write-filter",
+}
+
+
+def misplaced_option(args: argparse.Namespace) -> Optional[str]:
+    """What is wrong where `run` was given an option of the other route.
+
+    Args:
+        args: The parsed arguments of `run`.
+
+    Returns:
+        What to print, naming the option and the route it belongs to, or None
+        where every option given belongs to the route the run is taking.
+    """
+    if args.route == "direct":
+        options, route, fix = _SPEC_ROUTE_OPTIONS, "spec", "add --route spec"
+    else:
+        options, route, fix = _DIRECT_ROUTE_OPTIONS, "direct", "drop --route spec"
+    # An option counts as given where it is not the parser's default, which is
+    # read back from the parser rather than repeated here.
+    defaults = build_parser().parse_args(["run", str(args.source)])
+    for dest, name in options.items():
+        if getattr(args, dest) != getattr(defaults, dest):
+            return f"{name} is an option of the {route} route; {fix}"
+    return None
+
+
+def convert_command(args: argparse.Namespace) -> int:
+    """Converts one document through both routes and prints the report.
+
+    Args:
+        args: The parsed arguments of `convert`, or of `run` under the direct
+            route, which takes the same options.
+
+    Returns:
+        0 where the zip was written, and 1 where a conversion step failed. A
+        flagged field does not change the code: the flags are what a person
+        reads after the build, and no check blocks the write.
+    """
+    # `run SOURCE` converts the same document, under the other name.
+    document = Path(getattr(args, "document", None) or args.source)
+    if args.solutions is not None:
+        # The user named the two documents, so the folder is not asked.
+        solutions = args.solutions
+    else:
+        # Either half of a pair may be named, so the pairing goes both ways: name the
+        # solutions document and the questions document beside it is what converts, and
+        # the set is named after it. A solutions document with none beside it comes back
+        # as the document itself, and converts on its own.
+        document, solutions = pair.of(document)
+    if solutions is not None:
+        print(f"solutions {solutions}")
+    elif pair.questions_stem(document) is None:
+        # A pair whose two names share no stem, which is what the platform writes where
+        # it puts the time of the download in each name, is a sheet whose solutions the
+        # run did not find. A run that said nothing would read as a sheet with none,
+        # and the set it writes holds an empty answer for every question.
+        print(f"solutions none found beside {document}; pass --solutions FILE")
+    out_dir = Path(args.out)
+    settings = load_settings()
+    backend = choose_backend(settings)
+    lua = args.filter
+    try:
+        if args.write_filter:
+            out_dir.mkdir(parents=True, exist_ok=True)
+            lua = out_dir / "filter.lua"
+            lua.write_text(
+                routes.write_filter(document, solutions, backend)[0], encoding="utf-8"
+            )
+        result = routes.convert(
+            document,
+            solutions,
+            out_dir=out_dir,
+            cache_dir=args.cache,
+            backend=backend,
+            settings=settings,
+            lua=lua,
+            name=document.stem,
+        )
+    except (MathpixError, ModelUnavailable, ModelError, OSError, routes.BadReply) as error:
+        # A document that is not there raises an OSError here, because this route reads
+        # the file itself and in2lambda never sees the name. A reply that is not a JSON
+        # list of questions raises BadReply, as a reply that is not a spec raises
+        # BadSpec on the other route.
+        print(f"in2lambda-agent: {error}", file=sys.stderr)
+        return 1
+    except subprocess.CalledProcessError as error:
+        # pandoc read the document, or ran the filter, and refused. Its own
+        # message names the line; the exit status alone names nothing.
+        stderr = (error.stderr or b"").decode("utf-8", "replace").strip()
+        print(f"in2lambda-agent: {stderr or error}", file=sys.stderr)
+        return 1
+
+    for line in result.report():
+        print(line)
+    return 0 if result.zip_path else 1
+
+
 def main(argv: Optional[Sequence[str]] = None) -> int:
     """Runs the command.
 
@@ -319,6 +493,15 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
     """
     args = build_parser().parse_args(argv)
 
+    if args.command == "run":
+        wrong = misplaced_option(args)
+        if wrong:
+            print(f"in2lambda-agent: {wrong}", file=sys.stderr)
+            return 1
+
+    if args.command == "convert" or (args.command == "run" and args.route == "direct"):
+        return convert_command(args)
+
     if args.command == "corpus":
         rows = corpus.sweep(
             args.root,