Replay each document's draft log in the corpus sweep - #27
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
in2lambda-agent corpus --replayreruns each set's saved spec and no fixing round, so a document the rounds repaired in the spec-writing sweep replays tofaulted, and the replay sweep (t21's gate) reports 0 built over ExampleContents though the sweep built them. in2lambda's draft log exists for this: it holds every command with its ranges and no field values, andin2lambda draft replayrebuilds a draft from the frozen source and the log without the model.During a sweep, after the last round, copy each document's log (the
loglist of its draft, nothing else - the fields hold the document's text and must not leave the corpus) to--specsbeside the set's spec as<document>.commands.json. Under--replay, afterspec run, execute those commands in order through in2lambda's draft commands (package.py already runs one command; run the list, refusing on the first error and recording it as the row's reason), then validate and build as now with no model call. A document whose log replays to a clean report isbuilt. Update corpus.py's docstring and the README's corpus section: a set's spec and its logs are what a sweep keeps, and a replay is the spec, the logs and nothing from the model. Tests: a sweep with a fake backend whose round runs one command writes the log file; a replay with no backend at all builds that document. Done whencorpus <ExampleContents> PHYS40002 --replaybuilds every document the last sweep built, with zero model calls, and t21's baseline for ExampleContents can be written from it.Workbench ticket t27.