Repository navigation
test: the 23 tests left in KNOWN stop cutting text - #876
Conversation
The draft pull request opens before any code: the Spec as its checklist.
|
The pull request's board
🤖 AI-generated · Claude Opus 5.5 (Anthropic) |
The 23 test files #846 left in KNOWN (6 in tests/repro, 17 in tests/workflows) no longer cut text between literals. A step's run: is run with stubs and the test asserts on the calls, the exit code and $GITHUB_OUTPUT; an if: is evaluated with steps.evaluate; wiring is compared as YAML fields; a doc, template or prompt is read as structure, by its headings (steps.sections) or numbered steps (steps.numbered). The tests/repro files keep their own stubs and subprocess call, as the workflow way's runs-step guide asks. KNOWN in scripts/literal-slices.py is empty, and test_c3 no longer expects tests/workflows to hold a slice. Assisted-by: Claude Opus 5.5 (Anthropic)
A test runs literal-slices.py --all on each of the 23 files #846 left in KNOWN and asserts that every one still exists and none is reported. Assisted-by: Claude Opus 5.5 (Anthropic)
Claude review · risk high · complexity high · type testNothing that must be fixed · 1 minor The new commits fixed two earlier minors. The header comment in pull-request.yml now says KNOWN is empty. The --all guard is back in test_c3, as a tmp_path control file that must be reported exactly once, so a --all that reads nothing fails the test. I could not run the tests in this review (Bash was denied), so I read the control logic and did not execute it. The long docstring line in scripts/literal-slices.py no longer appears (no line is 140 characters or longer). The configuration-doc lookup in tests/workflows/test_configuration_doc.py:44 is not in this diff, and sections() in steps.py still accepts any heading level. That minor stays open. The description matches the diff. It carries the "How I tested it" and "Removed, with the reason (C2)" blocks twice, which is a cosmetic duplicate. The Credits block is empty, with the reason given in Decisions, and I see nothing to correct.
Policy floor: high (touches high-risk paths: .github/workflows/pull-request.yml, scripts/literal-slices.py, tests/repro/test_issue_549.py, tests/repro/test_issue_556.py, tests/repro/test_issue_575.py …). Reviewed 4cc51db (since 05cbd9d; review 2 of 5 automatic). Author trusted for auto-merge: false. 🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.15, 8 turns |
|
Two changes before this merges, both from the review's minors:
Then mark it ready. 🤖 AI-assisted · Claude Opus 5.5 (Anthropic) |
The scripts job's comment above literal-slices.py now says KNOWN is empty and that only a reasoned exception goes back on it. The test that the worst wording tests slice no more runs --all on a tmp_path file with one known slice in the same call and asserts it is reported, so it fails if --all silently reads nothing. Assisted-by: Claude Opus 5.5 (Anthropic)
📝 What changes
Convert each of the 23 listed tests so it no longer cuts text between literals, file by file, then empty
KNOWNin scripts/literal-slices.py (its type staysdict, soKNOWN == slices_by_file(...)in tests/workflows/test_literal_slices.py still holds, with both sides{}). No workflow, script, doc or template that the tests cover changes. The pull request is typedtest:because it changes only tests and the list. That keeps it outside test-first'sfix/featlimit of 20 tests (scripts/test-first.pyMAX_TESTS), which made #846 leave these files out. It still closes #873.Each slice is replaced by whichever of these fits what the test asserts:
run:(most tests/workflows files, the repro files 549, 556, 575, 577, 696 and 774): find the step byid/namewithtests/workflows/steps.py(step,run) and run it in bash. Stubgh,gitand the helper scripts on PATH, and set the env that picks the branch under test (for exampleLANE=bugfix-reproducedfor fix: the autopilot cannot bring main into a conflicting branch #549, an empty firstacceptedfor fix: a stop, an edit and a new acceptance break the autopilot's path #696). Then assert on the calls recorded, the exit code and$GITHUB_OUTPUT, not on the text of the branch. The tests/repro files whose way hasruns-step(review-policy.ymlrepro.ways) keep their ownsubprocesscall, as the repro guide requires, and do not import steps.py.if:or a job's wiring (fix_gate_script,gate, the required checks,who-decides, triage's non-write users): read it through YAML and evaluate the expression withsteps.evaluateon the events that matter, or compare structured fields (needs,with,permissions, step order by index insteps).{heading: body}by its headings, a template into its markers, and a prompt into its numbered steps. The test then asserts on that structure, for example the order of the steps or every setting appearing in its section. If the logic lives in a script under scripts/ (for exampleautopilot.py cap-comment,settledorissue-schema.py), the test calls the script instead.As each file stops slicing, its entry leaves
KNOWN.literal-slices.pyalready fails a file that has fewer slices than its count, so the list and the files move together. The KNOWN comment changes to say the list is empty and only a reasoned exception may be added back. Thetests/workflows tests/reprorun in the scripts job then reports 0 files listed.💡 Why
Closes #873. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.
🎯 Acceptance criteria
KNOWNis read, then it is empty, andpython3 scripts/literal-slices.pyover tests/workflows and tests/repro passes.Test: tests/workflows/test_literal_slices.py (new
test_c1_known_is_empty_and_the_tests_pass_it: assertsload().KNOWN == {}and thatliteral-slices.py tests/workflows tests/reproexits 0 reporting 0 files listed. It fails at the base, where KNOWN holds 23 files.)Test: tests/workflows/test_literal_slices.py (new
test_c2_none_of_the_23_files_slices: runsliteral-slices.py --allon each of the 23 paths and asserts that none is reported and every path still exists. It fails at the base. The converted files themselves pass in the fullpytest tests/run.)🧭 Decisions
🙌 Credits
🧪 How I tested it
How I tested it
literal-slices.py --allreported them. Both pass at HEAD.python3 scripts/literal-slices.py tests/workflows tests/reproprints "0 files still listed in KNOWN", andpython3 scripts/literal-slices.py --testpasses.python3 -m pytest -q tests, gives 681 passed and 1 skipped.run:and the test failed. I didn't keep those copies.test_c3_the_worst_wording_tests_of_tests_workflows_slice_no_morenow runs--allon tests/workflows and on a tmp_path file with one known slice, in the same call, and checks that the slice is reported. If--allreads nothing, the test fails.make pre-pr.Removed, with the reason (C2)
on:in pull-request.yml names "required checks" and the two check names. It checked only wording. A behaviour check replaces it: the two check names are worked out from the jobs' YAML (name, andtest-first.yml's job) and must equal the required list.steps.evaluateon itsif:, checks on YAML fields, or reading the doc as structure (steps.sectionsandsteps.numbered, new in tests/workflows/steps.py).How I tested it
literal-slices.py --allreported them. Both pass at HEAD.python3 scripts/literal-slices.pyprints "0 files still listed in KNOWN", andpython3 scripts/literal-slices.py --testpasses.python3 -m pytest -p no:cacheprovider --rootdir=. -q tests, gives 681 passed and 1 skipped.run:and the test failed. I didn't keep those copies.make pre-pr.Removed, with the reason (C2)
on:in pull-request.yml names "required checks" and the two check names. It checked only wording. It is replaced by a behaviour check: the two check names are worked out from the jobs' YAML (name, andtest-first.yml's job) and must equal the required list.test_c3_the_worst_wording_tests_of_tests_workflows_slice_no_moreno longer requires at least one slice to be found in tests/workflows, because none is left. The tests in the same file that use tmp_path still show that--allfinds a slice.steps.evaluateon itsif:, YAML field checks, or reading the doc as structure (steps.sectionsandsteps.numbered, new in tests/workflows/steps.py).🤖 AI-generated · Claude Opus 5.5 (Anthropic)