Skip to content

test: the 23 tests left in KNOWN stop cutting text - #876

Merged
soydiloreto merged 6 commits into
mainfrom
chore/873-the-23-tests-left-in
Nov 19, 2026
Merged

soydiloreto merged 6 commits into
mainfrom
chore/873-the-23-tests-left-in

Conversation

@dilux-bot

@dilux-bot dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

📝 What changes

Convert each of the 23 listed tests so it no longer cuts text between literals, file by file, then empty KNOWN in scripts/literal-slices.py (its type stays dict, so KNOWN == slices_by_file(...) in tests/workflows/test_literal_slices.py still holds, with both sides {}). No workflow, script, doc or template that the tests cover changes. The pull request is typed test: because it changes only tests and the list. That keeps it outside test-first's fix/feat limit of 20 tests (scripts/test-first.py MAX_TESTS), which made #846 leave these files out. It still closes #873.

Each slice is replaced by whichever of these fits what the test asserts:

  • A step's run: (most tests/workflows files, the repro files 549, 556, 575, 577, 696 and 774): find the step by id/name with tests/workflows/steps.py (step, run) and run it in bash. Stub gh, git and the helper scripts on PATH, and set the env that picks the branch under test (for example LANE=bugfix-reproduced for fix: the autopilot cannot bring main into a conflicting branch #549, an empty first accepted for fix: a stop, an edit and a new acceptance break the autopilot's path #696). Then assert on the calls recorded, the exit code and $GITHUB_OUTPUT, not on the text of the branch. The tests/repro files whose way has runs-step (review-policy.yml repro.ways) keep their own subprocess call, as the repro guide requires, and do not import steps.py.
  • A step's if: or a job's wiring (fix_gate_script, gate, the required checks, who-decides, triage's non-write users): read it through YAML and evaluate the expression with steps.evaluate on the events that matter, or compare structured fields (needs, with, permissions, step order by index in steps).
  • A prompt or a doc (commit cadence, pre-pr, the configuration page, who signs what, the PR template's markers): read the section as structure, not as a literal-to-literal cut. Markdown is split into {heading: body} by its headings, a template into its markers, and a prompt into its numbered steps. The test then asserts on that structure, for example the order of the steps or every setting appearing in its section. If the logic lives in a script under scripts/ (for example autopilot.py cap-comment, settled or issue-schema.py), the test calls the script instead.
  • If a slice checked only wording that no behaviour stands behind, the assertion is removed and the pull request says so, file by file (C2). No test function is deleted without a replacement behaviour test or that stated reason.

As each file stops slicing, its entry leaves KNOWN. literal-slices.py already fails a file that has fewer slices than its count, so the list and the files move together. The KNOWN comment changes to say the list is empty and only a reasoned exception may be added back. The tests/workflows tests/repro run in the scripts job then reports 0 files listed.

💡 Why

Closes #873. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.

🎯 Acceptance criteria

  • C1 Given scripts/literal-slices.py, when its KNOWN is read, then it is empty, and python3 scripts/literal-slices.py over tests/workflows and tests/repro passes.
    Test: tests/workflows/test_literal_slices.py (new test_c1_known_is_empty_and_the_tests_pass_it: asserts load().KNOWN == {} and that literal-slices.py tests/workflows tests/repro exits 0 reporting 0 files listed. It fails at the base, where KNOWN holds 23 files.)
  • C2 Given each of the 23 files, when the whole tests/ suite runs, then it passes, and no test was deleted without a behaviour test or a stated reason in the pull request (a slice that checked only wording).
    Test: tests/workflows/test_literal_slices.py (new test_c2_none_of_the_23_files_slices: runs literal-slices.py --all on each of the 23 paths and asserts that none is reported and every path still exists. It fails at the base. The converted files themselves pass in the full pytest tests/ run.)

🧭 Decisions

  • 2026-11-19: Added two pure readers to tests/workflows/steps.py, sections() (markdown by heading) and numbered() (numbered list items), shared by the doc, template and prompt tests, as the Spec allows.
  • 2026-11-19: Dropped test_c3's guard that --all finds some slice in tests/workflows: with every file converted none is left; the tmp_path tests of the same file still prove --all detects a slice.
  • 2026-11-19: Credits list nobody: the issue's author, @​soydiloreto, is the code owner who accepts and merges it, and AGENTS.md rule 10 says that maintainer is never a co-author (an earlier run listed them; they can add it back with /coauthor).

🙌 Credits

🧪 How I tested it

How I tested it

  • The C1 and C2 tests in tests/workflows/test_literal_slices.py both failed first, at the base, where KNOWN still listed 23 files and literal-slices.py --all reported them. Both pass at HEAD.
  • python3 scripts/literal-slices.py tests/workflows tests/repro prints "0 files still listed in KNOWN", and python3 scripts/literal-slices.py --test passes.
  • Running the whole suite, python3 -m pytest -q tests, gives 681 passed and 1 skipped.
  • For each test that now runs a step, I broke a scratch copy of that step's run: and the test failed. I didn't keep those copies.
  • test_c3_the_worst_wording_tests_of_tests_workflows_slice_no_more now runs --all on tests/workflows and on a tmp_path file with one known slice, in the same call, and checks that the slice is reported. If --all reads nothing, the test fails.
  • The local review's checks (conventions, policy, brief) pass. This repository has no make pre-pr.

Removed, with the reason (C2)

  • In tests/workflows/test_required_checks.py, I removed the assertion that the comment above on: in pull-request.yml names "required checks" and the two check names. It checked only wording. A behaviour check replaces it: the two check names are worked out from the jobs' YAML (name, and test-first.yml's job) and must equal the required list.
  • No other test function or assertion was removed. Each slice was replaced by one of these: a run of the step with stubs, steps.evaluate on its if:, checks on YAML fields, or reading the doc as structure (steps.sections and steps.numbered, new in tests/workflows/steps.py).

How I tested it

  • The C1 and C2 tests in tests/workflows/test_literal_slices.py both failed first, at the base, where KNOWN still listed 23 files and literal-slices.py --all reported them. Both pass at HEAD.
  • python3 scripts/literal-slices.py prints "0 files still listed in KNOWN", and python3 scripts/literal-slices.py --test passes.
  • Running the whole suite, python3 -m pytest -p no:cacheprovider --rootdir=. -q tests, gives 681 passed and 1 skipped.
  • For each test that now runs a step, I broke a scratch copy of that step's run: and the test failed. I didn't keep those copies.
  • The local review's checks (conventions, policy, brief) pass. This repository has no make pre-pr.

Removed, with the reason (C2)

  • In tests/workflows/test_required_checks.py, I removed the assertion that the comment above on: in pull-request.yml names "required checks" and the two check names. It checked only wording. It is replaced by a behaviour check: the two check names are worked out from the jobs' YAML (name, and test-first.yml's job) and must equal the required list.
  • In tests/workflows/test_literal_slices.py, test_c3_the_worst_wording_tests_of_tests_workflows_slice_no_more no longer requires at least one slice to be found in tests/workflows, because none is left. The tests in the same file that use tmp_path still show that --all finds a slice.
  • No other test function or assertion was removed. Each slice was replaced by a run of the step with stubs, steps.evaluate on its if:, YAML field checks, or reading the doc as structure (steps.sections and steps.numbered, new in tests/workflows/steps.py).

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

The draft pull request opens before any code: the Spec as its checklist.
@dilux-bot dilux-bot Bot added the autopilot Opened by the autopilot from an accepted issue label Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

The pull request's board

State
Waits on nobody

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot
dilux-bot Bot marked this pull request as ready for review November 19, 2026 13:20
dilux-bot Bot added 3 commits November 19, 2026 13:21
The 23 test files #846 left in KNOWN (6 in tests/repro, 17 in tests/workflows) no longer cut text between literals. A step's run: is run with stubs and the test asserts on the calls, the exit code and $GITHUB_OUTPUT; an if: is evaluated with steps.evaluate; wiring is compared as YAML fields; a doc, template or prompt is read as structure, by its headings (steps.sections) or numbered steps (steps.numbered). The tests/repro files keep their own stubs and subprocess call, as the workflow way's runs-step guide asks. KNOWN in scripts/literal-slices.py is empty, and test_c3 no longer expects tests/workflows to hold a slice.

Assisted-by: Claude Opus 5.5 (Anthropic)
A test runs literal-slices.py --all on each of the 23 files #846 left in KNOWN and asserts that every one still exists and none is reported.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot
dilux-bot Bot requested a review from soydiloreto as a code owner November 19, 2026 13:21
@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:high Set by the Claude review type:test The kind of change, read from the diff by the Claude review labels Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Claude review · risk high · complexity high · type test

Nothing that must be fixed · 1 minor

The new commits fixed two earlier minors. The header comment in pull-request.yml now says KNOWN is empty. The --all guard is back in test_c3, as a tmp_path control file that must be reported exactly once, so a --all that reads nothing fails the test. I could not run the tests in this review (Bash was denied), so I read the control logic and did not execute it. The long docstring line in scripts/literal-slices.py no longer appears (no line is 140 characters or longer). The configuration-doc lookup in tests/workflows/test_configuration_doc.py:44 is not in this diff, and sections() in steps.py still accepts any heading level. That minor stays open. The description matches the diff. It carries the "How I tested it" and "Removed, with the reason (C2)" blocks twice, which is a cosmetic duplicate. The Credits block is empty, with the reason given in Decisions, and I see nothing to correct.

  • minor tests/workflows/test_configuration_doc.py:44: Section lookup no longer requires a level-2 heading

Policy floor: high (touches high-risk paths: .github/workflows/pull-request.yml, scripts/literal-slices.py, tests/repro/test_issue_549.py, tests/repro/test_issue_556.py, tests/repro/test_issue_575.py …). Reviewed 4cc51db (since 05cbd9d; review 2 of 5 automatic). Author trusted for auto-merge: false.

🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.15, 8 turns

@soydiloreto

Copy link
Copy Markdown
Member

Two changes before this merges, both from the review's minors:

  1. .github/workflows/pull-request.yml, the comment above literal-slices.py --test (lines 154-156) still says KNOWN "lists the files that did before the check, and only shrinks". KNOWN is now empty: say that no test may cut text between literals, and that only a reasoned exception goes back on the list.
  2. tests/workflows/test_literal_slices.py, test_c3_the_worst_wording_tests_of_tests_workflows_slice_no_more: put back a positive control in place of the removed assert counts. Since tests/workflows no longer slices, it cannot be the real tree: run --all on a tmp_path file with one known slice in the same call and assert it is reported, so the test fails if --all silently reads nothing.

Then mark it ready.

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

@soydiloreto
soydiloreto marked this pull request as draft November 19, 2026 13:33
@dilux-bot
dilux-bot Bot marked this pull request as ready for review November 19, 2026 13:38
dilux-bot Bot added 2 commits November 19, 2026 13:38
The scripts job's comment above literal-slices.py now says KNOWN is empty and that only a reasoned exception goes back on it. The test that the worst wording tests slice no more runs --all on a tmp_path file with one known slice in the same call and asserts it is reported, so it fails if --all silently reads nothing.

Assisted-by: Claude Opus 5.5 (Anthropic)
@soydiloreto
soydiloreto merged commit 9cdf346 into main Nov 19, 2026
38 checks passed
@soydiloreto
soydiloreto deleted the chore/873-the-23-tests-left-in branch November 19, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autopilot Opened by the autopilot from an accepted issue complexity:high Set by the Claude review risk:high Set by the Claude review type:test The kind of change, read from the diff by the Claude review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: the 23 tests left in KNOWN stop cutting text

1 participant