Skip to content

feat: test first needs an assertion and tests check behaviour - #846

Merged
soydiloreto merged 18 commits into
mainfrom
fix/845-test-first-needs-an-assertion
Nov 19, 2026
Merged

soydiloreto merged 18 commits into
mainfrom
fix/845-test-first-needs-an-assertion

Conversation

@dilux-bot

@dilux-bot dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

📝 What changes

C1. In scripts/test-first.py verdict, a feat counts as failing first only when it is reproduced at the base, the same as a fix. That means an assertion failed. A broken result (an ImportError, a collection error, a missing file, no report) no longer counts. A new test for a new script therefore asserts on what it runs, for example a subprocess's exit code and output, so that the base fails an assertion and does not just error. The feat message says this. The self-test case feat broken → pass is flipped, and a case is added for it. rules/test-first.md, docs/testing-and-quality.md and the test-first: comment in .github/review-policy.yml say the same.

C2. A new script, scripts/literal-slices.py, with its own --test, reads test files with Python's ast module. It reports path:line (test function) for each place where a test cuts text between literals:

  • a slice whose bound is .index/.find/.rindex/.rfind("literal"), or a name assigned from one of those calls in the same function;
  • a fixed index into .split/.rsplit/.partition/.rpartition("literal").

The scripts job in .github/workflows/pull-request.yml runs it with --test and over tests/workflows tests/repro, and exits 1 naming each hit. The files that still slice (6 in tests/repro, issues 549, 575, 696, …, and 17 in tests/workflows left for C3's follow-up) are in the script's KNOWN, each with its number of slices, which can only shrink: a listed file with more slices than its count fails, naming each slice, so a new slice in a listed file is caught; one with fewer fails too, so its count is lowered or the file removed in the change that fixes it. A path given that does not exist exits 2 rather than scanning nothing. In .github/review-policy.yml, the reproduction guide gets one bullet: write the stubs and the subprocess call in the test file itself, since the runs-step check (#841) reads only that file's own process calls, so a helper such as tests/workflows/steps.py would read as "the test runs no step"; and never cut the workflow's text between two literals. The Test first comment there says a feature's test, like a fix's, fails on an assertion at the base.

C3, in part. 19 files of tests/workflows no longer cut text between literals, worst first (maintainer_bug_wiring, policy_on_receipts, autopilot_main_merge, settled_finding_rerun, …): each slice is replaced by a behaviour test or deleted. A behaviour test does one of two things:

  • finds the step by name or id through YAML, runs its run: in bash with stub gh and helper commands on PATH that record their calls, and asserts on the calls and on $GITHUB_OUTPUT;
  • or, when the logic is in a script under scripts/, calls that script.

The shared code goes in a helper, tests/workflows/steps.py, which is not a test file. A slice that only checked a sentence's wording is deleted. The rest of C3 (17 files of tests/workflows and 6 of tests/repro, the 23 still in KNOWN) goes to a follow-up Task under #845: this pull request already changes 20 tests, the most Test first runs (MAX_TESTS). C3 is met when KNOWN is empty.

💡 Why

Closes #845. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.

🎯 Acceptance criteria

  • C1 Given a feat pull request whose only new test errors at the base (ImportError), when test first checks it, then it does not count as failing first.
    Test: tests/workflows/test_test_first.py
  • C2 Given a new test that slices a workflow file between two literals, when the scripts job runs, then it fails naming the test.
    Test: tests/workflows/test_literal_slices.py
  • C3 Given tests/workflows, when the wording tests are counted as in this issue, then the ones that slice workflow text between literals are gone, each replaced by a behaviour test or deleted.
    Test: tests/workflows/test_literal_slices.py

🧭 Decisions

  • 2026-11-19: C3 follows the Spec's recommended option A: this pull request converts the tests/workflows files with the most literal slices, keeping within the 20 changed tests the Test first check runs (MAX_TESTS is not raised), and the files left stay in literal-slices.py's KNOWN list, which can only shrink, for the follow-up Task the Spec names.
  • 2026-11-19: Each test that uses tests/workflows/steps.py falls back, when the helper is missing (the Test first check's base tree), to stand-ins whose calls fail an assertion naming the helper, so the base reaches an assertion as the maintainer asked; and test_test_first.py's policy.py self-test check accepts outside a git checkout only the one failure that needs git ls-files, because the check runs it in a t
  • 2026-11-19: The copied missing-helper fallback is gone: each of the eleven tests that use tests/workflows/steps.py now falls back with one unittest.mock line whose calls fail an assertion naming the helper. It is not in a conftest.py they import: the Test first check runs each test at the base tree with only that test file in place, where a new conftest.py is missing just as steps.py is, so importing it would
  • 2026-11-19: C3 is not met in this pull request, so it should be unticked; the autopilot has no command to untick a box. The branch already adds or changes exactly 20 tests, the most the required Test first check runs (MAX_TESTS): converting any of the 23 files still in literal-slices.py's KNOWN changes it too, and the check then refuses the pull request ('split it'). The files left: tests/workflows test_autop
  • 2026-11-19: C3 moves to a follow-up Task under #​845, as the maintainer answered: this pull request merges with C1, C2 and the 19 tests/workflows files already converted, and C3's box should be unticked. Why: the required Test first check runs at most 20 changed tests (MAX_TESTS in scripts/test-first.py), and this branch already adds or changes exactly 20, so converting one more file makes the check refuse the
  • 2026-11-19: What changes, made to match the code (the autopilot has no command to edit it or untick a box, so a maintainer applies this): C2 also reports a f​ixed index into .rsplit("literal"), and KNOWN holds 23 files, 6 in tests/repro and 17 in tests/workflows. C3 is partly done here: 19 tests/workflows files no longer slice (maintainer_bug_wiring, policy_on_receipts, autopilot_main_merge, settled_finding_re

🙌 Credits

🧪 How I tested it

  • Ran the real Test first runner (scripts/test-first.py run through scripts/repro-run.sh run-ref, unconfined, with pytest 9.1.1 and PyYAML 6.0.3) over the 20 changed tests, with the base 5ffc03b and this head. It passes. 13 tests fail on an assertion at the base and pass at the head. 7 pass at both. None is broken.
  • Before the fix, the same run failed in two ways. The 11 tests that use tests/workflows/steps.py were broken at the base (ModuleNotFoundError: No module named 'steps'). test_test_first.py was reproduced at the head, because policy.py --test cannot run git ls-files in the check's tree, which has no .git.
  • Wrote each criterion's test first. With scripts/literal-slices.py absent, the C2 and C3 tests in tests/workflows/test_literal_slices.py fail on assertions. The C1 tests in tests/workflows/test_test_first.py fail on the old verdict.
  • Whole tests/ suite with pytest: 620 passed, 1 skipped. These self-tests pass: test-first.py, policy.py, literal-slices.py, repro-ways.py and rules.py --test. literal-slices.py tests/workflows tests/repro passes with 23 files left in KNOWN.
  • The local review's checks (conventions, risk floor, brief) pass. The repository has no make pre-pr.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

The draft pull request opens before any code: the Spec as its checklist.
@dilux-bot dilux-bot Bot added the autopilot Opened by the autopilot from an accepted issue label Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

The pull request's board

State
Waits on nobody

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

Test first counted a feat as failing first when its new test only broke at the base (an ImportError, a missing file, no report), so a test that never asserted anything passed the check. A feat now counts only when it is reproduced at the base, the same as a fix, and the message says to assert on what the test runs. The self-test's feat broken case is flipped, and the rule, the testing document and the review policy's comment say the same.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot has a question before it goes on. C1 is done and pushed (a feat's test must now fail an assertion at the base). C2 and C3 wait on the choice the Spec leaves to you: how to fit C3 within Test first's 20-test limit (MAX_TESTS). Today 33 files in tests/workflows slice text between literals (164 slices; worst: settled_finding_rerun 16, maintainer_bug_wiring 14, issue_805_labels 12, policy_on_receipts 12, conventions_fix_round 10), plus 6 in tests/repro. Options: (1) Option A, split: this PR does C2 with the 6 tests/repro files and the tests/workflows files not yet converted on a list in scripts/literal-slices.py that can only shrink, and converts the worst files here, keeping within 20 changed tests; the rest go to a Task sub-issue as test: PRs. C3 is then met only when that list is empty, so its box here would be ticked for the files converted, not all 33: say whether C3 should be reworded for this PR or left unticked until the sub-issue closes. (2) Option B, one PR: convert all 33 files here; Test first refuses it (more than 20 tests) and a person merges it over that check. (3) Option C: raise MAX_TESTS for this repository; it widens a limit the PRD does not ask for. I recommend 1: no limit is widened and every PR passes its checks; please also say how C3's box is meant to read under it.

Answer here in your own words; the autopilot goes on with your answer.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: answer the autopilot's question #846

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@soydiloreto

Copy link
Copy Markdown
Member

1 (Option A). C3 reads, for this pull request: the list in scripts/literal-slices.py exists, can only shrink, and the files converted here are off it. Open the Task sub-issue that empties the list, and tick C3 on that basis.

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot ended without a result. The agent finished without marking the pull request ready or saying why it stopped. Nothing was marked ready and the agent left no stop comment, so a person takes it from here (needs-human): a maintainer's answer here, a hint in words, brings it back, or they take it over. What it did is in the run: its summary, and the full transcript as an artifact.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot added needs-human The autopilot handed this over to a person and removed needs-human The autopilot handed this over to a person labels Nov 19, 2026
@dilux-bot
dilux-bot Bot marked this pull request as ready for review November 19, 2026 03:39
@dilux-bot
dilux-bot Bot requested a review from soydiloreto as a code owner November 19, 2026 03:39
dilux-bot Bot added 4 commits November 19, 2026 03:39
scripts/literal-slices.py reads the test files with Python's ast module and reports path:line (function) for each slice bounded by .index/.find/.rindex/.rfind on a literal, or by a name assigned from one, and for each fixed index into .split/.partition/.rpartition on a literal. The scripts job runs its self-test and runs it over tests/workflows and tests/repro. The files that sliced before the check are listed in KNOWN, which can only shrink: a listed file that no longer slices fails the check too.

The guidance in .github/review-policy.yml for a workflow's test now says to run the step instead of asserting on its text, and docs/testing-and-quality.md says the same.

Assisted-by: Claude Opus 5.5 (Anthropic)
The 19 tests/workflows files that cut a workflow's text between literals most often now test what the steps do: tests/workflows/steps.py finds a step through YAML, runs its run: in bash with stub gh and helper commands on PATH that record their calls, and returns the calls and $GITHUB_OUTPUT; where the logic is in a script under scripts/, the test calls that script. A slice that only checked a sentence's wording is deleted.

The 17 files left in tests/workflows each slice once; they stay in literal-slices.py's KNOWN list, which can only shrink, for the follow-up, so this pull request changes no more than the 20 tests the Test first check runs. literal-slices.py --all reports every slice, listed or not, and the C3 test counts with it.

Assisted-by: Claude Opus 5.5 (Anthropic)
The workflow way's guide said its test never runs the workflow, while the line below now tells it to run a step's run: with stubs. It says both now: never the workflow on GitHub, one step's run: on its own.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:medium Set by the Claude review type:fix The kind of change, read from the diff by the Claude review labels Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Claude review · risk high · complexity high · type feat

Nothing that must be fixed · 3 minor

Incremental review: the new commits reworded the repro guide so it no longer sends agents to tests/workflows/steps.py, which the runs-step gate rejects. A test covers it, so that thread is resolved. KNOWN in literal-slices.py counts slices per file, and that earlier thread stays settled. I read the diff of the scripts, policy, docs, steps.py and the self-tests. I only skimmed the 18 converted workflow tests, so that part is unchecked. C1 and C2 are done: verdict() counts a feat as failing first only when it is reproduced, and the docs, rules and policy comment say the same. C3 is not met, and the description's own box is unticked: 23 files remain in KNOWN, deferred to a follow-up under #845 because of MAX_TESTS. Minor: (1) the except ModuleNotFoundError fallback for steps.py also hides a missing yaml; (2) test_c6 in test_test_first.py matches the exact text of one policy.py FAIL line; (3) the description says 19 files were converted but the diff shows 18 besides the two new-check tests; recount. Not a new capability, so no version:major suggestion.

  • minor tests/workflows/test_autopilot_main_merge.py:1023: except ModuleNotFoundError also hides a missing yaml inside steps.py
  • minor tests/workflows/test_test_first.py:2991: test_c6 matches the exact text of one policy.py FAIL line
  • minor scripts/literal-slices.py:590: 23 files still slice (C3 not met); description says 19 converted, diff shows 18

Policy floor: high (touches high-risk paths: .github/review-policy.yml, .github/workflows/pull-request.yml, docs/testing-and-quality.md, rules/test-first.md, scripts/literal-slices.py …). Reviewed 25074d5 (whole pull request; review 7 of 5 automatic). Author trusted for auto-merge: false.

🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.36, 13 turns

@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: a check of the repository's own failed, which no fix round runs again: read it and decide (run it again, or take it over) #846

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot added complexity:high Set by the Claude review type:feat The kind of change, read from the diff by the Claude review and removed complexity:medium Set by the Claude review type:fix The kind of change, read from the diff by the Claude review labels Nov 19, 2026
@dilux-bot dilux-bot Bot changed the title fix: test first needs an assertion and tests check behaviour feat: test first needs an assertion and tests check behaviour Nov 19, 2026
@soydiloreto

Copy link
Copy Markdown
Member

Test first fails because the new tests import a module named steps that the base does not have, so at the base they stop with ModuleNotFoundError: No module named 'steps', which is not an assertion (the rule this pull request itself enforces). Put the step-running helper where the base already has it, or have each test load it so that at the base the test reaches its assertion and fails there. Then check Test first passes on your branch.

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

@soydiloreto

Copy link
Copy Markdown
Member

1: C3 moves to a follow-up, and this pull request merges with C1, C2 and the files already converted. In the description, untick C3 and say in the Decisions why (the 20-test limit of Test first, MAX_TESTS) and which 23 files stay in KNOWN; I open the follow-up Task for them under #845. Also make "What changes" match the code, since the review says it does not.

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot stopped here. The code is done and checked: C1, C2 and 19 converted tests/workflows files. The whole tests/ suite passes (637 passed, 1 skipped). The self-tests of test-first, literal-slices, policy, repro-ways and rules pass. literal-slices.py passes with 23 files in KNOWN, and the local review's checks pass. The repository has no make pre-pr. I stopped short of ready because the answer asks for two edits to the description, and the autopilot has no command for either: unticking C3, and rewriting What changes. Marking it ready with C3 ticked would leave the description untrue. The Decisions now record why C3 moves to the follow-up (MAX_TESTS, 20 changed tests, already reached), the 23 files that stay in KNOWN, and the corrected What changes text. For a maintainer: untick C3 and replace What changes with the logged text in #846, then mark it ready. Open the follow-up Task under #845.

A person takes it from here (needs-human): a maintainer's answer here, a hint in words, brings it back, or they take it over.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot added the needs-human The autopilot handed this over to a person label Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: a person takes it from here (needs-human): read why and decide #846

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@soydiloreto soydiloreto removed the needs-human The autopilot handed this over to a person label Nov 19, 2026
@soydiloreto
soydiloreto marked this pull request as ready for review November 19, 2026 08:34
@soydiloreto soydiloreto added the review:full Ask the Claude review for a full pass on the next push label Nov 19, 2026
…eds-an-assertion

# Conflicts:
#	.github/review-policy.yml
@dilux-bot dilux-bot Bot removed the review:full Ask the Claude review for a full pass on the next push label Nov 19, 2026
Comment thread scripts/literal-slices.py Resolved
KNOWN in scripts/literal-slices.py held whole files, so a new slicing test added to a listed file passed the scripts job. It now maps each file to its number of slices: more than recorded fails naming every slice in the file, fewer fails asking for the count to be lowered. A path given that does not exist now fails instead of scanning nothing, and the docstring names .rsplit. The self-test and tests/workflows/test_literal_slices.py cover a slice appended to a listed file, a stale count, a missing path, and that KNOWN matches today's counts.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot dilux-bot Bot added complexity:medium Set by the Claude review and removed complexity:high Set by the Claude review labels Nov 19, 2026
@soydiloreto soydiloreto added the review:full Ask the Claude review for a full pass on the next push label Nov 19, 2026
@dilux-bot dilux-bot Bot added complexity:high Set by the Claude review and removed review:full Ask the Claude review for a full pass on the next push complexity:medium Set by the Claude review labels Nov 19, 2026
Comment thread .github/review-policy.yml Resolved
dilux-bot Bot added 2 commits November 19, 2026 10:45
The python repro way has runs-step, whose gate reads only the test file's own process calls, so the bullet offering tests/workflows/steps.py sent reproductions to a helper the gate rejects as "the test runs no step". The bullet now says to write the stubs and the subprocess call in the test file itself, and a test checks that no runs-step guide names steps.py while each still forbids slicing the text.

Assisted-by: Claude Opus 5.5 (Anthropic)
@soydiloreto soydiloreto added the review:full Ask the Claude review for a full pass on the next push label Nov 19, 2026
@dilux-bot dilux-bot Bot removed the review:full Ask the Claude review for a full pass on the next push label Nov 19, 2026
@soydiloreto
soydiloreto merged commit b533c54 into main Nov 19, 2026
51 of 52 checks passed
@soydiloreto
soydiloreto deleted the fix/845-test-first-needs-an-assertion branch November 19, 2026 11:06
dilux-bot Bot added a commit that referenced this pull request Nov 19, 2026
The 23 test files #846 left in KNOWN (6 in tests/repro, 17 in tests/workflows) no longer cut text between literals. A step's run: is run with stubs and the test asserts on the calls, the exit code and $GITHUB_OUTPUT; an if: is evaluated with steps.evaluate; wiring is compared as YAML fields; a doc, template or prompt is read as structure, by its headings (steps.sections) or numbered steps (steps.numbered). The tests/repro files keep their own stubs and subprocess call, as the workflow way's runs-step guide asks. KNOWN in scripts/literal-slices.py is empty, and test_c3 no longer expects tests/workflows to hold a slice.

Assisted-by: Claude Opus 5.5 (Anthropic)
dilux-bot Bot added a commit that referenced this pull request Nov 19, 2026
A test runs literal-slices.py --all on each of the 23 files #846 left in KNOWN and asserts that every one still exists and none is reported.

Assisted-by: Claude Opus 5.5 (Anthropic)
soydiloreto pushed a commit that referenced this pull request Nov 19, 2026
## 📝 What changes

Convert each of the 23 listed tests so it no longer cuts text between
literals, file by file, then empty `KNOWN` in scripts/literal-slices.py
(its type stays `dict`, so `KNOWN == slices_by_file(...)` in
tests/workflows/test_literal_slices.py still holds, with both sides
`{}`). No workflow, script, doc or template that the tests cover
changes. The pull request is typed `test:` because it changes only tests
and the list. That keeps it outside test-first's `fix`/`feat` limit of
20 tests (scripts/test-first.py `MAX_TESTS`), which made #846 leave
these files out. It still closes #873.

Each slice is replaced by whichever of these fits what the test asserts:
- **A step's `run:` (most tests/workflows files, the repro files 549,
556, 575, 577, 696 and 774):** find the step by `id`/`name` with
`tests/workflows/steps.py` (`step`, `run`) and run it in bash. Stub
`gh`, `git` and the helper scripts on PATH, and set the env that picks
the branch under test (for example `LANE=bugfix-reproduced` for #549, an
empty first `accepted` for #696). Then assert on the calls recorded, the
exit code and `$GITHUB_OUTPUT`, not on the text of the branch. The
tests/repro files whose way has `runs-step` (review-policy.yml
`repro.ways`) keep their own `subprocess` call, as the repro guide
requires, and do not import steps.py.
- **A step's `if:` or a job's wiring (`fix_gate_script`, `gate`, the
required checks, `who-decides`, triage's non-write users):** read it
through YAML and evaluate the expression with `steps.evaluate` on the
events that matter, or compare structured fields (`needs`, `with`,
`permissions`, step order by index in `steps`).
- **A prompt or a doc (commit cadence, pre-pr, the configuration page,
who signs what, the PR template's markers):** read the section as
structure, not as a literal-to-literal cut. Markdown is split into
`{heading: body}` by its headings, a template into its markers, and a
prompt into its numbered steps. The test then asserts on that structure,
for example the order of the steps or every setting appearing in its
section. If the logic lives in a script under scripts/ (for example
`autopilot.py cap-comment`, `settled` or `issue-schema.py`), the test
calls the script instead.
- If a slice checked only wording that no behaviour stands behind, the
assertion is removed and the pull request says so, file by file (C2). No
test function is deleted without a replacement behaviour test or that
stated reason.

As each file stops slicing, its entry leaves `KNOWN`.
`literal-slices.py` already fails a file that has fewer slices than its
count, so the list and the files move together. The KNOWN comment
changes to say the list is empty and only a reasoned exception may be
added back. The `tests/workflows tests/repro` run in the scripts job
then reports 0 files listed.

## 💡 Why

Closes #873. The autopilot works it here in the open: one commit per
criterion, each box ticked once its commit is pushed.

## 🎯 Acceptance criteria

<!-- dx-checklist -->
- [x] **C1** Given scripts/literal-slices.py, when its `KNOWN` is read,
then it is empty, and `python3 scripts/literal-slices.py` over
tests/workflows and tests/repro passes.
Test: tests/workflows/test_literal_slices.py (new
`test_c1_known_is_empty_and_the_tests_pass_it`: asserts `load().KNOWN ==
{}` and that `literal-slices.py tests/workflows tests/repro` exits 0
reporting 0 files listed. It fails at the base, where KNOWN holds 23
files.)
- [x] **C2** Given each of the 23 files, when the whole tests/ suite
runs, then it passes, and no test was deleted without a behaviour test
or a stated reason in the pull request (a slice that checked only
wording).
Test: tests/workflows/test_literal_slices.py (new
`test_c2_none_of_the_23_files_slices`: runs `literal-slices.py --all` on
each of the 23 paths and asserts that none is reported and every path
still exists. It fails at the base. The converted files themselves pass
in the full `pytest tests/` run.)
<!-- /dx-checklist -->

## 🧭 Decisions

- 2026-11-19: Added two pure readers to tests/workflows/steps.py,
sections() (markdown by heading) and numbered() (numbered list items),
shared by the doc, template and prompt tests, as the Spec allows.
- 2026-11-19: Dropped test_c3's guard that --all finds some slice in
tests/workflows: with every file converted none is left; the tmp_path
tests of the same file still prove --all detects a slice.
- 2026-11-19: Credits list nobody: the issue's author, @​soydiloreto, is
the code owner who accepts and merges it, and AGENTS.md rule 10 says
that maintainer is never a co-author (an earlier run listed them; they
can add it back with /coauthor).

## 🙌 Credits

<!-- dx-credits -->
<!-- /dx-credits -->

## 🧪 How I tested it

<!-- dx-tested -->
## How I tested it

- The C1 and C2 tests in tests/workflows/test_literal_slices.py both
failed first, at the base, where KNOWN still listed 23 files and
`literal-slices.py --all` reported them. Both pass at HEAD.
- `python3 scripts/literal-slices.py tests/workflows tests/repro` prints
"0 files still listed in KNOWN", and `python3 scripts/literal-slices.py
--test` passes.
- Running the whole suite, `python3 -m pytest -q tests`, gives 681
passed and 1 skipped.
- For each test that now runs a step, I broke a scratch copy of that
step's `run:` and the test failed. I didn't keep those copies.
- `test_c3_the_worst_wording_tests_of_tests_workflows_slice_no_more` now
runs `--all` on tests/workflows and on a tmp_path file with one known
slice, in the same call, and checks that the slice is reported. If
`--all` reads nothing, the test fails.
- The local review's checks (conventions, policy, brief) pass. This
repository has no `make pre-pr`.

## Removed, with the reason (C2)

- In tests/workflows/test_required_checks.py, I removed the assertion
that the comment above `on:` in pull-request.yml names "required checks"
and the two check names. It checked only wording. A behaviour check
replaces it: the two check names are worked out from the jobs' YAML
(`name`, and `test-first.yml`'s job) and must equal the required list.
- No other test function or assertion was removed. Each slice was
replaced by one of these: a run of the step with stubs, `steps.evaluate`
on its `if:`, checks on YAML fields, or reading the doc as structure
(`steps.sections` and `steps.numbered`, new in
tests/workflows/steps.py).

## How I tested it

- The C1 and C2 tests in tests/workflows/test_literal_slices.py both
failed first, at the base, where KNOWN still listed 23 files and
`literal-slices.py --all` reported them. Both pass at HEAD.
- `python3 scripts/literal-slices.py` prints "0 files still listed in
KNOWN", and `python3 scripts/literal-slices.py --test` passes.
- Running the whole suite, `python3 -m pytest -p no:cacheprovider
--rootdir=. -q tests`, gives 681 passed and 1 skipped.
- For each test that now runs a step, I broke a scratch copy of that
step's `run:` and the test failed. I didn't keep those copies.
- The local review's checks (conventions, policy, brief) pass. This
repository has no `make pre-pr`.

## Removed, with the reason (C2)

- In tests/workflows/test_required_checks.py, I removed the assertion
that the comment above `on:` in pull-request.yml names "required checks"
and the two check names. It checked only wording. It is replaced by a
behaviour check: the two check names are worked out from the jobs' YAML
(`name`, and `test-first.yml`'s job) and must equal the required list.
- In tests/workflows/test_literal_slices.py,
`test_c3_the_worst_wording_tests_of_tests_workflows_slice_no_more` no
longer requires at least one slice to be found in tests/workflows,
because none is left. The tests in the same file that use tmp_path still
show that `--all` finds a slice.
- No other test function or assertion was removed. Each slice was
replaced by a run of the step with stubs, `steps.evaluate` on its `if:`,
YAML field checks, or reading the doc as structure (`steps.sections` and
`steps.numbered`, new in tests/workflows/steps.py).


🤖 AI-generated · Claude Opus 5.5 (Anthropic)

---------

Co-authored-by: dilux-bot[bot] <333187390+dilux-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autopilot Opened by the autopilot from an accepted issue complexity:high Set by the Claude review risk:high Set by the Claude review type:feat The kind of change, read from the diff by the Claude review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: test first needs an assertion and tests check behaviour

1 participant