Repository navigation
feat: test first needs an assertion and tests check behaviour - #846
Conversation
The draft pull request opens before any code: the Spec as its checklist.
|
The pull request's board
🤖 AI-generated · Claude Opus 5.5 (Anthropic) |
Test first counted a feat as failing first when its new test only broke at the base (an ImportError, a missing file, no report), so a test that never asserted anything passed the check. A feat now counts only when it is reproduced at the base, the same as a fix, and the message says to assert on what the test runs. The self-test's feat broken case is flipped, and the rule, the testing document and the review policy's comment say the same. Assisted-by: Claude Opus 5.5 (Anthropic)
|
The autopilot has a question before it goes on. C1 is done and pushed (a feat's test must now fail an assertion at the base). C2 and C3 wait on the choice the Spec leaves to you: how to fit C3 within Test first's 20-test limit (MAX_TESTS). Today 33 files in tests/workflows slice text between literals (164 slices; worst: settled_finding_rerun 16, maintainer_bug_wiring 14, issue_805_labels 12, policy_on_receipts 12, conventions_fix_round 10), plus 6 in tests/repro. Options: (1) Option A, split: this PR does C2 with the 6 tests/repro files and the tests/workflows files not yet converted on a list in scripts/literal-slices.py that can only shrink, and converts the worst files here, keeping within 20 changed tests; the rest go to a Task sub-issue as test: PRs. C3 is then met only when that list is empty, so its box here would be ticked for the files converted, not all 33: say whether C3 should be reworded for this PR or left unticked until the sub-issue closes. (2) Option B, one PR: convert all 33 files here; Test first refuses it (more than 20 tests) and a person merges it over that check. (3) Option C: raise MAX_TESTS for this repository; it widens a limit the PRD does not ask for. I recommend 1: no limit is widened and every PR passes its checks; please also say how C3's box is meant to read under it. Answer here in your own words; the autopilot goes on with your answer. 🤖 AI-generated · Claude Opus 5.5 (Anthropic) |
|
@soydiloreto: answer the autopilot's question #846 🤖 AI-generated · Claude Opus 5.5 (Anthropic) |
|
1 (Option A). C3 reads, for this pull request: the list in scripts/literal-slices.py exists, can only shrink, and the files converted here are off it. Open the Task sub-issue that empties the list, and tick C3 on that basis. 🤖 AI-assisted · Claude Opus 5.5 (Anthropic) |
|
The autopilot ended without a result. The agent finished without marking the pull request ready or saying why it stopped. Nothing was marked ready and the agent left no stop comment, so a person takes it from here ( 🤖 AI-generated · Claude Opus 5.5 (Anthropic) |
scripts/literal-slices.py reads the test files with Python's ast module and reports path:line (function) for each slice bounded by .index/.find/.rindex/.rfind on a literal, or by a name assigned from one, and for each fixed index into .split/.partition/.rpartition on a literal. The scripts job runs its self-test and runs it over tests/workflows and tests/repro. The files that sliced before the check are listed in KNOWN, which can only shrink: a listed file that no longer slices fails the check too. The guidance in .github/review-policy.yml for a workflow's test now says to run the step instead of asserting on its text, and docs/testing-and-quality.md says the same. Assisted-by: Claude Opus 5.5 (Anthropic)
The 19 tests/workflows files that cut a workflow's text between literals most often now test what the steps do: tests/workflows/steps.py finds a step through YAML, runs its run: in bash with stub gh and helper commands on PATH that record their calls, and returns the calls and $GITHUB_OUTPUT; where the logic is in a script under scripts/, the test calls that script. A slice that only checked a sentence's wording is deleted. The 17 files left in tests/workflows each slice once; they stay in literal-slices.py's KNOWN list, which can only shrink, for the follow-up, so this pull request changes no more than the 20 tests the Test first check runs. literal-slices.py --all reports every slice, listed or not, and the C3 test counts with it. Assisted-by: Claude Opus 5.5 (Anthropic)
The workflow way's guide said its test never runs the workflow, while the line below now tells it to run a step's run: with stubs. It says both now: never the workflow on GitHub, one step's run: on its own. Assisted-by: Claude Opus 5.5 (Anthropic)
Claude review · risk high · complexity high · type featNothing that must be fixed · 3 minor Incremental review: the new commits reworded the repro guide so it no longer sends agents to tests/workflows/steps.py, which the runs-step gate rejects. A test covers it, so that thread is resolved. KNOWN in literal-slices.py counts slices per file, and that earlier thread stays settled. I read the diff of the scripts, policy, docs, steps.py and the self-tests. I only skimmed the 18 converted workflow tests, so that part is unchecked. C1 and C2 are done: verdict() counts a feat as failing first only when it is
Policy floor: high (touches high-risk paths: .github/review-policy.yml, .github/workflows/pull-request.yml, docs/testing-and-quality.md, rules/test-first.md, scripts/literal-slices.py …). Reviewed 25074d5 (whole pull request; review 7 of 5 automatic). Author trusted for auto-merge: false. 🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.36, 13 turns |
|
@soydiloreto: a check of the repository's own failed, which no fix round runs again: read it and decide (run it again, or take it over) #846 🤖 AI-generated · Claude Opus 5.5 (Anthropic) |
|
Test first fails because the new tests import a module named 🤖 AI-assisted · Claude Opus 5.5 (Anthropic) |
|
1: C3 moves to a follow-up, and this pull request merges with C1, C2 and the files already converted. In the description, untick C3 and say in the Decisions why (the 20-test limit of Test first, MAX_TESTS) and which 23 files stay in KNOWN; I open the follow-up Task for them under #845. Also make "What changes" match the code, since the review says it does not. 🤖 AI-assisted · Claude Opus 5.5 (Anthropic) |
|
The autopilot stopped here. The code is done and checked: C1, C2 and 19 converted tests/workflows files. The whole tests/ suite passes (637 passed, 1 skipped). The self-tests of test-first, literal-slices, policy, repro-ways and rules pass. literal-slices.py passes with 23 files in KNOWN, and the local review's checks pass. The repository has no make pre-pr. I stopped short of ready because the answer asks for two edits to the description, and the autopilot has no command for either: unticking C3, and rewriting What changes. Marking it ready with C3 ticked would leave the description untrue. The Decisions now record why C3 moves to the follow-up (MAX_TESTS, 20 changed tests, already reached), the 23 files that stay in KNOWN, and the corrected What changes text. For a maintainer: untick C3 and replace What changes with the logged text in #846, then mark it ready. Open the follow-up Task under #845. A person takes it from here ( 🤖 AI-generated · Claude Opus 5.5 (Anthropic) |
|
@soydiloreto: a person takes it from here ( 🤖 AI-generated · Claude Opus 5.5 (Anthropic) |
…eds-an-assertion # Conflicts: # .github/review-policy.yml
KNOWN in scripts/literal-slices.py held whole files, so a new slicing test added to a listed file passed the scripts job. It now maps each file to its number of slices: more than recorded fails naming every slice in the file, fewer fails asking for the count to be lowered. A path given that does not exist now fails instead of scanning nothing, and the docstring names .rsplit. The self-test and tests/workflows/test_literal_slices.py cover a slice appended to a listed file, a stale count, a missing path, and that KNOWN matches today's counts. Assisted-by: Claude Opus 5.5 (Anthropic)
The python repro way has runs-step, whose gate reads only the test file's own process calls, so the bullet offering tests/workflows/steps.py sent reproductions to a helper the gate rejects as "the test runs no step". The bullet now says to write the stubs and the subprocess call in the test file itself, and a test checks that no runs-step guide names steps.py while each still forbids slicing the text. Assisted-by: Claude Opus 5.5 (Anthropic)
The 23 test files #846 left in KNOWN (6 in tests/repro, 17 in tests/workflows) no longer cut text between literals. A step's run: is run with stubs and the test asserts on the calls, the exit code and $GITHUB_OUTPUT; an if: is evaluated with steps.evaluate; wiring is compared as YAML fields; a doc, template or prompt is read as structure, by its headings (steps.sections) or numbered steps (steps.numbered). The tests/repro files keep their own stubs and subprocess call, as the workflow way's runs-step guide asks. KNOWN in scripts/literal-slices.py is empty, and test_c3 no longer expects tests/workflows to hold a slice. Assisted-by: Claude Opus 5.5 (Anthropic)
A test runs literal-slices.py --all on each of the 23 files #846 left in KNOWN and asserts that every one still exists and none is reported. Assisted-by: Claude Opus 5.5 (Anthropic)
## 📝 What changes
Convert each of the 23 listed tests so it no longer cuts text between
literals, file by file, then empty `KNOWN` in scripts/literal-slices.py
(its type stays `dict`, so `KNOWN == slices_by_file(...)` in
tests/workflows/test_literal_slices.py still holds, with both sides
`{}`). No workflow, script, doc or template that the tests cover
changes. The pull request is typed `test:` because it changes only tests
and the list. That keeps it outside test-first's `fix`/`feat` limit of
20 tests (scripts/test-first.py `MAX_TESTS`), which made #846 leave
these files out. It still closes #873.
Each slice is replaced by whichever of these fits what the test asserts:
- **A step's `run:` (most tests/workflows files, the repro files 549,
556, 575, 577, 696 and 774):** find the step by `id`/`name` with
`tests/workflows/steps.py` (`step`, `run`) and run it in bash. Stub
`gh`, `git` and the helper scripts on PATH, and set the env that picks
the branch under test (for example `LANE=bugfix-reproduced` for #549, an
empty first `accepted` for #696). Then assert on the calls recorded, the
exit code and `$GITHUB_OUTPUT`, not on the text of the branch. The
tests/repro files whose way has `runs-step` (review-policy.yml
`repro.ways`) keep their own `subprocess` call, as the repro guide
requires, and do not import steps.py.
- **A step's `if:` or a job's wiring (`fix_gate_script`, `gate`, the
required checks, `who-decides`, triage's non-write users):** read it
through YAML and evaluate the expression with `steps.evaluate` on the
events that matter, or compare structured fields (`needs`, `with`,
`permissions`, step order by index in `steps`).
- **A prompt or a doc (commit cadence, pre-pr, the configuration page,
who signs what, the PR template's markers):** read the section as
structure, not as a literal-to-literal cut. Markdown is split into
`{heading: body}` by its headings, a template into its markers, and a
prompt into its numbered steps. The test then asserts on that structure,
for example the order of the steps or every setting appearing in its
section. If the logic lives in a script under scripts/ (for example
`autopilot.py cap-comment`, `settled` or `issue-schema.py`), the test
calls the script instead.
- If a slice checked only wording that no behaviour stands behind, the
assertion is removed and the pull request says so, file by file (C2). No
test function is deleted without a replacement behaviour test or that
stated reason.
As each file stops slicing, its entry leaves `KNOWN`.
`literal-slices.py` already fails a file that has fewer slices than its
count, so the list and the files move together. The KNOWN comment
changes to say the list is empty and only a reasoned exception may be
added back. The `tests/workflows tests/repro` run in the scripts job
then reports 0 files listed.
## 💡 Why
Closes #873. The autopilot works it here in the open: one commit per
criterion, each box ticked once its commit is pushed.
## 🎯 Acceptance criteria
<!-- dx-checklist -->
- [x] **C1** Given scripts/literal-slices.py, when its `KNOWN` is read,
then it is empty, and `python3 scripts/literal-slices.py` over
tests/workflows and tests/repro passes.
Test: tests/workflows/test_literal_slices.py (new
`test_c1_known_is_empty_and_the_tests_pass_it`: asserts `load().KNOWN ==
{}` and that `literal-slices.py tests/workflows tests/repro` exits 0
reporting 0 files listed. It fails at the base, where KNOWN holds 23
files.)
- [x] **C2** Given each of the 23 files, when the whole tests/ suite
runs, then it passes, and no test was deleted without a behaviour test
or a stated reason in the pull request (a slice that checked only
wording).
Test: tests/workflows/test_literal_slices.py (new
`test_c2_none_of_the_23_files_slices`: runs `literal-slices.py --all` on
each of the 23 paths and asserts that none is reported and every path
still exists. It fails at the base. The converted files themselves pass
in the full `pytest tests/` run.)
<!-- /dx-checklist -->
## 🧭 Decisions
- 2026-11-19: Added two pure readers to tests/workflows/steps.py,
sections() (markdown by heading) and numbered() (numbered list items),
shared by the doc, template and prompt tests, as the Spec allows.
- 2026-11-19: Dropped test_c3's guard that --all finds some slice in
tests/workflows: with every file converted none is left; the tmp_path
tests of the same file still prove --all detects a slice.
- 2026-11-19: Credits list nobody: the issue's author, @soydiloreto, is
the code owner who accepts and merges it, and AGENTS.md rule 10 says
that maintainer is never a co-author (an earlier run listed them; they
can add it back with /coauthor).
## 🙌 Credits
<!-- dx-credits -->
<!-- /dx-credits -->
## 🧪 How I tested it
<!-- dx-tested -->
## How I tested it
- The C1 and C2 tests in tests/workflows/test_literal_slices.py both
failed first, at the base, where KNOWN still listed 23 files and
`literal-slices.py --all` reported them. Both pass at HEAD.
- `python3 scripts/literal-slices.py tests/workflows tests/repro` prints
"0 files still listed in KNOWN", and `python3 scripts/literal-slices.py
--test` passes.
- Running the whole suite, `python3 -m pytest -q tests`, gives 681
passed and 1 skipped.
- For each test that now runs a step, I broke a scratch copy of that
step's `run:` and the test failed. I didn't keep those copies.
- `test_c3_the_worst_wording_tests_of_tests_workflows_slice_no_more` now
runs `--all` on tests/workflows and on a tmp_path file with one known
slice, in the same call, and checks that the slice is reported. If
`--all` reads nothing, the test fails.
- The local review's checks (conventions, policy, brief) pass. This
repository has no `make pre-pr`.
## Removed, with the reason (C2)
- In tests/workflows/test_required_checks.py, I removed the assertion
that the comment above `on:` in pull-request.yml names "required checks"
and the two check names. It checked only wording. A behaviour check
replaces it: the two check names are worked out from the jobs' YAML
(`name`, and `test-first.yml`'s job) and must equal the required list.
- No other test function or assertion was removed. Each slice was
replaced by one of these: a run of the step with stubs, `steps.evaluate`
on its `if:`, checks on YAML fields, or reading the doc as structure
(`steps.sections` and `steps.numbered`, new in
tests/workflows/steps.py).
## How I tested it
- The C1 and C2 tests in tests/workflows/test_literal_slices.py both
failed first, at the base, where KNOWN still listed 23 files and
`literal-slices.py --all` reported them. Both pass at HEAD.
- `python3 scripts/literal-slices.py` prints "0 files still listed in
KNOWN", and `python3 scripts/literal-slices.py --test` passes.
- Running the whole suite, `python3 -m pytest -p no:cacheprovider
--rootdir=. -q tests`, gives 681 passed and 1 skipped.
- For each test that now runs a step, I broke a scratch copy of that
step's `run:` and the test failed. I didn't keep those copies.
- The local review's checks (conventions, policy, brief) pass. This
repository has no `make pre-pr`.
## Removed, with the reason (C2)
- In tests/workflows/test_required_checks.py, I removed the assertion
that the comment above `on:` in pull-request.yml names "required checks"
and the two check names. It checked only wording. It is replaced by a
behaviour check: the two check names are worked out from the jobs' YAML
(`name`, and `test-first.yml`'s job) and must equal the required list.
- In tests/workflows/test_literal_slices.py,
`test_c3_the_worst_wording_tests_of_tests_workflows_slice_no_more` no
longer requires at least one slice to be found in tests/workflows,
because none is left. The tests in the same file that use tmp_path still
show that `--all` finds a slice.
- No other test function or assertion was removed. Each slice was
replaced by a run of the step with stubs, `steps.evaluate` on its `if:`,
YAML field checks, or reading the doc as structure (`steps.sections` and
`steps.numbered`, new in tests/workflows/steps.py).
🤖 AI-generated · Claude Opus 5.5 (Anthropic)
---------
Co-authored-by: dilux-bot[bot] <333187390+dilux-bot[bot]@users.noreply.github.com>
📝 What changes
C1. In
scripts/test-first.pyverdict, afeatcounts as failing first only when it isreproducedat the base, the same as afix. That means an assertion failed. Abrokenresult (an ImportError, a collection error, a missing file, no report) no longer counts. A new test for a new script therefore asserts on what it runs, for example a subprocess's exit code and output, so that the base fails an assertion and does not just error. The feat message says this. The self-test casefeat broken → passis flipped, and a case is added for it. rules/test-first.md, docs/testing-and-quality.md and thetest-first:comment in .github/review-policy.yml say the same.C2. A new script,
scripts/literal-slices.py, with its own--test, reads test files with Python'sastmodule. It reportspath:line (test function)for each place where a test cuts text between literals:.index/.find/.rindex/.rfind("literal"), or a name assigned from one of those calls in the same function;.split/.rsplit/.partition/.rpartition("literal").The scripts job in .github/workflows/pull-request.yml runs it with
--testand overtests/workflows tests/repro, and exits 1 naming each hit. The files that still slice (6 in tests/repro, issues 549, 575, 696, …, and 17 in tests/workflows left for C3's follow-up) are in the script's KNOWN, each with its number of slices, which can only shrink: a listed file with more slices than its count fails, naming each slice, so a new slice in a listed file is caught; one with fewer fails too, so its count is lowered or the file removed in the change that fixes it. A path given that does not exist exits 2 rather than scanning nothing. In .github/review-policy.yml, the reproduction guide gets one bullet: write the stubs and thesubprocesscall in the test file itself, since the runs-step check (#841) reads only that file's own process calls, so a helper such as tests/workflows/steps.py would read as "the test runs no step"; and never cut the workflow's text between two literals. The Test first comment there says a feature's test, like a fix's, fails on an assertion at the base.C3, in part. 19 files of tests/workflows no longer cut text between literals, worst first (maintainer_bug_wiring, policy_on_receipts, autopilot_main_merge, settled_finding_rerun, …): each slice is replaced by a behaviour test or deleted. A behaviour test does one of two things:
run:in bash with stubghand helper commands on PATH that record their calls, and asserts on the calls and on$GITHUB_OUTPUT;The shared code goes in a helper,
tests/workflows/steps.py, which is not a test file. A slice that only checked a sentence's wording is deleted. The rest of C3 (17 files of tests/workflows and 6 of tests/repro, the 23 still in KNOWN) goes to a follow-up Task under #845: this pull request already changes 20 tests, the most Test first runs (MAX_TESTS). C3 is met when KNOWN is empty.💡 Why
Closes #845. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.
🎯 Acceptance criteria
featpull request whose only new test errors at the base (ImportError), when test first checks it, then it does not count as failing first.Test: tests/workflows/test_test_first.py
Test: tests/workflows/test_literal_slices.py
Test: tests/workflows/test_literal_slices.py
🧭 Decisions
🙌 Credits
🧪 How I tested it
scripts/test-first.py runthroughscripts/repro-run.sh run-ref, unconfined, with pytest 9.1.1 and PyYAML 6.0.3) over the 20 changed tests, with the base 5ffc03b and this head. It passes. 13 tests fail on an assertion at the base and pass at the head. 7 pass at both. None isbroken.tests/workflows/steps.pywerebrokenat the base (ModuleNotFoundError: No module named 'steps').test_test_first.pywasreproducedat the head, becausepolicy.py --testcannot rungit ls-filesin the check's tree, which has no.git.scripts/literal-slices.pyabsent, the C2 and C3 tests intests/workflows/test_literal_slices.pyfail on assertions. The C1 tests intests/workflows/test_test_first.pyfail on the old verdict.tests/suite with pytest: 620 passed, 1 skipped. These self-tests pass:test-first.py,policy.py,literal-slices.py,repro-ways.pyandrules.py--test.literal-slices.py tests/workflows tests/repropasses with 23 files left in KNOWN.make pre-pr.🤖 AI-generated · Claude Opus 5.5 (Anthropic)