Skip to content

fix: release scenario bug-by-person-merges fails - #879

Merged
soydiloreto merged 2 commits into
mainfrom
fix/860-release-scenario-bug-by-person
Nov 19, 2026
Merged

soydiloreto merged 2 commits into
mainfrom
fix/860-release-scenario-bug-by-person

Conversation

@dilux-bot

@dilux-bot dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

📝 What changes

The cause: since #849 (#842), G7 (scripts/autopilot.py decide, with check_evidence always passed by the decide command) requires each criterion's test to be run by a required check that concluded success at the head (criteria_unmet → check_of). In the test repository the autopilot's fixed change (kinds/engine-test/pack.yml fixed-change.file, scenarios/{scenario}/{n}.test.md) is the criterion's only "test", and the same pack lists scenarios/** in test-first.not-tests, so check_of returns None and G7 answers "C1: no required check runs scenarios/bug-by-person-merges/<n>.test.md (G6)". The autopilot's fix/<n>-… pull request never merges by policy, and bug-by-person-merges ends with its issue OPEN. tests/repro/test_issue_834.py still passes because it calls decide without head_checks (the old callers' path), which is why nothing caught it.

The fix (recommended, option A): check_evidence(env, repo) also returns the fixed change's pattern, fixed_file(org.load(), repo) (non-empty only in the test repository whose pack says autopilot: fixed-change), and decide/criteria_unmet take it as fixed="": a criterion whose present test is that pattern's file (the pattern with {n}, {id}, {scenario} read as one path segment) is met by being written and changed by the pull request, as before #849, with no required check asked of it, since it is a file the fixed change writes and no check can run. Everywhere else (fixed empty) G7 is unchanged, and in the test repository any other path still needs its required check green. decide's reason says when a criterion was met by the fixed change. The pack's comment on fixed-change and kinds/README.md's fixed-change row say so; scripts/autopilot.py --test gets a check for it.

Options considered and not recommended:

  • B: declare the fixed file as a test in the pack's test-first: with a check: (e.g. the test repository's pipeline check). Wrong: pipeline needs auto-merge, the job that runs decide, so G7 would wait on itself; and no check actually runs a .md file.
  • C: make the fixed change write a runnable Python test under tests/repro/ that a required check runs. Leaves each scenario's own directory (the core partition of fix: release scenarios wait in core, look duplicate and commit unsigned #709, release-scenarios.py core_problems), and makes "Test first" run it at the base for every scenario; much larger than the PRD's scope.

This widens no permission, secret, lane or limit outside the test repository; inside it, G7 stops asking a required check of the fixed change's own file only (the pull request body will say so).

💡 Why

Closes #860. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.

🎯 Acceptance criteria

  • C1 Given v7.19.0's candidate with the fix, when the release lane plays bug-by-person-merges in the test repository, then it passes and the lane releases it.
    Test: tests/workflows/test_fixed_change_required_check.py (builds the decide step's evidence files as the workflow writes them for the test repository, with the head's required checks green, and the fixed Spec of a person's Normal Bug from bug-by-person-merges; calls check_evidence and decide as the decide command does, and asserts it merges by policy; at the base it fails with "no required check runs scenarios/bug-by-person-merges/<n>.test.md")

🧭 Decisions

  • 2026-11-19: The f​ixed change's pattern is matched with {n}, {id} and {scenario} each one path segment ([^/]+), so only the file the f​ixed change writes is exempt from a required check; decide's reason says when the f​ixed change met it.

🙌 Credits

🧪 How I tested it

The required checks ran at 024f791 and passed: Scripts (self-tests), Test first (the test fails at the base and passes at the head), actionlint and the conventions, in https://github.com/DiluxOne/.github/actions/runs/44599924946. A maintainer marked this pull request ready: the autopilot at v7 (v7.18.0) refuses to finish a release lane's bug, because it reads a reproduction test from the PRD and this issue has only the lane's scenario. Reading the scenario as the evidence (scenario_evidence) is on main since #836, not in v7.18.0, and it ships in the release this fix unblocks.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

The draft pull request opens before any code: the Spec as its checklist.
@dilux-bot dilux-bot Bot added the autopilot Opened by the autopilot from an accepted issue label Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

The pull request's board

State
Waits on nobody

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

Since #842, G7 asks each criterion's test to be run by a required check green at the head. In the test repository the fixed change's file, scenarios/<scenario>/<n>.test.md, is the criterion's only test and is among the pack's not-tests, so no check runs it and bug-by-person-merges never merged. check_evidence now passes the fixed change's pattern (test repository only), and criteria_unmet meets a test that is that pattern's file by being written and changed by the pull request; every other test, and every other repository, is unchanged.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot ticked every criterion but ended without finishing. The checks (make pre-pr), its own review and the finish were left, so it continues once to do them (autopilot:continue). What it did is in the run.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot ended without a result. The agent finished without marking the pull request ready or saying why it stopped. Nothing was marked ready and the agent left no stop comment, so a person takes it from here (needs-human): a maintainer's answer here, a hint in words, brings it back, or they take it over. What it did is in the run: its summary, and the full transcript as an artifact.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot added the needs-human The autopilot handed this over to a person label Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: a person takes it from here (needs-human): read why and decide #879

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@soydiloreto
soydiloreto marked this pull request as ready for review November 19, 2026 19:35
@soydiloreto
soydiloreto self-requested a review as a code owner November 19, 2026 19:35
@soydiloreto soydiloreto added the review:full Ask the Claude review for a full pass on the next push label Nov 19, 2026
@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:medium Set by the Claude review type:fix The kind of change, read from the diff by the Claude review and removed review:full Ask the Claude review for a full pass on the next push labels Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Claude review · risk high · complexity medium · type fix

Nothing that must be fixed · 1 minor

The fix lets G7 treat the test repository's fixed-change file (scenarios/{scenario}/{n}.test.md) as met by being written and changed. It asks no required check of that file, and only where the pack declares autopilot: fixed-change. check_evidence passes fixed to decide through **check_evidence(...), and decide's reason says when the fixed change met the criterion. {n}, {id} and {scenario} are matched as one path segment each, fixed_file returns "" outside the test repository, and any other test there still needs its green required check. The new tests cover the fixed file merging, a missing file, another test and another repository. Nothing in the diff widens a permission or limit outside the test repository. Minor: criteria_unmet skips the check lookup for the whole criterion when any of its present tests is the fixed file, so a criterion line that named both the fixed file and a real test would not need the real test's check. The autopilot's own Spec never writes that, so it is not exploitable. This is not a new capability, so there is no version:major suggestion.

  • minor scripts/autopilot.py:765: A criterion naming the fixed file and another test skips the other test's check

Policy floor: high (touches high-risk paths: kinds/README.md, kinds/engine-test/pack.yml, scripts/autopilot.py, tests/workflows/test_fixed_change_required_check.py). Reviewed 024f791 (whole pull request; review 1 of 5 automatic). Author trusted for auto-merge: false.

🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.27, 7 turns

@soydiloreto
soydiloreto merged commit 6b2ca76 into main Nov 19, 2026
58 of 60 checks passed
@soydiloreto
soydiloreto deleted the fix/860-release-scenario-bug-by-person branch November 19, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autopilot Opened by the autopilot from an accepted issue complexity:medium Set by the Claude review needs-human The autopilot handed this over to a person risk:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: release scenario bug-by-person-merges fails

1 participant