Skip to content

fix: release scenario settled-rerun fails - #880

Merged
soydiloreto merged 3 commits into
mainfrom
fix/871-release-scenario-settled-rerun-fails
Nov 19, 2026
Merged

soydiloreto merged 3 commits into
mainfrom
fix/871-release-scenario-settled-rerun-fails

Conversation

@dilux-bot

@dilux-bot dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

📝 What changes

The cause is in review-reply.yml. settled-rerun settles the token finding with a comment in the pull request's conversation. The review posts a confirmed blocker inline, as a thread of its own (claude-review.yml, "each a thread to resolve"). The reply records a settlement, and with it re-runs the review, in only two cases. One is resolve on a pull_request_review_comment inside the bot's own thread. The other is settles on an issue_comment naming a finding "with no thread of its own", which is what the reply's prompt asks for. A conversation comment that settles a finding that has a thread fits neither case. So settled stays empty, nothing re-runs the review, and the scenario's rerun never holds. The reply's model also cannot see those threads: on an issue_comment, thread.json holds only the conversation. docs/agents.md says the same rule ("its thread resolved, or a dx-settled mark for a finding with no thread").

The fix, in review-reply.yml:

  1. "Pull request and thread": on an issue_comment, thread.json also lists the bot's open review threads (id, path, line, first comment), from the same GraphQL query the review-comment branch uses, and those threads are saved for the post step.
  2. The reply's prompt: on a conversation comment, settles may name any finding the review's summary lists, whether or not it has a thread.
  3. "Post the reply": on an issue_comment with a non-empty settles, the workflow writes the dx-settled mark as before (settled=true), so the existing "Re-run the review on a settled finding" step runs: it still checks who decides, that the review blocked at the head, and that a review run finished there. The match is deterministic: a thread matches when its first comment starts with **<severity>**: <title>, which is how claude-review.yml writes it.

One choice is a maintainer's: what happens to the finding's open thread.

  • A (recommended): when settles matches one of the bot's open threads, the reply also resolves that thread through the existing "Resolve the thread" step (its id in resolve.txt, resolve=true), with the token minted after the model ran. The settlement is then visible on the finding where it was raised, exactly as when someone answers inside the thread, so the two ways of settling give the same result.
  • B: only the mark and the re-run. The thread stays open until the re-run review lists it in resolved_threads, so it depends on the model doing that.

Either way, no permission, secret, lane or limit is widened: the steps, tokens and permissions are the ones already there (pull-requests: write resolves a thread today, actions: write re-runs today). The gates and what they check do not change. The release scenario stays as it is. The header comments of review-reply.yml and claude-review.yml (#699 C5), and docs/agents.md and docs/pull-requests.md, change to say that a conversation answer can settle a finding that has a thread.

💡 Why

Closes #871. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.

🎯 Acceptance criteria

  • C1 Given v7.19.0's candidate with the fix, when the release lane plays settled-rerun in the test repository, then it passes and the lane releases it.
    Test: tests/workflows/test_settled_finding_rerun.py (runs review-reply.yml's "Pull request and thread" step on an issue_comment, with gh stubbed, and asserts that thread.json lists the bot's open review threads. Runs its "Post the reply" step on an issue_comment whose settles names a finding that has an open bot thread, and asserts settled=true, the dx-settled mark in the reply and, with option A, resolve=true with that thread's id. At the base both fail on an assertion. Also asserts that a settles that matches no thread still settles with no resolve, which must not change.)

🧭 Decisions

  • 2026-11-19: Option A of the Spec, as C1's test names it: a conversation comment's settles r​esolves the bot's open thread whose first line is exactly <blocker|major|minor>: <title>; a title that only begins another finding's matches nothing.

🙌 Credits

🧪 How I tested it

The required checks ran at 165cfb6 and passed: Scripts (self-tests), Test first (the test fails at the base and passes at the head), actionlint and the conventions, in https://github.com/DiluxOne/.github/actions/runs/44599936545. A maintainer marked this pull request ready: the autopilot at v7 (v7.18.0) refuses to finish a release lane's bug, because it reads a reproduction test from the PRD and this issue has only the lane's scenario. Reading the scenario as the evidence (scenario_evidence) is on main since #836, not in v7.18.0, and it ships in the release this fix unblocks.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

The draft pull request opens before any code: the Spec as its checklist.
@dilux-bot dilux-bot Bot added the autopilot Opened by the autopilot from an accepted issue label Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

The pull request's board

State
Waits on nobody

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

A comment in the pull request's conversation that settled a finding the review had raised inline fitted neither case the reply recorded, so nothing re-ran the review and the release scenario settled-rerun never held. On an issue_comment, review-reply.yml now shows the reply the bot's open review threads, lets settles name any finding of the review's summary, and resolves the finding's thread (matched on its first line, as claude-review.yml writes it) along with the dx-settled mark that re-runs the review.

Closes #871

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot ticked every criterion but ended without finishing. The checks (make pre-pr), its own review and the finish were left, so it continues once to do them (autopilot:continue). What it did is in the run.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot mentioned this pull request Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot ended without a result. The agent finished without marking the pull request ready or saying why it stopped. Nothing was marked ready and the agent left no stop comment, so a person takes it from here (needs-human): a maintainer's answer here, a hint in words, brings it back, or they take it over. What it did is in the run: its summary, and the full transcript as an artifact.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot added the needs-human The autopilot handed this over to a person label Nov 19, 2026
@soydiloreto
soydiloreto marked this pull request as ready for review November 19, 2026 19:35
@soydiloreto
soydiloreto self-requested a review as a code owner November 19, 2026 19:35
@soydiloreto soydiloreto added the review:full Ask the Claude review for a full pass on the next push label Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: a person takes it from here (needs-human): read why and decide #880

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:medium Set by the Claude review type:fix The kind of change, read from the diff by the Claude review and removed review:full Ask the Claude review for a full pass on the next push labels Nov 19, 2026
@dilux-bot

dilux-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Claude review · risk high · complexity medium · type fix

Nothing that must be fixed · 3 minor · 1 pre-existing, not this pull request's

First review of this pull request (no earlier commits or threads). The diff does what the description says. On an issue_comment, thread.json now lists the bot's open review threads. The prompt lets settles name a finding that has a thread. When settles matches an open bot thread, "Post the reply" sets resolve=true for it (option A). The match is exact on the first line **<severity>**: <title>, as claude-review.yml writes it (line 811). The jq regex escaping is correct. A settles with no matching thread still settles and resolves nothing, and a title that only begins another finding's matches nothing; both are tested. The three copies of the rule (the two workflow header comments, docs/agents.md and docs/pull-requests.md) were updated. I did not run the tests; I read the tests and the workflow. Minor points are in the findings: they are not blockers and none has an inline comment.

  • minor .github/workflows/review-reply.yml:702: A failed thread resolution skips the review's re-run after the mark is posted
  • minor .github/workflows/review-reply.yml:690: Thread match is an exact title match, so near-misses silently leave the thread open
  • minor .github/workflows/review-reply.yml:508: The review-threads query now runs on every conversation comment
  • minor (pre-existing) .github/workflows/review-reply.yml:684: Resolving a thread needs only write permission, not who-decides

Policy floor: high (touches high-risk paths: .github/workflows/claude-review.yml, .github/workflows/review-reply.yml, docs/agents.md, tests/workflows/test_settled_finding_rerun.py). Reviewed 37a53f0 (whole pull request; review 2 of 5 automatic). Author trusted for auto-merge: false.

🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.32, 12 turns

@dilux-bot

dilux-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: a person takes it from here (needs-human): read why and decide #880

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@soydiloreto
soydiloreto merged commit 4ab8649 into main Nov 19, 2026
39 checks passed
@soydiloreto
soydiloreto deleted the fix/871-release-scenario-settled-rerun-fails branch November 19, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autopilot Opened by the autopilot from an accepted issue complexity:medium Set by the Claude review needs-human The autopilot handed this over to a person risk:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: release scenario settled-rerun fails

1 participant