From a174d1f1c9a5c1d23c4525301e4c6c5e4efeaa93 Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Mon, 26 Oct 2026 11:21:32 -0400 Subject: [PATCH 1/2] thirdshift: failed ticket run (grok exited 1: Internal error: {) 2026-10-26T15:21:32Z, host thirdshift. Uncommitted work at the time of failure is included in this commit. --- GLOSSARY.md | 2 +- prompts/README.md | 4 +- prompts/bug-continuation-pr.md | 8 +- prompts/bug-continuation.md | 8 +- site/architecture/index.html | 2 +- site/how-it-works/index.html | 2 +- site/index.html | 2 +- site/prompts/index.html | 32 ++- skills/thirdshift-diagnosing-bugs/SKILL.md | 16 +- src/prompt.rs | 18 +- src/prompts_page.rs | 32 ++- src/run.rs | 4 +- src/worktree.rs | 13 ++ tests/bug_run.rs | 220 +++++++++++++++++++++ 14 files changed, 344 insertions(+), 19 deletions(-) diff --git a/GLOSSARY.md b/GLOSSARY.md index 103e73ab..e8a590cb 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -96,7 +96,7 @@ One invocation of thirdshift that does factory work: a **Ticket run** or **Spec _Avoid_: invocation, job **Ticket run**: -One invocation of the factory on a single issue that is not a **Spec**, from launch to cleanup, normally ending in one pull request ready for review, or a **Self-merge** when its command or the **User config** asks for one. A Ticket run on an issue labelled `bug` reproduces the bug before fixing it, in a fresh Ticket run or a **Continuation**, except for Security fixes and **Base fixes**, which have their own flow. If the bug is not reproduced, the Ticket run ends successfully with triage notes and `needs-info` on the issue, without opening or merging a pull request. Started directly on an **Issue URL**, by a **Spec run** for one of its **Tickets**, by an **Architect run** for the Ticket its **Architecture review** published, by a **Pickup run** for the **Ready issue** it took, or by a **Security run** for a **Security finding** it fixes. +One invocation of the factory on a single issue that is not a **Spec**, from launch to cleanup, normally ending in one pull request ready for review, or a **Self-merge** when its command or the **User config** asks for one. A Ticket run on an issue labelled `bug` reproduces the bug before fixing it, in a fresh Ticket run or a **Continuation**, except for Security fixes and **Base fixes**, which have their own flow. A Continuation whose **Issue branch** has commits beyond the **Base branch** reproduces at their merge base, then checks the fix on the branch; a fix already there still goes on to review and the pull request. If the bug is not reproduced, including when it is absent at that merge base, the Ticket run ends successfully with triage notes and `needs-info` on the issue, without opening or merging a pull request. Started directly on an **Issue URL**, by a **Spec run** for one of its **Tickets**, by an **Architect run** for the Ticket its **Architecture review** published, by a **Pickup run** for the **Ready issue** it took, or by a **Security run** for a **Security finding** it fixes. _Avoid_: Run, Merge run (former names), auto-merge (GitHub's own feature) **Spec run**: diff --git a/prompts/README.md b/prompts/README.md index 0c61239c..8e4f13dd 100644 --- a/prompts/README.md +++ b/prompts/README.md @@ -29,8 +29,8 @@ With `harness codex`, each session runs `codex exec --json --dangerously-bypass- | Code review | In an Eval Trial, reviews the prepared historical diff with the production code-review Factory skill, under the implement session's sub-agent rules. Writes reports without fixing or publishing anything. | [code-review.md](code-review.md) | | Spec review | In a Spec run, starts the Spec review once every Ticket has landed on the Spec branch, before the Spec PR, a draft until then, is marked ready. | [spec-review.md](spec-review.md) | | Bug Ticket run, fresh | Starts the implement session on an issue labelled bug: diagnose before changing code, then implement only if reproduced. Security fixes and Base fixes keep their own flow. | [bug-fresh.md](bug-fresh.md) | -| Bug Ticket run, Continuation without a pull request | Diagnoses a bug in a Continuation with no pull request before changing code. A not reproduced ending hands the issue back for information without a PR. | [bug-continuation.md](bug-continuation.md) | -| Bug Ticket run, Continuation with an open pull request | Diagnoses a bug before continuing its open pull request. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information. | [bug-continuation-pr.md](bug-continuation-pr.md) | +| Bug Ticket run, Continuation without a pull request | Diagnoses a bug in a Continuation with no pull request before changing code. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending hands the issue back for information without a PR. | [bug-continuation.md](bug-continuation.md) | +| Bug Ticket run, Continuation with an open pull request | Diagnoses a bug before continuing its open pull request. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information. | [bug-continuation-pr.md](bug-continuation-pr.md) | | Architecture review | Starts the Architecture review, the session an Architect run opens with, in a worktree at the head of the Base branch. The line naming the focus is left out when the command gives none. | [architecture-review.md](architecture-review.md) | | Security review | When enabled, runs one guidance review after a Ticket run's opening session or a Spec run's Spec review, before Delivery pushes and enters the Repair loop. Each Ticket run and Base fix gets one too. Introduced findings, refusals or incomplete reviews hold Self-merge. | [security-review.md](security-review.md) | | Security audit | Starts the report-only Security audit in a throwaway worktree at origin's Base branch head. The threat-model line is included when a conventional document exists; artifacts stay under the repository's audit root. | [security-audit.md](security-audit.md) | diff --git a/prompts/bug-continuation-pr.md b/prompts/bug-continuation-pr.md index 9c3c75b5..54170d23 100644 --- a/prompts/bug-continuation-pr.md +++ b/prompts/bug-continuation-pr.md @@ -2,7 +2,7 @@ # Bug Ticket run, Continuation with an open pull request -Diagnoses a bug before continuing its open pull request. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information. +Diagnoses a bug before continuing its open pull request. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information. ``` /thirdshift-diagnosing-bugs @@ -10,6 +10,12 @@ Use the `thirdshift-diagnosing-bugs` skill before changing code. Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below. Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: `. +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, not on the branch, then check the fix on the branch. +Merge base commit: ``. +Use that commit as given; do not infer another from git history. +Hand back `needs-info` only when the bug does not reproduce at that merge base. +When it reproduces there and the branch already fixes it, that is the reproduced ending: continue to review and the pull request. + Implementation after reproduction: /thirdshift-implement diff --git a/prompts/bug-continuation.md b/prompts/bug-continuation.md index d17da129..1ec412b7 100644 --- a/prompts/bug-continuation.md +++ b/prompts/bug-continuation.md @@ -2,7 +2,7 @@ # Bug Ticket run, Continuation without a pull request -Diagnoses a bug in a Continuation with no pull request before changing code. A not reproduced ending hands the issue back for information without a PR. +Diagnoses a bug in a Continuation with no pull request before changing code. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending hands the issue back for information without a PR. ``` /thirdshift-diagnosing-bugs @@ -10,6 +10,12 @@ Use the `thirdshift-diagnosing-bugs` skill before changing code. Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below. Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: `. +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, not on the branch, then check the fix on the branch. +Merge base commit: ``. +Use that commit as given; do not infer another from git history. +Hand back `needs-info` only when the bug does not reproduce at that merge base. +When it reproduces there and the branch already fixes it, that is the reproduced ending: continue to review and the pull request. + Implementation after reproduction: /thirdshift-implement diff --git a/site/architecture/index.html b/site/architecture/index.html index 78409064..ca1a764f 100644 --- a/site/architecture/index.html +++ b/site/architecture/index.html @@ -225,7 +225,7 @@

Architecture

  • Pre-flight checks stop a Ticket run before anything is created: the Issue URL must match the clone's origin, the issue must be open, and the Base branch must be on origin.
  • The Claim labels the issue in-progress, so the issue list shows what the factory is working on.
  • A Worktree gives the Ticket run its own checkout beside your clone, so yours is never touched.
  • -
  • Delivery starts the implement session. On issues labelled bug, diagnosis comes first: a reproduced bug gets a failing regression test and fix; an unreproduced bug ends successfully with triage notes and needs-info, without PR delivery. Security fixes and Base fixes keep their own flow. Otherwise Delivery pushes the branch and checks that the pull request exists and targets the Base branch. thirdshift, not the agent, marks it ready.
  • +
  • Delivery starts the implement session. On issues labelled bug, diagnosis comes first: a reproduced bug gets a failing regression test and fix; a Continuation whose Issue branch is ahead of the Base branch reproduces at their merge base, and a fix already on the branch goes on to review and the pull request; an unreproduced bug ends successfully with triage notes and needs-info, without PR delivery. Security fixes and Base fixes keep their own flow. Otherwise Delivery pushes the branch and checks that the pull request exists and targets the Base branch. thirdshift, not the agent, marks it ready.
  • Keeping it green merges the Base branch in and watches CI. A conflict or the branch's own red check starts a Repair, at most 5 a Ticket run.
  • The Self-merge, when asked, merges exactly the head commit whose CI thirdshift watched.
  • Cleaning up removes the Worktree. A Failed ticket run pushes its work first and turns the pull request back into a draft.
  • diff --git a/site/how-it-works/index.html b/site/how-it-works/index.html index f1b47ba6..f2059bbc 100644 --- a/site/how-it-works/index.html +++ b/site/how-it-works/index.html @@ -58,7 +58,7 @@

    How thirdshift works

    01 / The human handoff

    Start with work you can describe

    The Day shift is you: settle what should change, what success looks like, and what is out of scope. Put that in a GitHub issue. One session's worth of work can stand alone; a larger Spec is an issue with Tickets as GitHub sub-issues. We'll follow a standalone issue first.

    -

    For issues labelled bug, a fresh Ticket run or Continuation uses /thirdshift-diagnosing-bugs before changing code. It captures a failing regression test, then fixes it. If the bug cannot be reproduced, it comments with triage notes, replaces ready-for-agent with needs-info and ends successfully with the comment link, without opening or merging a pull request. Security fixes and Base fixes keep their own flow.

    +

    For issues labelled bug, a fresh Ticket run or Continuation uses /thirdshift-diagnosing-bugs before changing code. A fresh run, or a Continuation with nothing on the Issue branch beyond the Base branch, reproduces on that checkout. A Continuation whose Issue branch is ahead of the Base branch reproduces at their merge base, then checks the fix on the branch: a bug there, with the fix already on the branch, goes on to review and the pull request. Otherwise it captures a failing regression test, then fixes it. If the bug cannot be reproduced at that checkout, it comments with triage notes, replaces ready-for-agent with needs-info and ends successfully with the comment link, without opening or merging a pull request. Security fixes and Base fixes keep their own flow.

    Human intent becomes the agent's brief
      diff --git a/site/index.html b/site/index.html index d943c4ec..a84016c0 100644 --- a/site/index.html +++ b/site/index.html @@ -90,7 +90,7 @@

      Your issues become pull requests overnight.

      Third shift

      Agents, with headless copies of the same skills

      1. /thirdshift-implement
      2. /thirdshift-code-review
      3. /thirdshift-pr

      thirdshift takes an Issue URL: one issue starts a Ticket run; an issue with sub-issues starts a Spec run. Its Factory skills, adapted to run with no human in the loop, implement the issue with /thirdshift-implement, review the work with /thirdshift-code-review and write the pull request with /thirdshift-pr.

      -

      Issues labelled bug first use /thirdshift-diagnosing-bugs to reproduce the reported failure and write a failing regression test before fixing it. If it cannot reproduce the bug, the Ticket run leaves triage notes and needs-info on the issue, ending successfully without a new pull request. Security fixes and Base fixes keep their own flow.

      +

      Issues labelled bug first use /thirdshift-diagnosing-bugs to reproduce the reported failure and write a failing regression test before fixing it. A Continuation whose Issue branch is ahead of the Base branch reproduces at their merge base, and a fix already on the branch still goes on to review and the pull request. If it cannot reproduce the bug, the Ticket run leaves triage notes and needs-info on the issue, ending successfully without a new pull request. Security fixes and Base fixes keep their own flow.

      Read every prompt and skill the agents get →

    diff --git a/site/prompts/index.html b/site/prompts/index.html index b6999d0c..839ce664 100644 --- a/site/prompts/index.html +++ b/site/prompts/index.html @@ -285,13 +285,19 @@

    Prompts and skills

    Bug Ticket run, Continuation without a pull request

    -

    Diagnoses a bug in a Continuation with no pull request before changing code. A not reproduced ending hands the issue back for information without a PR.

    +

    Diagnoses a bug in a Continuation with no pull request before changing code. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending hands the issue back for information without a PR.

    Sent by 2 · M · Implement

    /thirdshift-diagnosing-bugs <Issue URL>
     Use the `thirdshift-diagnosing-bugs` skill before changing code.
     Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below.
     Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: <comment URL>`.
     
    +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, not on the branch, then check the fix on the branch.
    +Merge base commit: `<merge base commit>`.
    +Use that commit as given; do not infer another from git history.
    +Hand back `needs-info` only when the bug does not reproduce at that merge base.
    +When it reproduces there and the branch already fixes it, that is the reproduced ending: continue to review and the pull request.
    +
     Implementation after reproduction:
     /thirdshift-implement <Issue URL>
     
    @@ -325,13 +331,19 @@ 

    Prompts and skills

    Bug Ticket run, Continuation with an open pull request

    -

    Diagnoses a bug before continuing its open pull request. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information.

    +

    Diagnoses a bug before continuing its open pull request. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information.

    Sent by 2 · M · Implement

    /thirdshift-diagnosing-bugs <Issue URL>
     Use the `thirdshift-diagnosing-bugs` skill before changing code.
     Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below.
     Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: <comment URL>`.
     
    +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, not on the branch, then check the fix on the branch.
    +Merge base commit: `<merge base commit>`.
    +Use that commit as given; do not infer another from git history.
    +Hand back `needs-info` only when the bug does not reproduce at that merge base.
    +When it reproduces there and the branch already fixes it, that is the reproduced ending: continue to review and the pull request.
    +
     Implementation after reproduction:
     /thirdshift-implement <Issue URL>
     
    @@ -1058,9 +1070,21 @@ 

    Prompts and skills

    Redact secrets from commands, output and captured artifacts. Build loops against environment variables so credentials stay out of the evidence. +## Where to reproduce + +The Session prompt says which checkout the bug is reproduced on. + +When it names a merge base, this is a Continuation whose Issue branch already has commits beyond the Base branch. Reproduce at that merge base. Use the commit the prompt names; do not choose another from git history. Then check the fix on the Issue branch. + +- The bug reproduces at the merge base and the branch already fixes it: take the reproduced ending. Do not rewrite the fix. Continue to the Session prompt's implementation steps, including review and the pull request. +- The bug reproduces at the merge base and the branch does not fix it: take the reproduced ending, finish the fix, then continue to those steps. +- The bug does not reproduce at the merge base: take the not reproduced ending. A passing check on the branch is not enough to hand the issue back. + +A fresh bug Ticket run, or a Continuation whose Session prompt names no merge base, reproduces against the checkout you are in. + ## 1. Build a tight feedback loop -Spend most of the diagnosis effort here. Read enough code to find a seam that reaches the reported symptom, then run an unattended pass/fail command against the current code. Prefer a focused test; a CLI fixture, HTTP replay, headless browser script or throwaway harness can help establish reproduction. Assert the exact reported symptom, rather than simply checking that execution succeeds. A setup or compilation failure is not reproduction. +Spend most of the diagnosis effort here. Read enough code to find a seam that reaches the reported symptom, then run an unattended pass/fail command against the checkout named under Where to reproduce. Prefer a focused test; a CLI fixture, HTTP replay, headless browser script or throwaway harness can help establish reproduction. Assert the exact reported symptom, rather than simply checking that execution succeeds. A setup or compilation failure is not reproduction. Tighten the loop: narrow inputs, pin time and seeds, and remove unrelated setup. For intermittent bugs, repeat the trigger and establish a reproducible failure rate. Record the command, output and conditions. Proceed only once the loop actually fails for the reported reason. If it does not, take the not reproduced ending. @@ -1072,7 +1096,7 @@

    Prompts and skills

    Before changing production code, turn the reproduction into a regression test at a seam that exercises the actual reported pattern. Run it and capture its failure for that reason. If no suitable seam can exercise the bug unattended, record that limitation through the not reproduced ending instead of guessing at a fix. -Make the smallest fix, watch the same test pass, then rerun the original reproduction and the repository's affected checks. Keep the regression test. Remove temporary instrumentation and throwaway artifacts. Continue the Session prompt's implementation, review, commit and PR steps. In the PR's Evidence, show the exact command and its failing-before and passing-after results, and explain the confirmed cause. End the final message with the Session prompt's reproduced completion line only after those steps succeed. +Make the smallest fix, watch the same test pass, then rerun the original reproduction and the repository's affected checks. Keep the regression test. When the Issue branch already fixes a bug you reproduced at the named merge base, keep that fix and its regression test. Remove temporary instrumentation and throwaway artifacts. Continue the Session prompt's implementation, review, commit and PR steps. In the PR's Evidence, show the exact command and its failing-before and passing-after results, and explain the confirmed cause. End the final message with the Session prompt's reproduced completion line only after those steps succeed. ## Not reproduced: needs information, no PR diff --git a/skills/thirdshift-diagnosing-bugs/SKILL.md b/skills/thirdshift-diagnosing-bugs/SKILL.md index 900376c0..8fc3f18a 100644 --- a/skills/thirdshift-diagnosing-bugs/SKILL.md +++ b/skills/thirdshift-diagnosing-bugs/SKILL.md @@ -11,9 +11,21 @@ Run headless: make routine choices yourself and finish with one of the two endin Redact secrets from commands, output and captured artifacts. Build loops against environment variables so credentials stay out of the evidence. +## Where to reproduce + +The Session prompt says which checkout the bug is reproduced on. + +When it names a merge base, this is a Continuation whose Issue branch already has commits beyond the Base branch. Reproduce at that merge base. Use the commit the prompt names; do not choose another from git history. Then check the fix on the Issue branch. + +- The bug reproduces at the merge base and the branch already fixes it: take the reproduced ending. Do not rewrite the fix. Continue to the Session prompt's implementation steps, including review and the pull request. +- The bug reproduces at the merge base and the branch does not fix it: take the reproduced ending, finish the fix, then continue to those steps. +- The bug does not reproduce at the merge base: take the not reproduced ending. A passing check on the branch is not enough to hand the issue back. + +A fresh bug Ticket run, or a Continuation whose Session prompt names no merge base, reproduces against the checkout you are in. + ## 1. Build a tight feedback loop -Spend most of the diagnosis effort here. Read enough code to find a seam that reaches the reported symptom, then run an unattended pass/fail command against the current code. Prefer a focused test; a CLI fixture, HTTP replay, headless browser script or throwaway harness can help establish reproduction. Assert the exact reported symptom, rather than simply checking that execution succeeds. A setup or compilation failure is not reproduction. +Spend most of the diagnosis effort here. Read enough code to find a seam that reaches the reported symptom, then run an unattended pass/fail command against the checkout named under Where to reproduce. Prefer a focused test; a CLI fixture, HTTP replay, headless browser script or throwaway harness can help establish reproduction. Assert the exact reported symptom, rather than simply checking that execution succeeds. A setup or compilation failure is not reproduction. Tighten the loop: narrow inputs, pin time and seeds, and remove unrelated setup. For intermittent bugs, repeat the trigger and establish a reproducible failure rate. Record the command, output and conditions. Proceed only once the loop actually fails for the reported reason. If it does not, take the not reproduced ending. @@ -25,7 +37,7 @@ Remove one input or step at a time, rerunning the loop, until every remaining el Before changing production code, turn the reproduction into a regression test at a seam that exercises the actual reported pattern. Run it and capture its failure for that reason. If no suitable seam can exercise the bug unattended, record that limitation through the not reproduced ending instead of guessing at a fix. -Make the smallest fix, watch the same test pass, then rerun the original reproduction and the repository's affected checks. Keep the regression test. Remove temporary instrumentation and throwaway artifacts. Continue the Session prompt's implementation, review, commit and PR steps. In the PR's Evidence, show the exact command and its failing-before and passing-after results, and explain the confirmed cause. End the final message with the Session prompt's reproduced completion line only after those steps succeed. +Make the smallest fix, watch the same test pass, then rerun the original reproduction and the repository's affected checks. Keep the regression test. When the Issue branch already fixes a bug you reproduced at the named merge base, keep that fix and its regression test. Remove temporary instrumentation and throwaway artifacts. Continue the Session prompt's implementation, review, commit and PR steps. In the PR's Evidence, show the exact command and its failing-before and passing-after results, and explain the confirmed cause. End the final message with the Session prompt's reproduced completion line only after those steps succeed. ## Not reproduced: needs information, no PR diff --git a/src/prompt.rs b/src/prompt.rs index 730ade4f..e9a10068 100644 --- a/src/prompt.rs +++ b/src/prompt.rs @@ -229,12 +229,28 @@ pub const BUG_REPRODUCED_LINE: &str = "Bug reproduced: fixed"; /// Diagnose first on a bug Ticket run, retaining its fresh or Continuation facts /// and its implementation steps for the reproduced ending only. -pub fn diagnosing_bugs(issue: &IssueUrl, implementation: &str) -> String { +/// +/// `merge_base` is set when a Continuation's Issue branch has commits the Base +/// branch does not. The session reproduces there, then checks the fix on the branch. +pub fn diagnosing_bugs(issue: &IssueUrl, implementation: &str, merge_base: Option<&str>) -> String { + let baseline = match merge_base { + Some(commit) => format!( + "This Continuation's Issue branch is ahead of the Base branch. \ + Reproduce at the merge base, not on the branch, then check the fix on the branch.\n\ + Merge base commit: `{commit}`.\n\ + Use that commit as given; do not infer another from git history.\n\ + Hand back `needs-info` only when the bug does not reproduce at that merge base.\n\ + When it reproduces there and the branch already fixes it, that is the reproduced ending: \ + continue to review and the pull request.\n\n" + ), + None => String::new(), + }; format!( "/thirdshift-diagnosing-bugs {url}\n\ Use the `thirdshift-diagnosing-bugs` skill before changing code.\n\ Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below.\n\ Completion lines (the final message's last line): reproduced: `{BUG_REPRODUCED_LINE}`; not reproduced: `{BUG_NOT_REPRODUCED_LINE}`.\n\n\ + {baseline}\ Implementation after reproduction:\n{implementation}", url = issue.url, ) diff --git a/src/prompts_page.rs b/src/prompts_page.rs index 9616e563..2bb2e3f7 100644 --- a/src/prompts_page.rs +++ b/src/prompts_page.rs @@ -198,26 +198,28 @@ fn prompts() -> Vec { title: "Bug Ticket run, fresh", when: "Starts the implement session on an issue labelled bug: diagnose before changing code, then implement only if reproduced. Security fixes and Base fixes keep their own flow.", sender: Units(&[IMPLEMENT]), - text: prompt::diagnosing_bugs(&issue, &prompt::fresh(&issue, BASE, BRANCH)), + text: prompt::diagnosing_bugs(&issue, &prompt::fresh(&issue, BASE, BRANCH), None), }, Prompt { id: "prompt-bug-continuation", title: "Bug Ticket run, Continuation without a pull request", - when: "Diagnoses a bug in a Continuation with no pull request before changing code. A not reproduced ending hands the issue back for information without a PR.", + when: "Diagnoses a bug in a Continuation with no pull request before changing code. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending hands the issue back for information without a PR.", sender: Units(&[IMPLEMENT]), text: prompt::diagnosing_bugs( &issue, &prompt::continuation(&issue, BASE, BRANCH, None), + Some(prompt::SECURITY_REVIEW_MERGE_BASE), ), }, Prompt { id: "prompt-bug-continuation-pr", title: "Bug Ticket run, Continuation with an open pull request", - when: "Diagnoses a bug before continuing its open pull request. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information.", + when: "Diagnoses a bug before continuing its open pull request. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information.", sender: Units(&[IMPLEMENT]), text: prompt::diagnosing_bugs( &issue, &prompt::continuation(&issue, BASE, BRANCH, Some(PR_URL)), + Some(prompt::SECURITY_REVIEW_MERGE_BASE), ), }, Prompt { @@ -746,6 +748,30 @@ mod tests { /// The golden-file test: the page carries the fragment as rendered now. /// The crates.io package leaves `site/` out, so it skips there. + #[test] + fn a_continuation_bug_prompt_names_the_merge_base_and_a_fresh_one_does_not() { + let prompts = prompts(); + let text = |id: &str| { + prompts + .iter() + .find(|prompt| prompt.id == id) + .unwrap_or_else(|| panic!("missing {id}")) + .text + .clone() + }; + let line = format!( + "Merge base commit: `{}`.", + prompt::SECURITY_REVIEW_MERGE_BASE + ); + for id in ["prompt-bug-continuation", "prompt-bug-continuation-pr"] { + let text = text(id); + assert!(text.contains(&line), "{id}: {text}"); + assert!(text.contains("Reproduce at the merge base"), "{id}"); + } + let fresh = text("prompt-bug-fresh"); + assert!(!fresh.contains("Merge base commit:"), "{fresh}"); + } + #[test] fn prompts_page_shows_what_the_binary_sends() { let site = Path::new(env!("CARGO_MANIFEST_DIR")).join("site"); diff --git a/src/run.rs b/src/run.rs index a5163d2d..ac155f72 100644 --- a/src/run.rs +++ b/src/run.rs @@ -572,7 +572,9 @@ impl<'a> Outside for LaunchAndGitHub<'a> { && !labels.has(crate::security::fixing::SECURITY_FIX) { opening.session = OpeningSession::BugDiagnosis; - opening.prompt = prompt::diagnosing_bugs(self.issue, &opening.prompt); + let merge_base = worktree.ahead_merge_base(base)?; + opening.prompt = + prompt::diagnosing_bugs(self.issue, &opening.prompt, merge_base.as_deref()); } } let mut pull_request = crate::pull_request::PullRequest::new( diff --git a/src/worktree.rs b/src/worktree.rs index 2cee57a7..3a3c9030 100644 --- a/src/worktree.rs +++ b/src/worktree.rs @@ -268,6 +268,19 @@ impl Worktree { }) } + /// The merge base with the published Base branch when this Issue branch + /// has commits that Base branch does not. `None` when this head is already + /// contained in the Base branch: a fresh branch, or a Continuation with + /// nothing beyond it. Bug diagnosis reproduces at this commit. + pub fn ahead_merge_base(&self, base: &str) -> Result> { + self.checkout.ordinary(&self.launch, |operation| { + let merge_base = + operation.run(&["merge-base", &format!("refs/remotes/origin/{base}"), "HEAD"])?; + let head = operation.head()?; + Ok((merge_base != head).then_some(merge_base)) + }) + } + /// Fetch and sample `origin/`, then merge that commit into the /// Issue branch. A clean outcome includes the selected commit for CI /// comparison, including when the merge had nothing to do. diff --git a/tests/bug_run.rs b/tests/bug_run.rs index 3eb663e5..57c6f842 100644 --- a/tests/bug_run.rs +++ b/tests/bug_run.rs @@ -14,6 +14,67 @@ gh pr create --base main --head issue-7 --title 'Fix bug' --body 'Closes #7' echo 'Bug reproduced: fixed' > "$FAKE_CLAUDE_FINAL_MESSAGE" "#; +/// Probes the checkout the way #1362 did, unless the prompt names a merge base. +/// An already-fixed branch then goes on to review and the pull request. +const DIAGNOSE_AT_NAMED_MERGE_BASE: &str = r#" +hand_back() { + gh fake labels 7 '["bug", "needs-info", "in-progress"]' + gh issue comment 7 --body '## Triage Notes' + printf '%s\n' 'Bug not reproduced: https://github.com/acme/widgets/issues/7#issuecomment-42' > "$FAKE_CLAUDE_FINAL_MESSAGE" +} +deliver_fix() { + reports=$(printf '%s\n' "$FAKE_CLAUDE_PROMPT" | sed -n 's/.*`\([^`]*\)\/standards\.md`.*/\1/p') + test -n "$reports" + printf 'Standards\n\nFiles read:\n- behavior.txt\n' > "$reports/standards.md" + printf 'Spec\n\nFiles read:\n- behavior.txt\n' > "$reports/spec.md" + printf '%s\n' '{"standards":0,"spec":0}' > "$reports/summary.json" + gh pr create --base main --head issue-7 --title 'Fix bug' --body 'Closes #7' + printf '%s\n' 'Bug reproduced: fixed' > "$FAKE_CLAUDE_FINAL_MESSAGE" +} +merge_base=$(printf '%s\n' "$FAKE_CLAUDE_PROMPT" | sed -n 's/^Merge base commit: `\(.*\)`\.$/\1/p') +if [ -n "$merge_base" ]; then + baseline=$(git show "$merge_base:behavior.txt") + if [ "$baseline" != broken ]; then + hand_back + exit 0 + fi + if [ "$(cat behavior.txt)" != fixed ]; then + printf '%s\n' fixed > behavior.txt + git add behavior.txt + git commit -q -m 'Fix bug' + fi + deliver_fix + exit 0 +fi +if [ "$(cat behavior.txt 2>/dev/null || true)" = broken ]; then + printf '%s\n' fixed > behavior.txt + git add behavior.txt + git commit -q -m 'Fix bug' + deliver_fix + exit 0 +fi +hand_back +"#; + +fn origin_merge_base(scenario: &Scenario, base: &str, branch: &str) -> String { + let output = std::process::Command::new("git") + .args([ + "-C", + &scenario.origin_dir().to_string_lossy(), + "merge-base", + &format!("refs/heads/{base}"), + &format!("refs/heads/{branch}"), + ]) + .output() + .unwrap(); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + String::from_utf8(output.stdout).unwrap().trim().to_string() +} + #[test] fn bug_runs_diagnose_before_implementation_in_fresh_runs_and_continuations() { for (continuing, label) in [(false, "bug"), (true, "BuG")] { @@ -218,3 +279,162 @@ echo 'Bug not reproduced: {url}' > "$FAKE_CLAUDE_FINAL_MESSAGE" assert_eq!(scenario.gh_state()["prs"], serde_json::json!([])); } } + +#[test] +fn a_continuation_reproduces_at_the_merge_base_and_delivers_the_existing_fix() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["bug", "ready-for-agent"]); + scenario.origin_has_commit("main", "behavior.txt", "broken\n", "Bug is present"); + scenario.origin_has_branch("issue-7", "main", &[]); + scenario.origin_has_commit("issue-7", "behavior.txt", "fixed\n", "Fix the bug"); + let merge_base = origin_merge_base(&scenario, "main", "issue-7"); + let history = scenario.origin_log("issue-7"); + scenario.agent_does(DIAGNOSE_AT_NAMED_MERGE_BASE); + + let result = scenario.run(&[&scenario.issue_url(7)]); + + assert_eq!( + result.stdout, "https://github.com/acme/widgets/pull/1\n", + "{}", + result.stderr + ); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let prompt = scenario.first_prompt(); + assert!( + prompt.contains(&format!("Merge base commit: `{merge_base}`.")), + "{prompt}" + ); + let calls = scenario.claude_calls(); + let skill = calls[0]["skill_files"]["thirdshift-diagnosing-bugs/SKILL.md"] + .as_str() + .unwrap(); + assert!(skill.contains("Reproduce at that merge base"), "{skill}"); + assert!(!scenario.issue_labels(7).contains(&"needs-info".to_string())); + assert!(scenario.gh_calls_of("issue", "comment").is_empty()); + assert!( + !result.stderr.contains("no review reports were left"), + "{}", + result.stderr + ); + assert_eq!(scenario.origin_log("issue-7"), history); +} + +#[test] +fn a_continuation_whose_merge_base_lacks_the_bug_still_needs_info() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["bug", "ready-for-agent"]); + scenario.origin_has_commit("main", "behavior.txt", "fine\n", "No bug"); + scenario.origin_has_branch("issue-7", "main", &[]); + scenario.origin_has_commit("issue-7", "notes.txt", "still fine\n", "Unrelated"); + let history = scenario.origin_log("issue-7"); + scenario.agent_does(DIAGNOSE_AT_NAMED_MERGE_BASE); + + let result = scenario.run(&[&scenario.issue_url(7)]); + + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + result.stdout, + "https://github.com/acme/widgets/issues/7#issuecomment-42\n" + ); + assert!( + result.stderr.contains("bug not reproduced"), + "{}", + result.stderr + ); + let prompt = scenario.first_prompt(); + let merge_base = origin_merge_base(&scenario, "main", "issue-7"); + assert!( + prompt.contains(&format!("Merge base commit: `{merge_base}`.")), + "{prompt}" + ); + assert_eq!(scenario.gh_state()["prs"], serde_json::json!([])); + assert_eq!(scenario.origin_log("issue-7"), history); + assert_eq!(scenario.issue_labels(7), ["bug", "needs-info"]); + assert!( + scenario + .gh_calls_of("issue", "comment") + .iter() + .any(|call| call.iter().any(|arg| arg.contains("Triage Notes"))) + ); +} + +#[test] +fn a_fresh_bug_run_reproduces_on_its_checkout_without_a_merge_base() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["bug", "ready-for-agent"]); + scenario.origin_has_commit("main", "behavior.txt", "broken\n", "Bug is present"); + scenario.agent_does(DIAGNOSE_AT_NAMED_MERGE_BASE); + + let result = scenario.run(&[&scenario.issue_url(7)]); + + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + result.stdout, "https://github.com/acme/widgets/pull/1\n", + "{}", + result.stderr + ); + let prompt = scenario.first_prompt(); + assert!(!prompt.contains("Merge base commit:"), "{prompt}"); + assert_eq!( + scenario.origin_file("issue-7", "behavior.txt").as_deref(), + Some("fixed\n") + ); +} + +#[test] +fn a_continuation_with_nothing_beyond_the_base_reproduces_on_its_checkout() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["bug", "ready-for-agent"]); + scenario.origin_has_commit("main", "behavior.txt", "broken\n", "Bug is present"); + scenario.origin_has_branch("issue-7", "main", &[]); + scenario.agent_does(DIAGNOSE_AT_NAMED_MERGE_BASE); + + let result = scenario.run(&[&scenario.issue_url(7)]); + + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + result.stdout, "https://github.com/acme/widgets/pull/1\n", + "{}", + result.stderr + ); + let prompt = scenario.first_prompt(); + assert!( + prompt.contains("You are continuing work on branch issue-7"), + "{prompt}" + ); + assert!(!prompt.contains("Merge base commit:"), "{prompt}"); + assert_eq!( + scenario.origin_file("issue-7", "behavior.txt").as_deref(), + Some("fixed\n") + ); +} + +#[test] +fn a_continuation_finishes_a_fix_the_branch_does_not_have_yet() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["bug", "ready-for-agent"]); + scenario.origin_has_commit("main", "behavior.txt", "broken\n", "Bug is present"); + scenario.origin_has_branch("issue-7", "main", &[]); + scenario.origin_has_commit("issue-7", "notes.txt", "not a fix\n", "Earlier work"); + scenario.agent_does(DIAGNOSE_AT_NAMED_MERGE_BASE); + + let result = scenario.run(&[&scenario.issue_url(7)]); + + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + result.stdout, "https://github.com/acme/widgets/pull/1\n", + "{}", + result.stderr + ); + let prompt = scenario.first_prompt(); + let merge_base = origin_merge_base(&scenario, "main", "issue-7"); + assert!( + prompt.contains(&format!("Merge base commit: `{merge_base}`.")), + "{prompt}" + ); + assert_eq!( + scenario.origin_file("issue-7", "behavior.txt").as_deref(), + Some("fixed\n") + ); + assert_eq!(scenario.origin_log("issue-7").unwrap()[0], "Fix bug"); +} From 19eb1047d485b6bdf69895b83d9228d4b979015f Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Mon, 26 Oct 2026 13:13:09 -0400 Subject: [PATCH 2/2] Keep Continuation diagnosis outcomes in the Factory skill --- prompts/bug-continuation-pr.md | 5 +---- prompts/bug-continuation.md | 5 +---- site/prompts/index.html | 12 +++--------- skills/thirdshift-diagnosing-bugs/SKILL.md | 2 +- src/prompt.rs | 8 ++------ src/prompts_page.rs | 4 ++-- 6 files changed, 10 insertions(+), 26 deletions(-) diff --git a/prompts/bug-continuation-pr.md b/prompts/bug-continuation-pr.md index 54170d23..96de265f 100644 --- a/prompts/bug-continuation-pr.md +++ b/prompts/bug-continuation-pr.md @@ -10,11 +10,8 @@ Use the `thirdshift-diagnosing-bugs` skill before changing code. Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below. Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: `. -This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, not on the branch, then check the fix on the branch. +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, then check the fix on the branch. Merge base commit: ``. -Use that commit as given; do not infer another from git history. -Hand back `needs-info` only when the bug does not reproduce at that merge base. -When it reproduces there and the branch already fixes it, that is the reproduced ending: continue to review and the pull request. Implementation after reproduction: /thirdshift-implement diff --git a/prompts/bug-continuation.md b/prompts/bug-continuation.md index 1ec412b7..30485d84 100644 --- a/prompts/bug-continuation.md +++ b/prompts/bug-continuation.md @@ -10,11 +10,8 @@ Use the `thirdshift-diagnosing-bugs` skill before changing code. Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below. Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: `. -This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, not on the branch, then check the fix on the branch. +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, then check the fix on the branch. Merge base commit: ``. -Use that commit as given; do not infer another from git history. -Hand back `needs-info` only when the bug does not reproduce at that merge base. -When it reproduces there and the branch already fixes it, that is the reproduced ending: continue to review and the pull request. Implementation after reproduction: /thirdshift-implement diff --git a/site/prompts/index.html b/site/prompts/index.html index 839ce664..c3f7eaf2 100644 --- a/site/prompts/index.html +++ b/site/prompts/index.html @@ -292,11 +292,8 @@

    Prompts and skills

    Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below. Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: <comment URL>`. -This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, not on the branch, then check the fix on the branch. +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, then check the fix on the branch. Merge base commit: `<merge base commit>`. -Use that commit as given; do not infer another from git history. -Hand back `needs-info` only when the bug does not reproduce at that merge base. -When it reproduces there and the branch already fixes it, that is the reproduced ending: continue to review and the pull request. Implementation after reproduction: /thirdshift-implement <Issue URL> @@ -338,11 +335,8 @@

    Prompts and skills

    Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below. Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: <comment URL>`. -This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, not on the branch, then check the fix on the branch. +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, then check the fix on the branch. Merge base commit: `<merge base commit>`. -Use that commit as given; do not infer another from git history. -Hand back `needs-info` only when the bug does not reproduce at that merge base. -When it reproduces there and the branch already fixes it, that is the reproduced ending: continue to review and the pull request. Implementation after reproduction: /thirdshift-implement <Issue URL> @@ -1096,7 +1090,7 @@

    Prompts and skills

    Before changing production code, turn the reproduction into a regression test at a seam that exercises the actual reported pattern. Run it and capture its failure for that reason. If no suitable seam can exercise the bug unattended, record that limitation through the not reproduced ending instead of guessing at a fix. -Make the smallest fix, watch the same test pass, then rerun the original reproduction and the repository's affected checks. Keep the regression test. When the Issue branch already fixes a bug you reproduced at the named merge base, keep that fix and its regression test. Remove temporary instrumentation and throwaway artifacts. Continue the Session prompt's implementation, review, commit and PR steps. In the PR's Evidence, show the exact command and its failing-before and passing-after results, and explain the confirmed cause. End the final message with the Session prompt's reproduced completion line only after those steps succeed. +Make the smallest fix, watch the same test pass, then rerun the original reproduction and the repository's affected checks. Keep the regression test. Remove temporary instrumentation and throwaway artifacts. Continue the Session prompt's implementation, review, commit and PR steps. In the PR's Evidence, show the exact command and its failing-before and passing-after results, and explain the confirmed cause. End the final message with the Session prompt's reproduced completion line only after those steps succeed. ## Not reproduced: needs information, no PR diff --git a/skills/thirdshift-diagnosing-bugs/SKILL.md b/skills/thirdshift-diagnosing-bugs/SKILL.md index 8fc3f18a..87bf5d24 100644 --- a/skills/thirdshift-diagnosing-bugs/SKILL.md +++ b/skills/thirdshift-diagnosing-bugs/SKILL.md @@ -37,7 +37,7 @@ Remove one input or step at a time, rerunning the loop, until every remaining el Before changing production code, turn the reproduction into a regression test at a seam that exercises the actual reported pattern. Run it and capture its failure for that reason. If no suitable seam can exercise the bug unattended, record that limitation through the not reproduced ending instead of guessing at a fix. -Make the smallest fix, watch the same test pass, then rerun the original reproduction and the repository's affected checks. Keep the regression test. When the Issue branch already fixes a bug you reproduced at the named merge base, keep that fix and its regression test. Remove temporary instrumentation and throwaway artifacts. Continue the Session prompt's implementation, review, commit and PR steps. In the PR's Evidence, show the exact command and its failing-before and passing-after results, and explain the confirmed cause. End the final message with the Session prompt's reproduced completion line only after those steps succeed. +Make the smallest fix, watch the same test pass, then rerun the original reproduction and the repository's affected checks. Keep the regression test. Remove temporary instrumentation and throwaway artifacts. Continue the Session prompt's implementation, review, commit and PR steps. In the PR's Evidence, show the exact command and its failing-before and passing-after results, and explain the confirmed cause. End the final message with the Session prompt's reproduced completion line only after those steps succeed. ## Not reproduced: needs information, no PR diff --git a/src/prompt.rs b/src/prompt.rs index e9a10068..0b0bde95 100644 --- a/src/prompt.rs +++ b/src/prompt.rs @@ -236,12 +236,8 @@ pub fn diagnosing_bugs(issue: &IssueUrl, implementation: &str, merge_base: Optio let baseline = match merge_base { Some(commit) => format!( "This Continuation's Issue branch is ahead of the Base branch. \ - Reproduce at the merge base, not on the branch, then check the fix on the branch.\n\ - Merge base commit: `{commit}`.\n\ - Use that commit as given; do not infer another from git history.\n\ - Hand back `needs-info` only when the bug does not reproduce at that merge base.\n\ - When it reproduces there and the branch already fixes it, that is the reproduced ending: \ - continue to review and the pull request.\n\n" + Reproduce at the merge base, then check the fix on the branch.\n\ + Merge base commit: `{commit}`.\n\n" ), None => String::new(), }; diff --git a/src/prompts_page.rs b/src/prompts_page.rs index 2bb2e3f7..5b36d6e4 100644 --- a/src/prompts_page.rs +++ b/src/prompts_page.rs @@ -746,8 +746,6 @@ mod tests { } } - /// The golden-file test: the page carries the fragment as rendered now. - /// The crates.io package leaves `site/` out, so it skips there. #[test] fn a_continuation_bug_prompt_names_the_merge_base_and_a_fresh_one_does_not() { let prompts = prompts(); @@ -772,6 +770,8 @@ mod tests { assert!(!fresh.contains("Merge base commit:"), "{fresh}"); } + /// The golden-file test: the page carries the fragment as rendered now. + /// The crates.io package leaves `site/` out, so it skips there. #[test] fn prompts_page_shows_what_the_binary_sends() { let site = Path::new(env!("CARGO_MANIFEST_DIR")).join("site");