diff --git a/GLOSSARY.md b/GLOSSARY.md index ac74027d..f344e21b 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -96,7 +96,7 @@ One invocation of thirdshift that does factory work: a **Ticket run** or **Spec _Avoid_: invocation, job **Ticket run**: -One invocation of the factory on a single issue that is not a **Spec**, from launch to cleanup, normally ending in one pull request ready for review, or a **Self-merge** when its command or the **User config** asks for one. A Ticket run on an issue labelled `bug` reproduces the bug before fixing it, in a fresh Ticket run or a **Continuation**, except for Security fixes and **Base fixes**, which have their own flow. If the bug is not reproduced, the Ticket run ends successfully with triage notes and `needs-info` on the issue, without opening or merging a pull request. Started directly on an **Issue URL**, by a **Spec run** for one of its **Tickets**, by an **Architect run** for the Ticket its **Architecture review** published, by a **Pickup run** for the **Ready issue** it took, or by a **Security run** for a **Security finding** it fixes. +One invocation of the factory on a single issue that is not a **Spec**, from launch to cleanup, normally ending in one pull request ready for review, or a **Self-merge** when its command or the **User config** asks for one. A Ticket run on an issue labelled `bug` reproduces the bug before fixing it, in a fresh Ticket run or a **Continuation**, except for Security fixes and **Base fixes**, which have their own flow. A Continuation whose **Issue branch** has commits beyond the **Base branch** reproduces at their merge base, then checks the fix on the branch; a fix already there still goes on to review and the pull request. If the bug is not reproduced, including when it is absent at that merge base, the Ticket run ends successfully with triage notes and `needs-info` on the issue, without opening or merging a pull request. Started directly on an **Issue URL**, by a **Spec run** for one of its **Tickets**, by an **Architect run** for the Ticket its **Architecture review** published, by a **Pickup run** for the **Ready issue** it took, or by a **Security run** for a **Security finding** it fixes. _Avoid_: Run, Merge run (former names), auto-merge (GitHub's own feature) **Spec run**: diff --git a/prompts/README.md b/prompts/README.md index 0c61239c..8e4f13dd 100644 --- a/prompts/README.md +++ b/prompts/README.md @@ -29,8 +29,8 @@ With `harness codex`, each session runs `codex exec --json --dangerously-bypass- | Code review | In an Eval Trial, reviews the prepared historical diff with the production code-review Factory skill, under the implement session's sub-agent rules. Writes reports without fixing or publishing anything. | [code-review.md](code-review.md) | | Spec review | In a Spec run, starts the Spec review once every Ticket has landed on the Spec branch, before the Spec PR, a draft until then, is marked ready. | [spec-review.md](spec-review.md) | | Bug Ticket run, fresh | Starts the implement session on an issue labelled bug: diagnose before changing code, then implement only if reproduced. Security fixes and Base fixes keep their own flow. | [bug-fresh.md](bug-fresh.md) | -| Bug Ticket run, Continuation without a pull request | Diagnoses a bug in a Continuation with no pull request before changing code. A not reproduced ending hands the issue back for information without a PR. | [bug-continuation.md](bug-continuation.md) | -| Bug Ticket run, Continuation with an open pull request | Diagnoses a bug before continuing its open pull request. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information. | [bug-continuation-pr.md](bug-continuation-pr.md) | +| Bug Ticket run, Continuation without a pull request | Diagnoses a bug in a Continuation with no pull request before changing code. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending hands the issue back for information without a PR. | [bug-continuation.md](bug-continuation.md) | +| Bug Ticket run, Continuation with an open pull request | Diagnoses a bug before continuing its open pull request. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information. | [bug-continuation-pr.md](bug-continuation-pr.md) | | Architecture review | Starts the Architecture review, the session an Architect run opens with, in a worktree at the head of the Base branch. The line naming the focus is left out when the command gives none. | [architecture-review.md](architecture-review.md) | | Security review | When enabled, runs one guidance review after a Ticket run's opening session or a Spec run's Spec review, before Delivery pushes and enters the Repair loop. Each Ticket run and Base fix gets one too. Introduced findings, refusals or incomplete reviews hold Self-merge. | [security-review.md](security-review.md) | | Security audit | Starts the report-only Security audit in a throwaway worktree at origin's Base branch head. The threat-model line is included when a conventional document exists; artifacts stay under the repository's audit root. | [security-audit.md](security-audit.md) | diff --git a/prompts/bug-continuation-pr.md b/prompts/bug-continuation-pr.md index 9c3c75b5..96de265f 100644 --- a/prompts/bug-continuation-pr.md +++ b/prompts/bug-continuation-pr.md @@ -2,7 +2,7 @@ # Bug Ticket run, Continuation with an open pull request -Diagnoses a bug before continuing its open pull request. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information. +Diagnoses a bug before continuing its open pull request. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information. ``` /thirdshift-diagnosing-bugs @@ -10,6 +10,9 @@ Use the `thirdshift-diagnosing-bugs` skill before changing code. Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below. Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: `. +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, then check the fix on the branch. +Merge base commit: ``. + Implementation after reproduction: /thirdshift-implement diff --git a/prompts/bug-continuation.md b/prompts/bug-continuation.md index d17da129..30485d84 100644 --- a/prompts/bug-continuation.md +++ b/prompts/bug-continuation.md @@ -2,7 +2,7 @@ # Bug Ticket run, Continuation without a pull request -Diagnoses a bug in a Continuation with no pull request before changing code. A not reproduced ending hands the issue back for information without a PR. +Diagnoses a bug in a Continuation with no pull request before changing code. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending hands the issue back for information without a PR. ``` /thirdshift-diagnosing-bugs @@ -10,6 +10,9 @@ Use the `thirdshift-diagnosing-bugs` skill before changing code. Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below. Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: `. +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, then check the fix on the branch. +Merge base commit: ``. + Implementation after reproduction: /thirdshift-implement diff --git a/site/architecture/index.html b/site/architecture/index.html index 9b822487..3de98c5d 100644 --- a/site/architecture/index.html +++ b/site/architecture/index.html @@ -226,7 +226,7 @@

Architecture

  • Pre-flight checks stop a Ticket run before anything is created: the Issue URL must match the clone's origin, the issue must be open, and the Base branch must be on origin.
  • The Claim labels the issue in-progress, so the issue list shows what the factory is working on.
  • A Worktree gives the Ticket run its own checkout beside your clone, so yours is never touched.
  • -
  • Delivery starts the implement session. On issues labelled bug, diagnosis comes first: a reproduced bug gets a failing regression test and fix; an unreproduced bug ends successfully with triage notes and needs-info, without PR delivery. Security fixes and Base fixes keep their own flow. Otherwise Delivery pushes the branch and checks that the pull request exists and targets the Base branch. thirdshift, not the agent, marks it ready.
  • +
  • Delivery starts the implement session. On issues labelled bug, diagnosis comes first: a reproduced bug gets a failing regression test and fix; a Continuation whose Issue branch is ahead of the Base branch reproduces at their merge base, and a fix already on the branch goes on to review and the pull request; an unreproduced bug ends successfully with triage notes and needs-info, without PR delivery. Security fixes and Base fixes keep their own flow. Otherwise Delivery pushes the branch and checks that the pull request exists and targets the Base branch. thirdshift, not the agent, marks it ready.
  • Keeping it green merges the Base branch in and watches CI. A conflict or the branch's own red check starts a Repair, at most 5 a Ticket run.
  • The Self-merge, when asked, merges exactly the head commit whose CI thirdshift watched.
  • Cleaning up removes the Worktree. A Failed ticket run pushes its work first and turns the pull request back into a draft.
  • diff --git a/site/how-it-works/index.html b/site/how-it-works/index.html index 6ffde1fc..2c41808e 100644 --- a/site/how-it-works/index.html +++ b/site/how-it-works/index.html @@ -58,7 +58,7 @@

    How thirdshift works

    01 / The human handoff

    Start with work you can describe

    The Day shift is you: settle what should change, what success looks like, and what is out of scope. Put that in a GitHub issue. One session's worth of work can stand alone; a larger Spec is an issue with Tickets as GitHub sub-issues. We'll follow a standalone issue first.

    -

    For issues labelled bug, a fresh Ticket run or Continuation uses /thirdshift-diagnosing-bugs before changing code. It captures a failing regression test, then fixes it. If the bug cannot be reproduced, it comments with triage notes, replaces ready-for-agent with needs-info and ends successfully with the comment link, without opening or merging a pull request. Security fixes and Base fixes keep their own flow.

    +

    For issues labelled bug, a fresh Ticket run or Continuation uses /thirdshift-diagnosing-bugs before changing code. A fresh run, or a Continuation with nothing on the Issue branch beyond the Base branch, reproduces on that checkout. A Continuation whose Issue branch is ahead of the Base branch reproduces at their merge base, then checks the fix on the branch: a bug there, with the fix already on the branch, goes on to review and the pull request. Otherwise it captures a failing regression test, then fixes it. If the bug cannot be reproduced at that checkout, it comments with triage notes, replaces ready-for-agent with needs-info and ends successfully with the comment link, without opening or merging a pull request. Security fixes and Base fixes keep their own flow.

    Human intent becomes the agent's brief
      diff --git a/site/index.html b/site/index.html index 8e12936b..9fefc532 100644 --- a/site/index.html +++ b/site/index.html @@ -90,7 +90,7 @@

      Your issues become pull requests overnight.

      Third shift

      Agents, with headless copies of the same skills

      1. /thirdshift-implement
      2. /thirdshift-code-review
      3. /thirdshift-pr

      thirdshift takes an Issue URL: one issue starts a Ticket run; an issue with sub-issues starts a Spec run. Its Factory skills, adapted to run with no human in the loop, implement the issue with /thirdshift-implement, review the work with /thirdshift-code-review and write the pull request with /thirdshift-pr.

      -

      Issues labelled bug first use /thirdshift-diagnosing-bugs to reproduce the reported failure and write a failing regression test before fixing it. If it cannot reproduce the bug, the Ticket run leaves triage notes and needs-info on the issue, ending successfully without a new pull request. Security fixes and Base fixes keep their own flow.

      +

      Issues labelled bug first use /thirdshift-diagnosing-bugs to reproduce the reported failure and write a failing regression test before fixing it. A Continuation whose Issue branch is ahead of the Base branch reproduces at their merge base, and a fix already on the branch still goes on to review and the pull request. If it cannot reproduce the bug, the Ticket run leaves triage notes and needs-info on the issue, ending successfully without a new pull request. Security fixes and Base fixes keep their own flow.

      Read every prompt and skill the agents get →

    diff --git a/site/prompts/index.html b/site/prompts/index.html index b6999d0c..c3f7eaf2 100644 --- a/site/prompts/index.html +++ b/site/prompts/index.html @@ -285,13 +285,16 @@

    Prompts and skills

    Bug Ticket run, Continuation without a pull request

    -

    Diagnoses a bug in a Continuation with no pull request before changing code. A not reproduced ending hands the issue back for information without a PR.

    +

    Diagnoses a bug in a Continuation with no pull request before changing code. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending hands the issue back for information without a PR.

    Sent by 2 · M · Implement

    /thirdshift-diagnosing-bugs <Issue URL>
     Use the `thirdshift-diagnosing-bugs` skill before changing code.
     Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below.
     Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: <comment URL>`.
     
    +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, then check the fix on the branch.
    +Merge base commit: `<merge base commit>`.
    +
     Implementation after reproduction:
     /thirdshift-implement <Issue URL>
     
    @@ -325,13 +328,16 @@ 

    Prompts and skills

    Bug Ticket run, Continuation with an open pull request

    -

    Diagnoses a bug before continuing its open pull request. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information.

    +

    Diagnoses a bug before continuing its open pull request. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information.

    Sent by 2 · M · Implement

    /thirdshift-diagnosing-bugs <Issue URL>
     Use the `thirdshift-diagnosing-bugs` skill before changing code.
     Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below.
     Completion lines (the final message's last line): reproduced: `Bug reproduced: fixed`; not reproduced: `Bug not reproduced: <comment URL>`.
     
    +This Continuation's Issue branch is ahead of the Base branch. Reproduce at the merge base, then check the fix on the branch.
    +Merge base commit: `<merge base commit>`.
    +
     Implementation after reproduction:
     /thirdshift-implement <Issue URL>
     
    @@ -1058,9 +1064,21 @@ 

    Prompts and skills

    Redact secrets from commands, output and captured artifacts. Build loops against environment variables so credentials stay out of the evidence. +## Where to reproduce + +The Session prompt says which checkout the bug is reproduced on. + +When it names a merge base, this is a Continuation whose Issue branch already has commits beyond the Base branch. Reproduce at that merge base. Use the commit the prompt names; do not choose another from git history. Then check the fix on the Issue branch. + +- The bug reproduces at the merge base and the branch already fixes it: take the reproduced ending. Do not rewrite the fix. Continue to the Session prompt's implementation steps, including review and the pull request. +- The bug reproduces at the merge base and the branch does not fix it: take the reproduced ending, finish the fix, then continue to those steps. +- The bug does not reproduce at the merge base: take the not reproduced ending. A passing check on the branch is not enough to hand the issue back. + +A fresh bug Ticket run, or a Continuation whose Session prompt names no merge base, reproduces against the checkout you are in. + ## 1. Build a tight feedback loop -Spend most of the diagnosis effort here. Read enough code to find a seam that reaches the reported symptom, then run an unattended pass/fail command against the current code. Prefer a focused test; a CLI fixture, HTTP replay, headless browser script or throwaway harness can help establish reproduction. Assert the exact reported symptom, rather than simply checking that execution succeeds. A setup or compilation failure is not reproduction. +Spend most of the diagnosis effort here. Read enough code to find a seam that reaches the reported symptom, then run an unattended pass/fail command against the checkout named under Where to reproduce. Prefer a focused test; a CLI fixture, HTTP replay, headless browser script or throwaway harness can help establish reproduction. Assert the exact reported symptom, rather than simply checking that execution succeeds. A setup or compilation failure is not reproduction. Tighten the loop: narrow inputs, pin time and seeds, and remove unrelated setup. For intermittent bugs, repeat the trigger and establish a reproducible failure rate. Record the command, output and conditions. Proceed only once the loop actually fails for the reported reason. If it does not, take the not reproduced ending. diff --git a/skills/thirdshift-diagnosing-bugs/SKILL.md b/skills/thirdshift-diagnosing-bugs/SKILL.md index 900376c0..87bf5d24 100644 --- a/skills/thirdshift-diagnosing-bugs/SKILL.md +++ b/skills/thirdshift-diagnosing-bugs/SKILL.md @@ -11,9 +11,21 @@ Run headless: make routine choices yourself and finish with one of the two endin Redact secrets from commands, output and captured artifacts. Build loops against environment variables so credentials stay out of the evidence. +## Where to reproduce + +The Session prompt says which checkout the bug is reproduced on. + +When it names a merge base, this is a Continuation whose Issue branch already has commits beyond the Base branch. Reproduce at that merge base. Use the commit the prompt names; do not choose another from git history. Then check the fix on the Issue branch. + +- The bug reproduces at the merge base and the branch already fixes it: take the reproduced ending. Do not rewrite the fix. Continue to the Session prompt's implementation steps, including review and the pull request. +- The bug reproduces at the merge base and the branch does not fix it: take the reproduced ending, finish the fix, then continue to those steps. +- The bug does not reproduce at the merge base: take the not reproduced ending. A passing check on the branch is not enough to hand the issue back. + +A fresh bug Ticket run, or a Continuation whose Session prompt names no merge base, reproduces against the checkout you are in. + ## 1. Build a tight feedback loop -Spend most of the diagnosis effort here. Read enough code to find a seam that reaches the reported symptom, then run an unattended pass/fail command against the current code. Prefer a focused test; a CLI fixture, HTTP replay, headless browser script or throwaway harness can help establish reproduction. Assert the exact reported symptom, rather than simply checking that execution succeeds. A setup or compilation failure is not reproduction. +Spend most of the diagnosis effort here. Read enough code to find a seam that reaches the reported symptom, then run an unattended pass/fail command against the checkout named under Where to reproduce. Prefer a focused test; a CLI fixture, HTTP replay, headless browser script or throwaway harness can help establish reproduction. Assert the exact reported symptom, rather than simply checking that execution succeeds. A setup or compilation failure is not reproduction. Tighten the loop: narrow inputs, pin time and seeds, and remove unrelated setup. For intermittent bugs, repeat the trigger and establish a reproducible failure rate. Record the command, output and conditions. Proceed only once the loop actually fails for the reported reason. If it does not, take the not reproduced ending. diff --git a/src/prompt.rs b/src/prompt.rs index 730ade4f..0b0bde95 100644 --- a/src/prompt.rs +++ b/src/prompt.rs @@ -229,12 +229,24 @@ pub const BUG_REPRODUCED_LINE: &str = "Bug reproduced: fixed"; /// Diagnose first on a bug Ticket run, retaining its fresh or Continuation facts /// and its implementation steps for the reproduced ending only. -pub fn diagnosing_bugs(issue: &IssueUrl, implementation: &str) -> String { +/// +/// `merge_base` is set when a Continuation's Issue branch has commits the Base +/// branch does not. The session reproduces there, then checks the fix on the branch. +pub fn diagnosing_bugs(issue: &IssueUrl, implementation: &str, merge_base: Option<&str>) -> String { + let baseline = match merge_base { + Some(commit) => format!( + "This Continuation's Issue branch is ahead of the Base branch. \ + Reproduce at the merge base, then check the fix on the branch.\n\ + Merge base commit: `{commit}`.\n\n" + ), + None => String::new(), + }; format!( "/thirdshift-diagnosing-bugs {url}\n\ Use the `thirdshift-diagnosing-bugs` skill before changing code.\n\ Follow the skill to one of its two endings. Its not reproduced ending replaces the implementation steps below.\n\ Completion lines (the final message's last line): reproduced: `{BUG_REPRODUCED_LINE}`; not reproduced: `{BUG_NOT_REPRODUCED_LINE}`.\n\n\ + {baseline}\ Implementation after reproduction:\n{implementation}", url = issue.url, ) diff --git a/src/prompts_page.rs b/src/prompts_page.rs index 9616e563..5b36d6e4 100644 --- a/src/prompts_page.rs +++ b/src/prompts_page.rs @@ -198,26 +198,28 @@ fn prompts() -> Vec { title: "Bug Ticket run, fresh", when: "Starts the implement session on an issue labelled bug: diagnose before changing code, then implement only if reproduced. Security fixes and Base fixes keep their own flow.", sender: Units(&[IMPLEMENT]), - text: prompt::diagnosing_bugs(&issue, &prompt::fresh(&issue, BASE, BRANCH)), + text: prompt::diagnosing_bugs(&issue, &prompt::fresh(&issue, BASE, BRANCH), None), }, Prompt { id: "prompt-bug-continuation", title: "Bug Ticket run, Continuation without a pull request", - when: "Diagnoses a bug in a Continuation with no pull request before changing code. A not reproduced ending hands the issue back for information without a PR.", + when: "Diagnoses a bug in a Continuation with no pull request before changing code. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending hands the issue back for information without a PR.", sender: Units(&[IMPLEMENT]), text: prompt::diagnosing_bugs( &issue, &prompt::continuation(&issue, BASE, BRANCH, None), + Some(prompt::SECURITY_REVIEW_MERGE_BASE), ), }, Prompt { id: "prompt-bug-continuation-pr", title: "Bug Ticket run, Continuation with an open pull request", - when: "Diagnoses a bug before continuing its open pull request. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information.", + when: "Diagnoses a bug before continuing its open pull request. When the Issue branch is ahead of the Base branch, the prompt names their merge base and reproduction happens there. A not reproduced ending leaves the existing PR unmerged and hands the issue back for information.", sender: Units(&[IMPLEMENT]), text: prompt::diagnosing_bugs( &issue, &prompt::continuation(&issue, BASE, BRANCH, Some(PR_URL)), + Some(prompt::SECURITY_REVIEW_MERGE_BASE), ), }, Prompt { @@ -744,6 +746,30 @@ mod tests { } } + #[test] + fn a_continuation_bug_prompt_names_the_merge_base_and_a_fresh_one_does_not() { + let prompts = prompts(); + let text = |id: &str| { + prompts + .iter() + .find(|prompt| prompt.id == id) + .unwrap_or_else(|| panic!("missing {id}")) + .text + .clone() + }; + let line = format!( + "Merge base commit: `{}`.", + prompt::SECURITY_REVIEW_MERGE_BASE + ); + for id in ["prompt-bug-continuation", "prompt-bug-continuation-pr"] { + let text = text(id); + assert!(text.contains(&line), "{id}: {text}"); + assert!(text.contains("Reproduce at the merge base"), "{id}"); + } + let fresh = text("prompt-bug-fresh"); + assert!(!fresh.contains("Merge base commit:"), "{fresh}"); + } + /// The golden-file test: the page carries the fragment as rendered now. /// The crates.io package leaves `site/` out, so it skips there. #[test] diff --git a/src/run.rs b/src/run.rs index a5163d2d..ac155f72 100644 --- a/src/run.rs +++ b/src/run.rs @@ -572,7 +572,9 @@ impl<'a> Outside for LaunchAndGitHub<'a> { && !labels.has(crate::security::fixing::SECURITY_FIX) { opening.session = OpeningSession::BugDiagnosis; - opening.prompt = prompt::diagnosing_bugs(self.issue, &opening.prompt); + let merge_base = worktree.ahead_merge_base(base)?; + opening.prompt = + prompt::diagnosing_bugs(self.issue, &opening.prompt, merge_base.as_deref()); } } let mut pull_request = crate::pull_request::PullRequest::new( diff --git a/src/worktree.rs b/src/worktree.rs index e49a86f0..1105c286 100644 --- a/src/worktree.rs +++ b/src/worktree.rs @@ -268,6 +268,19 @@ impl Worktree { }) } + /// The merge base with the published Base branch when this Issue branch + /// has commits that Base branch does not. `None` when this head is already + /// contained in the Base branch: a fresh branch, or a Continuation with + /// nothing beyond it. Bug diagnosis reproduces at this commit. + pub fn ahead_merge_base(&self, base: &str) -> Result> { + self.checkout.ordinary(&self.launch, |operation| { + let merge_base = + operation.run(&["merge-base", &format!("refs/remotes/origin/{base}"), "HEAD"])?; + let head = operation.head()?; + Ok((merge_base != head).then_some(merge_base)) + }) + } + /// Fetch and sample `origin/`, then merge that commit into the /// Issue branch. A clean outcome includes the selected commit for CI /// comparison, including when the merge had nothing to do. diff --git a/tests/bug_run.rs b/tests/bug_run.rs index 3eb663e5..57c6f842 100644 --- a/tests/bug_run.rs +++ b/tests/bug_run.rs @@ -14,6 +14,67 @@ gh pr create --base main --head issue-7 --title 'Fix bug' --body 'Closes #7' echo 'Bug reproduced: fixed' > "$FAKE_CLAUDE_FINAL_MESSAGE" "#; +/// Probes the checkout the way #1362 did, unless the prompt names a merge base. +/// An already-fixed branch then goes on to review and the pull request. +const DIAGNOSE_AT_NAMED_MERGE_BASE: &str = r#" +hand_back() { + gh fake labels 7 '["bug", "needs-info", "in-progress"]' + gh issue comment 7 --body '## Triage Notes' + printf '%s\n' 'Bug not reproduced: https://github.com/acme/widgets/issues/7#issuecomment-42' > "$FAKE_CLAUDE_FINAL_MESSAGE" +} +deliver_fix() { + reports=$(printf '%s\n' "$FAKE_CLAUDE_PROMPT" | sed -n 's/.*`\([^`]*\)\/standards\.md`.*/\1/p') + test -n "$reports" + printf 'Standards\n\nFiles read:\n- behavior.txt\n' > "$reports/standards.md" + printf 'Spec\n\nFiles read:\n- behavior.txt\n' > "$reports/spec.md" + printf '%s\n' '{"standards":0,"spec":0}' > "$reports/summary.json" + gh pr create --base main --head issue-7 --title 'Fix bug' --body 'Closes #7' + printf '%s\n' 'Bug reproduced: fixed' > "$FAKE_CLAUDE_FINAL_MESSAGE" +} +merge_base=$(printf '%s\n' "$FAKE_CLAUDE_PROMPT" | sed -n 's/^Merge base commit: `\(.*\)`\.$/\1/p') +if [ -n "$merge_base" ]; then + baseline=$(git show "$merge_base:behavior.txt") + if [ "$baseline" != broken ]; then + hand_back + exit 0 + fi + if [ "$(cat behavior.txt)" != fixed ]; then + printf '%s\n' fixed > behavior.txt + git add behavior.txt + git commit -q -m 'Fix bug' + fi + deliver_fix + exit 0 +fi +if [ "$(cat behavior.txt 2>/dev/null || true)" = broken ]; then + printf '%s\n' fixed > behavior.txt + git add behavior.txt + git commit -q -m 'Fix bug' + deliver_fix + exit 0 +fi +hand_back +"#; + +fn origin_merge_base(scenario: &Scenario, base: &str, branch: &str) -> String { + let output = std::process::Command::new("git") + .args([ + "-C", + &scenario.origin_dir().to_string_lossy(), + "merge-base", + &format!("refs/heads/{base}"), + &format!("refs/heads/{branch}"), + ]) + .output() + .unwrap(); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + String::from_utf8(output.stdout).unwrap().trim().to_string() +} + #[test] fn bug_runs_diagnose_before_implementation_in_fresh_runs_and_continuations() { for (continuing, label) in [(false, "bug"), (true, "BuG")] { @@ -218,3 +279,162 @@ echo 'Bug not reproduced: {url}' > "$FAKE_CLAUDE_FINAL_MESSAGE" assert_eq!(scenario.gh_state()["prs"], serde_json::json!([])); } } + +#[test] +fn a_continuation_reproduces_at_the_merge_base_and_delivers_the_existing_fix() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["bug", "ready-for-agent"]); + scenario.origin_has_commit("main", "behavior.txt", "broken\n", "Bug is present"); + scenario.origin_has_branch("issue-7", "main", &[]); + scenario.origin_has_commit("issue-7", "behavior.txt", "fixed\n", "Fix the bug"); + let merge_base = origin_merge_base(&scenario, "main", "issue-7"); + let history = scenario.origin_log("issue-7"); + scenario.agent_does(DIAGNOSE_AT_NAMED_MERGE_BASE); + + let result = scenario.run(&[&scenario.issue_url(7)]); + + assert_eq!( + result.stdout, "https://github.com/acme/widgets/pull/1\n", + "{}", + result.stderr + ); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let prompt = scenario.first_prompt(); + assert!( + prompt.contains(&format!("Merge base commit: `{merge_base}`.")), + "{prompt}" + ); + let calls = scenario.claude_calls(); + let skill = calls[0]["skill_files"]["thirdshift-diagnosing-bugs/SKILL.md"] + .as_str() + .unwrap(); + assert!(skill.contains("Reproduce at that merge base"), "{skill}"); + assert!(!scenario.issue_labels(7).contains(&"needs-info".to_string())); + assert!(scenario.gh_calls_of("issue", "comment").is_empty()); + assert!( + !result.stderr.contains("no review reports were left"), + "{}", + result.stderr + ); + assert_eq!(scenario.origin_log("issue-7"), history); +} + +#[test] +fn a_continuation_whose_merge_base_lacks_the_bug_still_needs_info() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["bug", "ready-for-agent"]); + scenario.origin_has_commit("main", "behavior.txt", "fine\n", "No bug"); + scenario.origin_has_branch("issue-7", "main", &[]); + scenario.origin_has_commit("issue-7", "notes.txt", "still fine\n", "Unrelated"); + let history = scenario.origin_log("issue-7"); + scenario.agent_does(DIAGNOSE_AT_NAMED_MERGE_BASE); + + let result = scenario.run(&[&scenario.issue_url(7)]); + + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + result.stdout, + "https://github.com/acme/widgets/issues/7#issuecomment-42\n" + ); + assert!( + result.stderr.contains("bug not reproduced"), + "{}", + result.stderr + ); + let prompt = scenario.first_prompt(); + let merge_base = origin_merge_base(&scenario, "main", "issue-7"); + assert!( + prompt.contains(&format!("Merge base commit: `{merge_base}`.")), + "{prompt}" + ); + assert_eq!(scenario.gh_state()["prs"], serde_json::json!([])); + assert_eq!(scenario.origin_log("issue-7"), history); + assert_eq!(scenario.issue_labels(7), ["bug", "needs-info"]); + assert!( + scenario + .gh_calls_of("issue", "comment") + .iter() + .any(|call| call.iter().any(|arg| arg.contains("Triage Notes"))) + ); +} + +#[test] +fn a_fresh_bug_run_reproduces_on_its_checkout_without_a_merge_base() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["bug", "ready-for-agent"]); + scenario.origin_has_commit("main", "behavior.txt", "broken\n", "Bug is present"); + scenario.agent_does(DIAGNOSE_AT_NAMED_MERGE_BASE); + + let result = scenario.run(&[&scenario.issue_url(7)]); + + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + result.stdout, "https://github.com/acme/widgets/pull/1\n", + "{}", + result.stderr + ); + let prompt = scenario.first_prompt(); + assert!(!prompt.contains("Merge base commit:"), "{prompt}"); + assert_eq!( + scenario.origin_file("issue-7", "behavior.txt").as_deref(), + Some("fixed\n") + ); +} + +#[test] +fn a_continuation_with_nothing_beyond_the_base_reproduces_on_its_checkout() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["bug", "ready-for-agent"]); + scenario.origin_has_commit("main", "behavior.txt", "broken\n", "Bug is present"); + scenario.origin_has_branch("issue-7", "main", &[]); + scenario.agent_does(DIAGNOSE_AT_NAMED_MERGE_BASE); + + let result = scenario.run(&[&scenario.issue_url(7)]); + + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + result.stdout, "https://github.com/acme/widgets/pull/1\n", + "{}", + result.stderr + ); + let prompt = scenario.first_prompt(); + assert!( + prompt.contains("You are continuing work on branch issue-7"), + "{prompt}" + ); + assert!(!prompt.contains("Merge base commit:"), "{prompt}"); + assert_eq!( + scenario.origin_file("issue-7", "behavior.txt").as_deref(), + Some("fixed\n") + ); +} + +#[test] +fn a_continuation_finishes_a_fix_the_branch_does_not_have_yet() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["bug", "ready-for-agent"]); + scenario.origin_has_commit("main", "behavior.txt", "broken\n", "Bug is present"); + scenario.origin_has_branch("issue-7", "main", &[]); + scenario.origin_has_commit("issue-7", "notes.txt", "not a fix\n", "Earlier work"); + scenario.agent_does(DIAGNOSE_AT_NAMED_MERGE_BASE); + + let result = scenario.run(&[&scenario.issue_url(7)]); + + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + result.stdout, "https://github.com/acme/widgets/pull/1\n", + "{}", + result.stderr + ); + let prompt = scenario.first_prompt(); + let merge_base = origin_merge_base(&scenario, "main", "issue-7"); + assert!( + prompt.contains(&format!("Merge base commit: `{merge_base}`.")), + "{prompt}" + ); + assert_eq!( + scenario.origin_file("issue-7", "behavior.txt").as_deref(), + Some("fixed\n") + ); + assert_eq!(scenario.origin_log("issue-7").unwrap()[0], "Fix bug"); +}