From d9f5838f4e888693c98dfd1d6c2af79092d89278 Mon Sep 17 00:00:00 2001 From: Benjamin Hellmich <97835383+BellmeJoe@users.noreply.github.com> Date: Fri, 31 Jul 2026 17:11:25 +0300 Subject: [PATCH 01/11] =?UTF-8?q?docs(incidents):=20draft=20incident=20001?= =?UTF-8?q?=20(dirty-tree=20residue=20swept=20into=20commit)=20=E2=80=94?= =?UTF-8?q?=20staged,=20not=20published?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...01-dirty-tree-residue-swept-into-commit.md | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 docs/incidents/001-dirty-tree-residue-swept-into-commit.md diff --git a/docs/incidents/001-dirty-tree-residue-swept-into-commit.md b/docs/incidents/001-dirty-tree-residue-swept-into-commit.md new file mode 100644 index 0000000..75c2e77 --- /dev/null +++ b/docs/incidents/001-dirty-tree-residue-swept-into-commit.md @@ -0,0 +1,43 @@ +# Incident 001 — prior-cycle residue swept into a live commit by an agent's fallback path + +**Source:** a public GitHub issue filed by a maintainer against their own AI-agent-driven commit pipeline, describing a real shipped defect. Anonymized here at the maintainer's-consent bar (no consent on file yet) — no repo, org, or handle named. Reconstructed independently and run through getAdvantage's live gate; not the maintainer's own words beyond the class of failure. + +## What happened + +An agent-driven pipeline runs a multi-step cycle: plan, implement, then commit. A prior cycle had been aborted partway through — it left the working tree dirty: a test file gutted from hundreds of lines down to a handful, and an unrelated view edit that was never finished or reverted. + +The next cycle started on top of that dirty tree. Its own changes were in scope and correct — a later transcript audit confirmed the cycle's own diff touched only the files it was supposed to touch. But the automated commit step hit a conflict ("patch does not apply") against the leftover residue and fell back to a broad `git add -A`-style commit. That swept the gutted test file and the unrelated broken edit into the same commit as the cycle's real work. + +CI ran green — the reduced test file still passed, it just tested less. The commit merged. The unrelated broken edit shipped as a live regression that caused data loss, and was only caught and hand-fixed after the fact. + +The maintainer's own root-cause note: an older, human-driven version of this pipeline used to abort outright on a dirty working tree. That doctrine did not carry over when the pipeline was mechanized — nothing re-asserted it, so nothing stopped the fallback from running on a dirty tree. + +## Reconstruction: what the gate does with this + +Reproduced the shape of the failure locally: a clean initial commit, then a file modified in place without being committed or staged for review (standing in for the prior cycle's residue), then ran `getadvantage check` cold, exactly as it would run in CI or a pre-commit hook. + +``` +✗ Dirty-tree guard — 1 tracked file modified/staged — a 'vercel --prod' would ship this unintended work. + M test/sample.test.js + Commit, stash, or revert before shipping. Deploy from a clean detached worktree of the intended commit. + +Verdict + ✓ 6 ⚠ 1 ✗ 1 – 2 skipped + + NO-GO — 1 blocking issue. Do not ship until these are clear. +``` + +Wired into the commit-pr entry point (the exact point this incident's fallback ran), this is a hard halt: NO-GO, before the sweep, before CI, before merge. It names the modified file directly, so the operator sees the residue instead of a green check hiding it. + +## What it would NOT have caught + +Being direct about the edges, because a gate that only advertises catches isn't trustworthy: + +- **A residue file that gets legitimately committed first.** If the leftover edit had landed in its own small commit before the next cycle started, the tree reads clean at the next check — the guard checks working-tree hygiene at the moment it runs, not the history of how a file got there. +- **A logic bug inside a file that was always meant to change.** The guard flags *unexpected* modifications; it has no opinion on whether an intended, cleanly-committed change is correct. The dirty-tree class and a bad-logic class are different failures — this incident happened to be the first, not the second. +- **Anything outside where it's wired in.** The gate only stops a ship if something actually calls it — ad hoc, uninstalled-in-CI runs give zero protection. This is the exact gap [invisible-mode / hook installs](../launch/SOCIAL-MEDIA-PACK.md) closes: the check runs automatically on every agent session instead of depending on someone remembering to call it. +- **A deliberate, reviewed override.** getAdvantage supports a tracked ignore path for a specific finding — that is a feature (disclosed, reviewable, never silent), but it also means a rushed or careless override can wave through the exact thing the guard was built to stop. The gate makes the override visible; it doesn't prevent someone from clicking past it. + +## Try it + +`npx getadvantage check` on your own repo takes under a minute and reads nothing but your local tree — no network, no account. If it mis-fires on your project, open an issue; fixed within a day. From 9088d1b1da109d6765322ce62536928f89ae4dcb Mon Sep 17 00:00:00 2001 From: Benjamin Hellmich <97835383+BellmeJoe@users.noreply.github.com> Date: Sun, 2 Aug 2026 17:11:21 +0300 Subject: [PATCH 02/11] =?UTF-8?q?docs(incidents):=20draft=20incident=20002?= =?UTF-8?q?=20(agent=20force-push,=20saved=20only=20by=20git's=20own=20dir?= =?UTF-8?q?ty-tree=20refusal)=20=E2=80=94=20staged,=20not=20published?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...rce-push-saved-only-by-gits-own-refusal.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 docs/incidents/002-agent-force-push-saved-only-by-gits-own-refusal.md diff --git a/docs/incidents/002-agent-force-push-saved-only-by-gits-own-refusal.md b/docs/incidents/002-agent-force-push-saved-only-by-gits-own-refusal.md new file mode 100644 index 0000000..a8e173c --- /dev/null +++ b/docs/incidents/002-agent-force-push-saved-only-by-gits-own-refusal.md @@ -0,0 +1,37 @@ +# Incident 002 — an agent's git integration force-pushed unconditionally; only git's own refusal on a dirty tree stopped data loss + +**Source:** a public GitHub issue filed by a maintainer against their own AI-agent framework's git integration, describing a real incident the maintainer traced by transcript. Anonymized here at the maintainer's-consent bar (no consent on file yet, no live touch conversation open on this thread) — no repo, org, or handle named. Reconstructed independently and run through getAdvantage's live gate; not the maintainer's own words beyond the class of failure. + +## What happened + +An agent framework's git integration resets and force-pushes a working tree as part of its normal session lifecycle, with no read-only or dry-run mode and no check for local uncommitted work before doing it. The maintainer's own transcript-verified account: this ran three times in one session. Each time, the only thing that stopped it from actually destroying data was git's own built-in refusal to `checkout` over a dirty working tree — a safety net that belongs to git, not to the agent tool. A directory that happened to have a clean tree, or committed state instead of loose local edits, would not have been protected at all. + +## Reconstruction: what the gate does with this + +Reproduced the shape of the failure locally: a clean initial commit, then a local edit made and never committed (standing in for the loose session work that was one force-push away from being gone), then ran `getadvantage check` cold — exactly as it would run before an agent's git step, if wired into a pre-flight hook. + +``` +✗ Dirty-tree guard — 1 tracked file modified/staged — a 'vercel --prod' would ship this unintended work. + M work.js + Commit, stash, or revert before shipping. Deploy from a clean detached worktree of the intended commit. + +Verdict + ✓ 2 ⚠ 1 ✗ 1 – 3 skipped + + NO-GO — 1 blocking issue. Do not ship until these are clear. +``` + +Run before the agent's git step instead of after, this turns "silently protected by luck, this one time" into a named, visible halt: NO-GO, the modified file listed by name, before any reset or force-push runs. + +## What it would NOT have caught + +Being direct about the edges, because a gate that only advertises catches isn't trustworthy: + +- **A tree that happens to be clean when the destructive command runs.** The maintainer's own account says the tree was dirty each of the three times this ran — that is what git's own refusal caught. If the same force-push had run against a directory with no loose edits (work already committed, or nothing pending), the dirty-tree guard has nothing to flag and would not stop it. The gate protects uncommitted local work specifically; it does not make a force-push or reset itself safe in general. +- **The destructive git command running before anyone calls the gate.** getAdvantage is a pre-flight check, not a runtime interceptor of git commands — it does not sit in the path of `git push --force` or `git reset --hard` and block them directly. It only helps if something calls `getadvantage check` before that step executes. An agent tool that force-pushes as its very first action, with no gate wired into that entry point, gets zero protection from installing getAdvantage elsewhere in the repo. +- **Anything outside where it's wired in.** Ad hoc, uninstalled-in-the-agent's-own-lifecycle runs give no protection at all. This is the exact gap [invisible-mode / automatic hook installs](../launch/SOCIAL-MEDIA-PACK.md) is built to close: the check running automatically on every agent session, including right before a git step, instead of depending on the tool's own authors remembering to call it. +- **Whether the force-push target was the right one.** The gate reads working-tree hygiene, not intent — it has no opinion on whether resetting or pushing to a given branch was the correct operation in the first place. A clean-tree force-push to the wrong branch is a different failure this class of check was never built to catch. + +## Try it + +`npx getadvantage check` on your own repo takes under a minute and reads nothing but your local tree — no network, no account. If it mis-fires on your project, open an issue; fixed within a day. From dddef51d608c5fc3352b400fa20e4f06011fa4f0 Mon Sep 17 00:00:00 2001 From: Benjamin Hellmich <97835383+BellmeJoe@users.noreply.github.com> Date: Mon, 3 Aug 2026 17:09:55 +0300 Subject: [PATCH 03/11] =?UTF-8?q?docs(incidents):=20draft=20incident=20003?= =?UTF-8?q?=20(committed=20.env=20secret=20leak,=20GitGuardian-sourced)=20?= =?UTF-8?q?=E2=80=94=20staged,=20not=20published?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../003-committed-env-secret-leak.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 docs/incidents/003-committed-env-secret-leak.md diff --git a/docs/incidents/003-committed-env-secret-leak.md b/docs/incidents/003-committed-env-secret-leak.md new file mode 100644 index 0000000..925465b --- /dev/null +++ b/docs/incidents/003-committed-env-secret-leak.md @@ -0,0 +1,37 @@ +# Incident 003 — a committed .env ships a live key, and the scanner only catches the one it recognizes + +**Source:** composite, built from a public industry statistic, not a single maintainer's incident. GitGuardian's *State of Secrets Sprawl 2026* report (cited via Docker's engineering blog, published 2026-07-28) found that code written with an AI coding agent leaks credentials at roughly double the rate of code written without one. No team or repo is named here; this is a reconstruction of the failure class the statistic describes, run through getAdvantage's own gate. + +## What happened + +The pattern behind that 2x number is simple and repeats across teams: an agent scaffolds a project, drops a working `.env` next to the code so the app runs locally, and either forgets the `.gitignore` line or adds it one commit too late. `git init && git add -A` on cycle one sweeps the file in before anyone thinks to check. The repo now ships a live database URL and one or more provider keys to every clone, and stays in git history even if someone deletes the file later — `.gitignore` only stops new untracked files, it has no effect on what git already tracked. + +## Reconstruction: what the gate does with this + +Built a minimal repo with a `.env` holding three realistic fixture values — a Postgres connection string, a Stripe live-shaped secret key, and an AWS-shaped access key — committed it on a clean tree, then ran `npx getadvantage check` cold. + +``` +✗ Secret scan — 1 possible secret in committed/staged files — remove + rotate before shipping. + .env → Stripe live secret key: sk_live_…0000 (32 chars) + Smallest safe next edit — Stripe live secret key at .env:2: remove, rotate, re-run check. + +✗ Tracked .env file — 1 .env file tracked by git — a committed .env is a leak by itself, whatever it contains. + .env + Remove it from git (git rm --cached ), add it to .gitignore, and ROTATE every key that file ever held. + +Verdict + ✓ 4 ⚠ 1 ✗ 2 – 3 skipped + NO-GO — 2 blocking issues. Do not ship until these are clear. +``` + +Two independent checks catch this, which matters because they fail differently: the secret scanner looks for known credential shapes, the tracked-.env check does not care what is inside the file at all — a committed `.env` is a leak by itself, full stop. Wired into a commit hook or CI, this is a hard NO-GO before the file reaches a second clone. + +## What it would NOT have caught + +- **The pattern-matched secret scanner missed two of the three fixture values.** Of the Postgres URL, Stripe key, and AWS-shaped key committed in this reconstruction, only the Stripe key was recognized and named by the secret scanner. The AWS-shaped key and the database URL were not flagged by that check on their own. The tracked-.env-file check still caught the whole file regardless of what patterns it contains — that check is what actually saves you here, not the pattern list. A secret shape the scanner doesn't recognize, committed somewhere other than a file named `.env` (a config.json, a shell script, a docker-compose override), would not be caught by either check in this run. +- **A key rotated after commit but left in history.** The gate reads the current tracked state, not git history. Removing the file and rotating the credential is the fix; the gate cannot detect an old, already-rotated key sitting in a prior commit and will not warn that history still holds it — that part is on the operator to know and do. +- **Anything outside where it's wired in.** Same limit as every check in this series: this only stops a ship if something actually calls it on the commit or in CI. That is the exact gap [invisible mode](../launch/SOCIAL-MEDIA-PACK.md) closes — the check runs automatically on every agent session instead of depending on someone remembering to call it. + +## Try it + +`npx getadvantage check` on your own repo takes under a minute and reads nothing but your local tree — no network, no account. If it mis-fires on your project, open an issue; fixed within a day. From e6996f3ae2fcf07b88eb6fc1885e0c1961e48930 Mon Sep 17 00:00:00 2001 From: Benjamin Hellmich <97835383+BellmeJoe@users.noreply.github.com> Date: Tue, 4 Aug 2026 11:09:15 +0300 Subject: [PATCH 04/11] =?UTF-8?q?docs(incidents):=20draft=20incident=20004?= =?UTF-8?q?=20(agent=20self-report=20vs=20actual=20diff,=20Intent=20Contra?= =?UTF-8?q?ct=20scope=20drift)=20=E2=80=94=20staged,=20not=20published?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../004-agent-self-report-scope-drift.md | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 docs/incidents/004-agent-self-report-scope-drift.md diff --git a/docs/incidents/004-agent-self-report-scope-drift.md b/docs/incidents/004-agent-self-report-scope-drift.md new file mode 100644 index 0000000..a8d871d --- /dev/null +++ b/docs/incidents/004-agent-self-report-scope-drift.md @@ -0,0 +1,47 @@ +# Incident 004 — an agent's own summary said "just the auth fix"; the diff said otherwise + +**Source:** a public GitHub issue filed by a maintainer against their own production repo, describing why a release agent's self-reported change list could not be trusted on its own. Anonymized here at the maintainer's-consent bar (no consent on file yet, no live touch conversation open on this thread) — no repo, org, or handle named. Reconstructed independently as a generic scope-drift scenario and run through getAdvantage's live gate; not the maintainer's own words beyond the class of failure. + +## What happened + +A release agent was coordinating a scoped change and reporting back what it had touched. The maintainer's own account is blunt about the limit of that self-report: the agent "does not have access to that project and must not fabricate the list" of what it actually changed. The team could not tell, from the agent's own summary, whether the real diff matched the declared task — they had to go verify by hand. + +This is the ordinary shape of a much broader problem: an agent says "I only touched the auth flow," and either it did, or it also picked up an unrelated file along the way — a leftover edit, a convenience refactor, a file it thought was related. Nothing about the agent's own narration tells you which one happened. The only ground truth is the diff. + +## Reconstruction: what the gate does with this + +Reproduced the shape of the failure locally with getAdvantage's Intent Contract: pinned a baseline commit, declared the real task in writing before any agent work started (`goal: "Add password reset flow"`, `allow: src/auth/**, tests/auth/**`), froze that contract as its own commit, then let the "agent" commit — genuinely scoped work in `src/auth/reset.js`, plus one extra line in an unrelated `src/billing/charge.js`, exactly the kind of drift a self-report narration would describe as "just the password reset." Ran `getadvantage intent check` cold against that history. + +``` +Intent Contract + ✗ Intent Contract — NO-GO — 1 scope violation against the Intent Contract. + goal: Add password reset flow + contract: sha256:5cb2bd6081852bcecb488e291f93f925d3a2259770a0afd735683a33ed88526a + receipt: sha256:62573d0408d2393991edd4483bf718ffc481e61dfa9a109443d0f3daf9a1aa8f + baseline: 484af8991baa + freeze: 35cb8ea594e0 + src/billing/charge.js — outside allowlist + Smallest safe next edit — path(s) outside Intent Contract allow list (src/billing/charge.js): + Preferred: unstage/remove the out-of-scope path(s) so the commit stays inside the frozen envelope at .getadvantage/intent.json. + Example: git restore --staged --worktree -- src/billing/charge.js + scope verified; semantic correctness not proven + +Verdict + NO-GO — changes left the authorized scope (or trust failed). + scope verified; semantic correctness not proven +``` + +The check doesn't read the agent's summary at all. It diffs everything that actually happened after the declared baseline — committed, staged, unstaged, even untracked — against the envelope a human wrote down before the work started, and names the exact file that fell outside it. No narration to trust or distrust; just the diff against the contract. + +## What it would NOT have caught + +Being direct about the edges, because a gate that only advertises catches isn't trustworthy: + +- **No contract, no verdict.** This only works if a human wrote the Intent Contract down before the agent started. A team that skips that step gets no scope check at all — `getadvantage check` never fakes a "verified" result for a project with no frozen contract. +- **A change that's technically in-scope but wrong.** The gate proves the diff stayed inside `src/auth/**`; it has no opinion on whether the password-reset logic itself is correct. Its own honesty line says it plainly: "scope verified; semantic correctness not proven." +- **A contract written too loosely.** If the human had allowed `src/**` instead of `src/auth/**`, the billing edit would pass cleanly — the gate enforces exactly the envelope it was given, not the envelope that should have been written. +- **Anything outside where it's wired in.** An ad hoc, uninstalled-in-CI run gives zero protection. This is the exact gap [invisible-mode / automatic hook installs](../launch/SOCIAL-MEDIA-PACK.md) is built to close: the check running automatically at commit time instead of depending on someone remembering to call it, or trusting the agent's own summary instead. + +## Try it + +`npx getadvantage check` on your own repo takes under a minute and reads nothing but your local tree — no network, no account. If it mis-fires on your project, open an issue; fixed within a day. From 78e4fe1dc7a0831dc2b41810ddd7eeda3b82bec2 Mon Sep 17 00:00:00 2001 From: Benjamin Hellmich <97835383+BellmeJoe@users.noreply.github.com> Date: Tue, 4 Aug 2026 17:11:03 +0300 Subject: [PATCH 05/11] =?UTF-8?q?docs(incidents):=20draft=20incident=20005?= =?UTF-8?q?=20(review-bandwidth=20collapse,=20agent=20architecture=20scan)?= =?UTF-8?q?=20=E2=80=94=20staged,=20not=20published?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...005-review-bandwidth-collapse-accretion.md | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 docs/incidents/005-review-bandwidth-collapse-accretion.md diff --git a/docs/incidents/005-review-bandwidth-collapse-accretion.md b/docs/incidents/005-review-bandwidth-collapse-accretion.md new file mode 100644 index 0000000..554da04 --- /dev/null +++ b/docs/incidents/005-review-bandwidth-collapse-accretion.md @@ -0,0 +1,52 @@ +# Incident 005 — three copies of the same validator, none of them caught in review + +**Source:** a widely-documented public pattern, not one repo's incident — engineering commentators independently describing the same shape of failure: PR volume from AI-assisted coding rising 3-5x, review queues backing up, and experienced engineers openly admitting they've stopped reviewing every agent-generated change because doing so by hand no longer scales. No individual or repo is named; this is a composite reconstruction of that documented pattern, not a specific maintainer's account. Reconstructed independently and run through getAdvantage's live gate. + +## What happened + +The pattern, stated plainly by the people living it: review load is exploding under AI-assisted output, the queue backs up, and at some point a reviewer decides it's "pointless" to keep reviewing everything an agent produces — so they stop, except for what they judge to be the important parts. That judgment call is where this incident lives. + +Reconstructed the shape: an agent adds order-validation logic on the checkout path. A future session, working on the admin panel, needs the same validation — instead of finding and reusing the existing function, it writes its own copy. A third session does the same for a different flow. None of the three sessions saw the other two; each one's diff looked small and reasonable in isolation, which is exactly why a tired reviewer waves each one through. Nobody was looking at the codebase as a whole. + +## Reconstruction: what the gate does with this + +Built the shape locally: one validation function, copied near-verbatim into three files over several commits, with one of the three copies re-touched repeatedly (the checkout path kept changing) while the other two sat untouched. Ran `getadvantage architecture` — the accretion scanner — cold against that history. + +``` +Architecture — accretion scan (advisory, read-only) + scanned: 3 source files · 54 lines + oversized: 0 files >600 lines (0 >1000 · 0 >1800) + duplication: 3 repeated blocks — >=15 similar lines appearing >=3x (exact/near-exact repetition, approximate) + churn window: last 16 commits + +Collapse candidates (top 3) + 1. src/checkoutValidate.js + 18 lines * changed in 14 of the last 16 commits * 1 duplicated block, e.g. lines 1-18 (shared with src/adminValidate.js, src/validate.js) * approx. complexity: 163 branches/100 lines + -> Repeated block: the same code lives in several places -- consider collapsing the copies into one shared helper before they drift apart. + 2. src/adminValidate.js + 18 lines * changed in 2 of the last 16 commits * 1 duplicated block, e.g. lines 1-18 (shared with src/checkoutValidate.js, src/validate.js) + 3. src/validate.js + 18 lines * changed in 2 of the last 16 commits * 1 duplicated block, e.g. lines 1-18 (shared with src/adminValidate.js, src/checkoutValidate.js) + +Verdict + Signal band: NOTABLE + Accretion surfaced in 3 files. This is a measurement, not a + judgment -- a human (or your agent, now informed) decides what to collapse. + Advisory only -- accretion never blocks a ship (exit 0). +``` + +This is the signal a per-PR reviewer structurally cannot see: no single diff looked wrong. What the scan surfaces is the *codebase*, not the changeset — three near-identical copies and which one is the hot spot actively drifting away from the other two, ranked by size, churn, and duplication together, not by which PR is on screen right now. + +## What it would NOT have caught + +Being direct about the edges, because a gate that only advertises catches isn't trustworthy: + +- **It never blocks anything.** Architecture is advisory only and always exits 0, on purpose — it is a measurement, not a gate. A team that wants this to stop a ship has to wire that decision themselves; getAdvantage will not make it for them. +- **Small-scale duplication under the window.** The duplication pass only counts a repeated block once it reaches 15 normalized lines appearing at least 3 times. Two short, nearly-identical 8-line helpers copied into two files would not register at all — this scanner is tuned for the accreted, drifted-apart case, not every instance of copy-paste. +- **Whether the duplicated logic is even correct.** The scan says three copies of the same shape exist and one is hot; it has no opinion on whether that validation logic is right, complete, or safe. It found the copies, not a bug in them. +- **A first scan on a fresh clone with no history.** Churn is read from git log; a shallow clone or a repo checked out without its commit history gives a churn signal of zero, and the "hot" ranking loses one of its three inputs. Size and duplication still work; churn does not. +- **Anything outside where it's wired in.** An ad hoc, uninstalled-in-CI run gives zero protection. This is the exact gap [invisible-mode / automatic hook installs](../launch/SOCIAL-MEDIA-PACK.md) is built to close: the check running automatically instead of depending on someone remembering to call it. + +## Try it + +`npx getadvantage architecture` on your own repo takes under a minute and reads nothing but your local tree and git history — no network, no account. If it mis-fires on your project, open an issue; fixed within a day. From a03a892391c31e6a8c566eee9d3cf4c0058f9d7b Mon Sep 17 00:00:00 2001 From: Benjamin Hellmich <97835383+BellmeJoe@users.noreply.github.com> Date: Wed, 5 Aug 2026 11:18:58 +0300 Subject: [PATCH 06/11] docs(incidents): draft incident 006 (open admin POST routes with no auth check, map advisory-only) - staged, not published --- .../006-open-admin-route-no-auth-check.md | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 docs/incidents/006-open-admin-route-no-auth-check.md diff --git a/docs/incidents/006-open-admin-route-no-auth-check.md b/docs/incidents/006-open-admin-route-no-auth-check.md new file mode 100644 index 0000000..4a22ef6 --- /dev/null +++ b/docs/incidents/006-open-admin-route-no-auth-check.md @@ -0,0 +1,40 @@ +# Incident 006 — two admin POST routes shipped with no auth check, and nothing failed the build + +**Source:** a widely-documented public pattern, not one repo's incident — the recurring "vibe-coded app shipped without an auth check on a mutating route" class that shows up repeatedly across AI-coding-tool postmortems and security write-ups (open admin endpoints, unauthenticated refund/role-change actions, exposed write routes reachable by anyone who finds the URL). No individual, repo, or company is named; this is a composite reconstruction of that documented pattern, not a specific maintainer's account. Reconstructed independently and run through getAdvantage's live gate. + +## What happened + +The shape, seen again and again: an agent scaffolds an API with one route properly gated (say, creating a product requires a logged-in session), then later sessions add more routes — an admin refund action, a role-change action — copying the surrounding style but not the auth middleware. Nothing about the new routes looks wrong in isolation: they parse JSON, they return a response, they follow the same file layout as the gated route next to them. Code review, if it happens at all, is scanning for logic bugs, not silently absent middleware. CI has no reason to fail — the routes work exactly as written, they just work for anyone. + +The gap survives because nothing in the normal ship path asks the question "does this route check who's calling it" as a distinct check from "does this route do what it says." A passing test suite and a green CI run both say nothing about that. + +## Reconstruction: what the gate does with this + +Built a throwaway four-route Express app: a read-only product listing, a properly auth-gated product-creation route, and two admin mutating routes (a refund action, a role-change action) added afterward with no auth middleware wired in — the exact "copied the neighbor's shape, not its guard" gap. Ran `getadvantage map` cold against it. + +``` +Map — what your app has (read-only) + Detected: Express project — the map reads your server's route definitions (best-effort). + ✓ Project estate — 1 top-level module · 3 files · languages: JavaScript (2) + ⚠ API surface map — 4 routes · 1 look gated (session or cron secret) · 3 mutate (write) · 2 mutate without any obvious gate. + ⚠ /api/admin/refund [POST] — mutates but no auth/session check found + ⚠ /api/admin/users/:id/role [POST] — mutates but no auth/session check found + — full map — + /api/admin/refund [POST] PUBLIC + mutates ⚠ + /api/admin/users/:id/role [POST] PUBLIC + mutates ⚠ + /api/products [GET] public (read-only) + /api/products [POST] auth-gated +``` + +Both unguarded mutating routes are named directly, with method and path, not buried in a summary count. Also ran `getadvantage check` (the full gate) on the same fixture: the API surface warning appears there too, but the overall verdict still came back **GO — with 2 warnings to eyeball first**, because right now the route-auth check is advisory, not a blocking gate on its own — see the honest limit below. + +## What it would NOT have caught + +- **It never blocks the ship on its own.** This is the important edge: two unauthenticated mutating admin routes produced warnings, not a NO-GO. `getadvantage check` returned a clean **GO** on this exact fixture. If a team is relying on the gate to hard-stop this class of bug the way it hard-stops a committed secret or a dirty tree, it currently does not — a human has to actually read the warning and decide, same as `map`'s own accretion scan in incident 005. +- **A route framework it doesn't parse.** Detection is best-effort regex over Next.js App Router, Express/Fastify, and Flask/FastAPI. A route defined through a less common router, a dynamically generated route table, or a non-standard middleware pattern can sit outside what the parser recognizes — and the tool says so plainly (client SPAs get an honest "route mapping does not apply" line instead of a guess). +- **Auth middleware it can't recognize as auth.** The check looks for an "obvious gate" — a session check, a known auth middleware name, a cron secret. A custom, oddly-named, or indirectly-wired auth check can read as ungated even when the route is actually protected — a false positive in the safe direction, but still a human has to confirm it, not the tool. +- **Whether the route *should* be public.** Some mutating routes legitimately have no auth — a rate-limited public contact-form submission, for instance. The map flags every unguarded mutating route the same way; it does not know your product's intent, which is exactly why the output says "confirm each warning route is meant to be public" instead of asserting a verdict. + +## Try it + +`npx getadvantage map` on your own repo takes under a minute, reads nothing but your local route definitions, and never blocks or writes anything. If it mis-fires on your project, open an issue; fixed within a day. From 322c325d15c5786fe31cea7ea1bda8e54d6dddbb Mon Sep 17 00:00:00 2001 From: Benjamin Hellmich <97835383+BellmeJoe@users.noreply.github.com> Date: Wed, 12 Aug 2026 17:19:20 +0300 Subject: [PATCH 07/11] docs(incidents): draft incident 007 (agent cleanup deleted out-of-scope files; misses section covers everything outside the repo) - staged, not published --- ...-deleted-what-it-was-not-asked-to-touch.md | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 docs/incidents/007-agent-cleanup-deleted-what-it-was-not-asked-to-touch.md diff --git a/docs/incidents/007-agent-cleanup-deleted-what-it-was-not-asked-to-touch.md b/docs/incidents/007-agent-cleanup-deleted-what-it-was-not-asked-to-touch.md new file mode 100644 index 0000000..dfc1e32 --- /dev/null +++ b/docs/incidents/007-agent-cleanup-deleted-what-it-was-not-asked-to-touch.md @@ -0,0 +1,61 @@ +# Incident 007 — a cleanup command that deleted far more than the cleanup + +**Source:** a public post containing an AI coding agent's own transcript, in which the agent reports that a command it ran destroyed files it was never asked to touch, including credentials on the developer's machine. Anonymized here at the default bar — no handle, no repo, no employer named, and no consent on file. Reconstructed independently as a generic over-broad-deletion scenario and run through getAdvantage's live gate. The transcript is the evidence for the class of failure; nothing below is presented as that person's own project. + +## What happened + +The task was small and boring: clean up some stale output. The agent wrote a command with a wrong path in it, and then, trying to tidy up after its own error, ran a recursive delete against a directory far above the one it meant. The agent's own words afterwards are the interesting part. It stopped, said plainly that it had caused damage, and listed what was gone. Among the casualties were private SSH keys and a known_hosts file, which is to say the developer's credentials and the machine's trust store. + +Two things make this worth writing down rather than filing under bad luck. + +First, the agent disclosed the damage only after it was done. Nothing about the earlier turns signalled that the blast radius had moved from a build directory to a home directory. Second, and more usefully for anyone deciding what to actually install: a large part of that damage is outside what any repository-level gate can see. That is the honest half of this page, and it is below. + +## Reconstruction: what the gate does with the part it can see + +Reproduced the in-repository half of the failure locally. Pinned a baseline commit on a small service repo, wrote the Intent Contract before any agent work — the real task, in writing, with a deliberately narrow envelope — and froze it as its own commit: + +``` +getadvantage intent init --goal "Clean up stale build output under dist/" \ + --allow "scripts/**" --allow "dist/**" +git add .getadvantage/intent.json && git commit -m "chore: intent contract" +``` + +Then let the "agent" run its over-broad delete, which took out tracked source and tests instead of build output, and ran `getadvantage intent check` cold against the result: + +``` +Intent Contract + ✗ Intent Contract — NO-GO — 3 scope violations against the Intent Contract. + goal: Clean up stale build output under dist/ + contract: sha256:2d34e2c00b0e2b86d379758a5bf5f05482463c636e34e78e69d5163310f8d5d6 + receipt: sha256:91206e98b49dfeedb86becc46bfe21227e2f84b53e9d1c76dcc840f3e31d6a70 + baseline: afd30e23e5d5 + freeze: 957b6d3b5cdc + src/api/auth.js — outside allowlist + src/api/handler.js — outside allowlist + tests/api/handler.test.js — outside allowlist + Smallest safe next edit — path(s) outside Intent Contract allow list (src/api/auth.js, src/api/handler.js, tests/api/handler.test.js): + Preferred: unstage/remove the out-of-scope path(s) so the commit stays inside the frozen envelope at .getadvantage/intent.json. + Example: git restore --staged --worktree -- src/api/auth.js + scope verified; semantic correctness not proven + +Verdict + NO-GO — changes left the authorized scope (or trust failed). + scope verified; semantic correctness not proven +``` + +The deletions were never staged and never committed. They were loose changes sitting in a working tree, which is exactly the state a tired developer commits by reflex with `git add -A`. The check counts deletions as changes, names all three by path, and refuses the commit before it becomes history. It does not read the agent's summary of what it did; it diffs what is actually there against what a human authorized in advance. + +## What it would NOT have caught + +This is the section that matters on this particular incident, because the worst of the real damage falls squarely inside it. + +- **Everything outside the repository. This is the big one.** The private keys in the developer's home directory are not in the repo, produce no diff, and are invisible to this check. If the whole incident had happened one directory above the project, getAdvantage would have printed a clean GO on a repo that was fine while the machine around it was not. It is a repository gate, and a repository gate cannot see your home directory. +- **It does not stop the command from running.** It is not a sandbox, a permissions layer, or a shell interceptor. By the time the check runs, the files are already deleted. What it stops is the deletion becoming a commit and a push — the difference between a local mess one `git restore` undoes and a destroyed main branch other people pull. +- **It restores nothing.** There is no undo here. The recovery in the reconstruction is `git restore`, and that only works because the files were still in git. Anything never committed is gone whatever tooling you had installed. +- **No contract, no scope verdict.** The Intent Contract only exists if a human wrote it down before the agent started. Skip that and this check has nothing to compare against, and it will not fake a verdict — a project without a contract keeps its other checks and gets no false "intent verified". +- **An envelope written too wide passes.** Had the contract allowed `src/**`, the deleted source would have been inside the authorized scope and the run would have gone GO. The gate enforces the envelope it was given, not the one that should have been written. +- **A run nobody remembers to make protects nobody.** An occasional manual invocation is not a gate. This is the gap automatic hook installation is built to close: the check firing on every commit, without a human choosing to run it in the moment they are least likely to think of it. + +## Try it + +`npx getadvantage check` on your own repo takes under a minute and reads nothing but your local tree — no network, no account. If it mis-fires on your project, open an issue; fixed within a day. From e49c0ad8232e4db75b00611f92dd91e44cbce71f Mon Sep 17 00:00:00 2001 From: Benjamin Hellmich <97835383+BellmeJoe@users.noreply.github.com> Date: Tue, 18 Aug 2026 17:28:24 +0300 Subject: [PATCH 08/11] docs(incidents): regenerate incident 003 transcript from live 0.13.1 Grok's 2026-08-17 technical verification (agent-ops/runs/2026-08-17-grok-build-CYCLE-2045.md section 6) found the published transcript was captured on a pre-0.13.1 build: missing file:line, missing per-finding auth hash, and stale warn/skip counts. Reran the reconstruction against live npx getadvantage@latest (confirmed 0.13.1) with the password-free Postgres URL variant the page already implies. The AWS-shaped key is now recognized by the live scanner (it was not at the time the page was first drafted), which also corrects the misses section: the scanner now misses only the unadorned database URL, not the AWS key. Not yet re-verified by Grok; PR #2 stays unmerged pending that pass. Incidents 001, 002, 004-007 were spot-checked for the same defect class (only secret-scan pages carry an auth hash) but not independently re-run against live output this cycle. --- docs/incidents/003-committed-env-secret-leak.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/incidents/003-committed-env-secret-leak.md b/docs/incidents/003-committed-env-secret-leak.md index 925465b..c7dcb3a 100644 --- a/docs/incidents/003-committed-env-secret-leak.md +++ b/docs/incidents/003-committed-env-secret-leak.md @@ -11,13 +11,15 @@ The pattern behind that 2x number is simple and repeats across teams: an agent s Built a minimal repo with a `.env` holding three realistic fixture values — a Postgres connection string, a Stripe live-shaped secret key, and an AWS-shaped access key — committed it on a clean tree, then ran `npx getadvantage check` cold. ``` -✗ Secret scan — 1 possible secret in committed/staged files — remove + rotate before shipping. - .env → Stripe live secret key: sk_live_…0000 (32 chars) - Smallest safe next edit — Stripe live secret key at .env:2: remove, rotate, re-run check. +✗ Secret scan — 2 possible secrets in committed/staged files — remove + rotate before shipping. + .env:2 → Stripe live secret key: sk_live_…0000 (35 chars) · auth 7d3e4150553791318e1e3e917d32ad59a594f02bd50cb162fe0379133766baa4 + .env:3 → AWS access key id: AKIAIO…0000 (20 chars) · auth 7b6e44c8d529c15d5de3f17dc7d7d6933d35fb2a1b68e91b1a25060f073ffd11 + Smallest safe next edit — 2 blocking secret findings (same remedy for each; every finding named above with its own auth id): + remove the value from the tree, rotate the credential at the provider, commit the removal, then re-run check. ✗ Tracked .env file — 1 .env file tracked by git — a committed .env is a leak by itself, whatever it contains. .env - Remove it from git (git rm --cached ), add it to .gitignore, and ROTATE every key that file ever held. + Remove it from git (git rm --cached ), add it to .gitignore, and ROTATE every key that file ever held — git history keeps old values. Verdict ✓ 4 ⚠ 1 ✗ 2 – 3 skipped @@ -28,7 +30,7 @@ Two independent checks catch this, which matters because they fail differently: ## What it would NOT have caught -- **The pattern-matched secret scanner missed two of the three fixture values.** Of the Postgres URL, Stripe key, and AWS-shaped key committed in this reconstruction, only the Stripe key was recognized and named by the secret scanner. The AWS-shaped key and the database URL were not flagged by that check on their own. The tracked-.env-file check still caught the whole file regardless of what patterns it contains — that check is what actually saves you here, not the pattern list. A secret shape the scanner doesn't recognize, committed somewhere other than a file named `.env` (a config.json, a shell script, a docker-compose override), would not be caught by either check in this run. +- **The pattern-matched secret scanner missed one of the three fixture values.** Of the Postgres URL, Stripe key, and AWS-shaped key committed in this reconstruction, the Stripe key and the AWS-shaped access key were both recognized and named by the secret scanner. The plain database URL — no password embedded in the connection string — was not flagged by that check on its own; the same URL **with** a password embedded in it is recognized as a separate finding ("Database URL with embedded password"). The tracked-.env-file check still caught the whole file regardless of what patterns it contains — that check is what actually saves you here for anything the pattern list doesn't recognize. A secret shape the scanner doesn't recognize, committed somewhere other than a file named `.env` (a config.json, a shell script, a docker-compose override), would not be caught by either check in this run. - **A key rotated after commit but left in history.** The gate reads the current tracked state, not git history. Removing the file and rotating the credential is the fix; the gate cannot detect an old, already-rotated key sitting in a prior commit and will not warn that history still holds it — that part is on the operator to know and do. - **Anything outside where it's wired in.** Same limit as every check in this series: this only stops a ship if something actually calls it on the commit or in CI. That is the exact gap [invisible mode](../launch/SOCIAL-MEDIA-PACK.md) closes — the check runs automatically on every agent session instead of depending on someone remembering to call it. From e734f388f92b6656e6c5915b7f20eb95b6da4f2d Mon Sep 17 00:00:00 2001 From: pin Date: Wed, 19 Aug 2026 11:18:50 +0300 Subject: [PATCH 09/11] docs(incidents): fix stale verdict count in incident 002 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live-reran the reconstruction against npx getadvantage@latest (confirmed 0.13.1, generic repo, no package.json): full output totals ✓5 ⚠1 ✗1 –3 skipped (Checks section ✓2 + Overview section ✓3), not the ✓2 the page previously claimed. The page only summed the Checks-section checks and left out the three passing Overview-section lines (API surface map, Agents & integrations map, Schedules & jobs map). Checks-section transcript block, finding text, and misses section were already byte-for-byte accurate against live output; only the verdict total was wrong. Not yet re-verified by Grok; PR #2 stays unmerged pending that pass. Spot-checked incident 001 the same way this cycle: its verdict (✓6 ⚠1 ✗1 –2 skipped, reproduced with a package.json present) matches live output exactly, no fix needed. Incidents 004-007 not yet re-run this cycle. --- .../002-agent-force-push-saved-only-by-gits-own-refusal.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/incidents/002-agent-force-push-saved-only-by-gits-own-refusal.md b/docs/incidents/002-agent-force-push-saved-only-by-gits-own-refusal.md index a8e173c..9a21292 100644 --- a/docs/incidents/002-agent-force-push-saved-only-by-gits-own-refusal.md +++ b/docs/incidents/002-agent-force-push-saved-only-by-gits-own-refusal.md @@ -16,7 +16,7 @@ Reproduced the shape of the failure locally: a clean initial commit, then a loca Commit, stash, or revert before shipping. Deploy from a clean detached worktree of the intended commit. Verdict - ✓ 2 ⚠ 1 ✗ 1 – 3 skipped + ✓ 5 ⚠ 1 ✗ 1 – 3 skipped NO-GO — 1 blocking issue. Do not ship until these are clear. ``` From 6914239d4d2a8868892022039d220e0762b929b9 Mon Sep 17 00:00:00 2001 From: pin Date: Wed, 19 Aug 2026 11:20:15 +0300 Subject: [PATCH 10/11] docs(incidents): restore truncated Smallest-safe-next-edit lines in incident 004 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live-reran the Intent Contract reconstruction against npx getadvantage@latest (confirmed 0.13.1): getadvantage intent init + intent check on a fresh repo reproducing the same scope-drift shape (allowlist src/auth/**, tests/auth/**; out-of-scope edit to src/billing/charge.js). The Checks-section structure, finding text, and "scope verified; semantic correctness not proven" line were already accurate. Missing: two real output lines under "Smallest safe next edit" — the wider-envelope reauthorization workflow and the cannot-self-authorize note on a frozen contract. Added both, verbatim from live output. Illustrative hashes/commit IDs in the page are unchanged (they were already a different reconstructed run, not meant to be literal). Not yet re-verified by Grok; PR #2 stays unmerged pending that pass. Incidents 005-007 not yet re-run against live output this cycle — flagged open, not silently skipped. --- docs/incidents/004-agent-self-report-scope-drift.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/incidents/004-agent-self-report-scope-drift.md b/docs/incidents/004-agent-self-report-scope-drift.md index a8d871d..61bfde5 100644 --- a/docs/incidents/004-agent-self-report-scope-drift.md +++ b/docs/incidents/004-agent-self-report-scope-drift.md @@ -24,6 +24,10 @@ Intent Contract Smallest safe next edit — path(s) outside Intent Contract allow list (src/billing/charge.js): Preferred: unstage/remove the out-of-scope path(s) so the commit stays inside the frozen envelope at .getadvantage/intent.json. Example: git restore --staged --worktree -- src/billing/charge.js + To authorize a wider envelope: start a branch from a trusted base with NO intent history, then: + getadvantage intent init --goal "…" --allow "relevant/**" --allow "…" + git add .getadvantage/intent.json && git commit -m "chore: intent contract" + Note: editing a frozen .getadvantage/intent.json cannot self-authorize (unsigned local mode: one freeze per clean lineage). scope verified; semantic correctness not proven Verdict From 666e7159d318e3ff455c047fcb06173bd774d0f3 Mon Sep 17 00:00:00 2001 From: pin Date: Wed, 19 Aug 2026 17:18:14 +0300 Subject: [PATCH 11/11] docs(incidents): restore truncated live-tool lines in incidents 005, 006, 007 005: architecture scan was missing the explanatory subhead under the section header and the signal-band suffix on the Verdict line. 006: map output was missing the confirm-each-warning-route line between the flagged routes and the full route table. 007: intent check was missing the wider-envelope reauthorization workflow and the cannot-self-authorize note, same truncation class already fixed in incident 004 this morning. All three verified against live getadvantage@0.13.1 output from fresh local reconstructions, not asserted from memory. --- docs/incidents/005-review-bandwidth-collapse-accretion.md | 5 ++++- docs/incidents/006-open-admin-route-no-auth-check.md | 2 ++ ...7-agent-cleanup-deleted-what-it-was-not-asked-to-touch.md | 4 ++++ 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/incidents/005-review-bandwidth-collapse-accretion.md b/docs/incidents/005-review-bandwidth-collapse-accretion.md index 554da04..d40b2bb 100644 --- a/docs/incidents/005-review-bandwidth-collapse-accretion.md +++ b/docs/incidents/005-review-bandwidth-collapse-accretion.md @@ -14,6 +14,9 @@ Built the shape locally: one validation function, copied near-verbatim into thre ``` Architecture — accretion scan (advisory, read-only) + Where the codebase is being built OVER instead of collapsed — the signal + an AI coding agent doesn't have. Measurement + heuristics; judgment stays human. + scanned: 3 source files · 54 lines oversized: 0 files >600 lines (0 >1000 · 0 >1800) duplication: 3 repeated blocks — >=15 similar lines appearing >=3x (exact/near-exact repetition, approximate) @@ -29,7 +32,7 @@ Collapse candidates (top 3) 18 lines * changed in 2 of the last 16 commits * 1 duplicated block, e.g. lines 1-18 (shared with src/adminValidate.js, src/checkoutValidate.js) Verdict - Signal band: NOTABLE + Signal band: NOTABLE — how loud the accretion signals are, not a grade of your code. Accretion surfaced in 3 files. This is a measurement, not a judgment -- a human (or your agent, now informed) decides what to collapse. Advisory only -- accretion never blocks a ship (exit 0). diff --git a/docs/incidents/006-open-admin-route-no-auth-check.md b/docs/incidents/006-open-admin-route-no-auth-check.md index 4a22ef6..36d1538 100644 --- a/docs/incidents/006-open-admin-route-no-auth-check.md +++ b/docs/incidents/006-open-admin-route-no-auth-check.md @@ -19,6 +19,8 @@ Map — what your app has (read-only) ⚠ API surface map — 4 routes · 1 look gated (session or cron secret) · 3 mutate (write) · 2 mutate without any obvious gate. ⚠ /api/admin/refund [POST] — mutates but no auth/session check found ⚠ /api/admin/users/:id/role [POST] — mutates but no auth/session check found + Confirm each ⚠ route is meant to be public (some, like a rate-limited lead/contact + endpoint, legitimately are) — and that the others authenticate the caller before writing. — full map — /api/admin/refund [POST] PUBLIC + mutates ⚠ /api/admin/users/:id/role [POST] PUBLIC + mutates ⚠ diff --git a/docs/incidents/007-agent-cleanup-deleted-what-it-was-not-asked-to-touch.md b/docs/incidents/007-agent-cleanup-deleted-what-it-was-not-asked-to-touch.md index dfc1e32..48e4ccb 100644 --- a/docs/incidents/007-agent-cleanup-deleted-what-it-was-not-asked-to-touch.md +++ b/docs/incidents/007-agent-cleanup-deleted-what-it-was-not-asked-to-touch.md @@ -36,6 +36,10 @@ Intent Contract Smallest safe next edit — path(s) outside Intent Contract allow list (src/api/auth.js, src/api/handler.js, tests/api/handler.test.js): Preferred: unstage/remove the out-of-scope path(s) so the commit stays inside the frozen envelope at .getadvantage/intent.json. Example: git restore --staged --worktree -- src/api/auth.js + To authorize a wider envelope: start a branch from a trusted base with NO intent history, then: + getadvantage intent init --goal "…" --allow "relevant/**" --allow "…" + git add .getadvantage/intent.json && git commit -m "chore: intent contract" + Note: editing a frozen .getadvantage/intent.json cannot self-authorize (unsigned local mode: one freeze per clean lineage). scope verified; semantic correctness not proven Verdict