Repository navigation
docs: propose a stepwise plan to make FTW safe for agents to change - #1679
Conversation
Agents write most of FTW, and review bots raised 1,150 findings on 573 of their pull requests. This plan groups those findings into recurring mistake classes, proposes a structural fix for each at the highest level that works, backtests each fix against the history, and orders the fixes into independently mergeable steps. Steps 1, 2 and 4 are draft PRs #1676, #1677 and #1678. tools/agent-pitfalls holds the script and the labels so a reviewer can rerun or spot-check every number.
Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a73a0129b9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Status: proposed. Every step below is a separate pull request. Steps 1, 2 | ||
| and 4 exist as drafts. The rest are designs. Nothing has merged. |
There was a problem hiding this comment.
The commit is based directly on the merged #1676, #1677, and #1678 commits (07ad1cb, 69613e1, and 592b8ee), so describing steps 1, 2, and 4 as drafts and saying nothing has merged makes the plan stale on arrival. Update the status and subsequent draft references so readers can distinguish completed checks from remaining proposals.
AGENTS.md reference: AGENTS.md:L146-L152
Useful? React with 👍 / 👎.
| for c in json.load(open(src)): | ||
| if c["user"]["login"] not in BOTS: continue |
There was a problem hiding this comment.
When the documented repository-wide comments endpoint includes a bot-reviewed human or external PR, this filter still adds it because it checks only the reviewer's login; no PR author, branch, or commit metadata is fetched. The classifier and backtest are then explicitly told every row came from an AI-agent-written PR, so those rows silently contaminate the agent-specific counts and priorities. Fetch and apply the stated PR-origin criteria, or describe the population as all bot-reviewed PRs.
AGENTS.md reference: AGENTS.md:L9-L13
Useful? React with 👍 / 👎.
| The two "no check found" rules that pass today (planner and database) | ||
| cost one import check each, written the same way as the cloudlink check. |
There was a problem hiding this comment.
The current tree already executes db.QueryRowContext in go/internal/backup/archive.go:1258, outside go/internal/state, so the database rule does not pass today and a straightforward import/query check would fail immediately. Present this as an existing violation requiring migration or a deliberate exception rather than as a passing, low-cost follow-up.
AGENTS.md reference: AGENTS.md:L69-L70
Useful? React with 👍 / 👎.
| for res in ex.map(lambda ch: ask(make_prompt(ch), model), chunks): | ||
| for k, j in res.items(): done[k] = parse(j) |
There was a problem hiding this comment.
If an LLM returns an incorrect but syntactically valid ID, this loop records it without checking that the ID belongs to the current chunk. A hallucinated future ID is then treated as completed and skipped on a later run, silently attaching another finding's label or backtest result to that row even when the final labeled count appears complete; restrict accepted keys to the IDs submitted in ch.
Useful? React with 👍 / 👎.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Propose focused changes to make FTW easier for agents to change safely. The plan now distinguishes model estimates, executable checks and simulator evidence.
The CSV contains 1,150 bot comments on 573 PRs. These are not 1,150 confirmed, distinct defects. Codex labelled every comment; Grok labelled 920 and agreed on 726 (78.9%). That agreement measures label consistency, not defect validity or prevention.
The model assigned 257 comments to at least one of six hypothetical structures. After dropping S6, the union is 249. Neither total measures protection delivered by these draft PRs. In particular, S2 includes a lifecycle harness that #1676 does not add, and S3 assumes a typed freshness guarantee that #1677's count check does not provide.
Executable evidence:
Validation: recomputed CSV row/PR counts, rater agreement and both structure unions. The draft changes docs only; no changeset. Nothing is merged, and the owner retains the merge and scope decisions.