Skip to content
connabyssPublic

About

Checks whether the "fixed" in your AI coding agent's commits is true. It takes each fix back out and runs your tests. Free, with no key.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

3 stars

Watchers

1 watching

Forks

Use this GitHub action with your project
Add this Action to an existing workflow or create a new one
View on Marketplace

Latest commit

 

History

479 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Bite

Would your tests catch it if the bug came back?

Bite checks whether an AI coding agent's "fixed" is actually true.
It takes each fix back out of a copy of your project and runs your tests. Free, no key.

PyPI Python Tests License: MIT

Open in GitHub Codespaces
Try it in your browser, no install

Try it · Install guide · GitHub Action · Changelog · Discussions · Contribute

Read this in: Tiếng Việt · Español · 简体中文


bite --demo: one fix is noticed by a test, the other has no test at all

Try it now

Free, no key, and it never changes your files. You need Python 3.10 or newer and git. Paste these two lines into any terminal (PowerShell, cmd, Terminal):

python -m pip install bite-check
python -m bite_check --free

Outside a project, Bite first shows a demo on a tiny made-up one. To check your own project, type the second line and a space, drag your project's folder into the window (that types its path), and press Enter.

No install at all: open Bite in GitHub Codespaces (the button at the top) opens a terminal in your browser with Bite already installed. Type bite --demo, or check a famous project straight from its link: bite --free https://github.com/pallets/click. (Codespaces runs on your GitHub account's free monthly hours.)

Any project on GitHub, GitLab, Codeberg or Bitbucket works from its link, on your own computer too: python -m bite_check --free https://github.com/OWNER/NAME. Bite clones it into a folder of its own and runs its tests there, which runs that project's code, so check only projects you'd run yourself.

Other ways: pipx run bite-check --free or uvx bite-check --free (one line, no install; on Linux or a Mac, use one of these if pip says "externally managed"). Commits never say "fix"? Add --any. To keep a bite command installed: install guide.

Why you need this

When an AI fixes a bug for you, it often says "Fixed, and I added a test." If you don't read code, you can't tell if that's true.

A test is a tiny robot that checks one thing and shouts when it's wrong. Bite puts the old bug back on purpose and watches whether any test shouts. It's like testing a smoke alarm with real smoke instead of trusting the little green light.

If a test shouts, that fix is guarded. If they all stay quiet, the bug could come back and nobody would notice. Bite names that fix, so you can tell your AI: "add a test that fails without this fix."

bite --demo shows both cases on a tiny made-up shop:

  9661edcf ✅ noticed by test_discount_never_below_zero Fix discounts over 100% making the price negative
  e5c329ec ⚠️ NO TEST NOTICED           Fix totals showing long decimals like 0.30000000000000004

Bite does this: takes each recent fix back out of a copy of your project, runs your tests, and tells you whether any test noticed. It never changes your files or pushes anything.

You decide that: whether to add the missing test, or leave it because the change didn't need one (a rename, a tidy-up). A ✅ means a test would catch that fix breaking. It doesn't prove the code has no other bugs.

Works with pytest, npm test, cargo test, go test, dotnet test, Maven and Gradle (Java and Kotlin). Full list: SUPPORTS.md. How Bite differs from coverage and mutation testing: Bite, mutmut and coverage.

Everything else Bite can do

What How More
Check every pull request Add the GitHub Action (below) or the bite-check app Pull requests
Check Claude's "fixed" the moment it says it /plugin marketplace add connabyss/bite, then /plugin install bite-check@bite (and fixed-or-not@bite for the skill and /bite) Live checks
Same, in Cursor, Copilot, Codex, Windsurf, Gemini CLI bite mcp Live checks
Teach any coding AI to prove its fixes (a free skill) npx skills add connabyss/bite The skill · also on Agensi
Ask in plain words bite ask "is last week's login fix guarded?" Ask
Catch a fix a later commit quietly undid Built into every free check Comeback alarm
Write the missing test (your own key) bite --write-tests Write tests
Score, badge, JSON for scripts bite --json Score and results
Speed up a slow check bite --free --jobs 4 (not with pytest -n auto: it's slower) When a check is slow

The pull request check. Add this as .github/workflows/bite.yml. It runs the free check on each pull request's commits and fails when a change can be taken back out and your tests still pass. No key, no cost.

name: bite
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  bite:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0
      # install what your tests need here, e.g. for Python:
      - uses: actions/setup-python@v7
      - run: python -m pip install pytest -r requirements.txt
      - uses: connabyss/bite@v1.0.8

The paid step (optional). With your own Anthropic API key, Bite writes one small test of a commit's exact claim and runs it; --budget caps what a run spends ($1 by default). It runs on Anthropic's Agent SDK, so it needs an API key and never uses a Claude login or plan (Anthropic's Agent SDK rule: https://code.claude.com/docs/en/agent-sdk/overview). Setup: install guide.

Updating Bite

bite --version says when a newer one is out. This line always gets the newest version:

pipx install --force bite-check --pip-args="--no-cache-dir"

Installed with pip? python -m pip install --upgrade --no-cache-dir bite-check. Why not pipx upgrade: install guide. What changed: CHANGELOG.md.

Does it work?

In prosemirror-py, a Claude commit said it fixed a memory leak. Bite wrote a test of that claim: it fails on that commit and on today's code. After 20,000 documents the cache still held all 20,000, using 35 MB. The tested fix was merged upstream. (The full story.)

"Reverting the claimed fix and making the change prove itself under a real test is exactly the kind of check that should exist for every "fixed" commit."

— Ivan Zdravkov, on DEV

Famous repos Bite has checked with bite --free (Oct 30-31, 2026):

Project Language Changes checked Noticed by a test Real gaps
Rich Python 37 15 1 small: <kbd> keys in Markdown
Click Python 37 30 1 small: help on custom setting types
qs JavaScript 40 40 none ✅
chi Go 27 22 1 small: XML compression
serde_json Rust 16 12 none ✅
FluentValidation C# 9 5 none ✅

More: the full scoreboard (20 projects, 5 languages), and bugs fixed or merged upstream in pyclibrary, drf-pydantic, django-tailwind-cli, west, mizani, pamqp and more.

Bite is listed on awesome-python-testing and awesome-testing.

Limits, plainly

  • Most claims hold. Across all rounds so far, Bite tested 137 "fixed" claims that it could run tests for (all from commits made with Claude Code, the AI those rounds looked at). 8 were false (about 1 in 17), and 4 of those were still broken on today's code. On well-reviewed projects it usually finds nothing, and that's the right answer.
  • Quiet is not a pass. Bite only shows a failing test that the second check calls fair and real. That second check has thrown out more than 10 failing tests that weren't. A claim that holds proves that one claim, not that the commit has no other bugs.
  • It needs the project's tests to run. If they don't run on a clean copy, Bite can't check that commit.
  • Bug hunting is weaker than claim checking. In a 10-project overnight hunt, two blind AI judges rated 21 of 22 proven fixes as real bugs and none as wrong, but Bite found only 1 of the 10 bugs the maintainers fixed soon after.
  • Small samples, AI judges. The judges read code without running it. The human checks are the maintainers' fixes and merges listed above.

Join in

Looking for your first open-source PR? These good first issues are small, and each one says where to look and when it's done. Comment on one to claim it. Or try to fool Bite: sneak a "fix" past it.

🏆 Fooled Bite: nobody yet. Be the first: your name, your trick, and the test that now catches it go right here. (How to enter. Bite's own 2 known blind spots are in Still fools Bite, so they don't count.)

Bug reports, fixes and AI-assisted PRs are all welcome: CONTRIBUTING.md has the one-minute setup. If Bite flagged something wrongly, or missed a claim you know was false, please open an issue. Questions and ideas go in Discussions.

Made by these people

Thank you to everyone who has helped make Bite. Want to be here? Pick a good first issue.

snehaapratap
snehaapratap

💻
gauravjangra3165
gauravjangra3165

🚇
bfs2021
bfs2021

💻
HarshRajSinghania
HarshRajSinghania

💻
hrithiksaini99
hrithiksaini99

💻 📖
The-Aizen-404
The-Aizen-404

💻
SatriaBagusNurhadianto
SatriaBagusNurhadianto

💻
hamid-digitalmarketer
hamid-digitalmarketer

💻
zheng-guannan577
zheng-guannan577

💻
AneesFathima-A
AneesFathima-A

💻

Also by me: Cold Case Unit, which opens a murder case on every GitHub project its owner walked away from.

MIT licensed.

About

Checks whether the "fixed" in your AI coding agent's commits is true. It takes each fix back out and runs your tests. Free, with no key.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

3 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages