Documentation: basantbhattarai.com.np/evalproof
Treat LLM and agent evaluation as measurement. Calibrate the judge, test agents through their tool calls, and gate CI on regressions that are real rather than noise.
Try it in the browser: evalproof.streamlit.app. Check whether a drop between two runs is real, or replay the measured run and read the twelve answers that changed. It runs this repository's code; after a quiet spell it takes about half a minute to wake.
An LLM judge is a classifier with error rates. If it passes bad answers at rate FPR and
good answers at rate TPR, its pass rate converges to FPR + (TPR − FPR) · p, not to the
true rate p. That is classify-and-count, and it is biased the way any classifier's count
is — the effect is worked through with a seeded simulation in
The classifier was 95% accurate. The trend it counted was a third too flat
(lab).
evalproof:
- measures the judge against a small gold-labelled sample — confusion matrix, TPR, FPR, Cohen's κ;
- corrects its pass rate with adjusted classify-and-count and prediction-powered inference, each with a confidence interval;
- compares runs on paired cases — exact McNemar for pass/fail, paired bootstrap for scores, and a paired permutation test when each case runs several times;
- gates CI only when a drop is both larger than your tolerance and statistically significant, plus latency and cost budgets, with distinct exit codes;
- checks the run files it gates —
verifyproves, with no model call, that a committed run was made with this suite and covers the dataset as it is now; - tests agents — asserts on the tools an agent called and what it said, across several turns, several builds of the same system and repeated trials, with adapters for ADK, LangChain/LangGraph, the Claude Agent SDK, OpenAI Agents, LlamaIndex, CrewAI, Pydantic AI, DSPy, Haystack, AutoGen and Semantic Kernel (Testing agents);
- finds what is not tested —
scanlists the model calls, agents and tools in a Python or TypeScript repository that no suite reaches, anddraftwrites inputs for them, never expected answers (Find what is not tested); - tests a system in any language — a
command:or HTTP target, and for TypeScript and JavaScript theevalproofnpm package'srunCommandandserve(TypeScript and JavaScript).
It is local-first: Ollama is the default, every major hosted provider is one target: line
away (Providers), a logged-in Claude Code or Codex can be the judge with no API
key, and every model call is cached by content so re-runs are free and reproducible. Results
from promptfoo, Inspect, DeepEval, lm-eval, garak, Google ADK and Claude Code plugin evals can be
imported and gated without re-running
them. The test cases you already have in promptfoo, DeepEval, Ragas, LangSmith, OpenAI Evals,
Inspect, Braintrust, Langfuse, Phoenix or a Google ADK eval set can be turned into a dataset and a
suite with evalproof import-dataset auto FILE --out cases.jsonl --suite-out suite.yaml, or when
you set up with evalproof init --from FILE
(Import test cases).
Teams that ship an LLM feature, a RAG pipeline or an agent and change its prompt, model, retrieval or tools often. Use it when you need to know whether a change made answers worse before it merges, whether a cheaper model is good enough, whether the LLM judge you score with can be trusted, and which model calls and tools in the codebase no test covers. It suits a team that already writes test cases in another tool and wants a CI gate that does not fail on noise. It is not a tracing or monitoring service and does not watch production traffic; it scores runs of a suite, locally or in CI.
evalproof is a command-line tool and a Python library, published on PyPI. It needs Python 3.11 to 3.14 and has four runtime dependencies (httpx, numpy, pydantic, PyYAML); no vendor SDK, no optional extras.
Run it without installing (needs uv):
uvx evalproof --version
uvx evalproof@1.4.0 init # pin a versionInstall the CLI on your PATH, isolated from your projects:
uv tool install evalproof # or: pipx install evalproof
evalproof --version # evalproof 1.4.0Add it to a project, to call it from Python or pin it with the project's other dev tools:
uv add --dev evalproof # or: pip install evalproof
uv run evalproof --versionUpgrade with uv tool upgrade evalproof, pipx upgrade evalproof or
pip install -U evalproof. Run files from older versions keep working; see
Upgrading for what to change and the changelog for everything
that changed.
In a container: docker run --rm -v "$PWD:/work" ghcr.io/technobasant/evalproof:latest --version
(Python 3.13, linux/amd64 and linux/arm64; no coding-agent CLIs in it, see
Docker).
In GitHub Actions, no install step is needed: the composite action
technobasant/evalproof@v1.4.0 runs the evalproof of the tag you pin, and pr-comment: "true"
posts the report as a pull request comment. In GitLab CI, evalproof init --platform gitlab
writes a file you include from .gitlab-ci.yml; in Bitbucket Pipelines,
evalproof init --platform bitbucket writes bitbucket-pipelines.yml
(In CI). evalproof hook install adds a
git pre-push hook that runs verify before every push.
Which credential a run needs depends on what the system and the judge are. Keyless mode needs no secrets: CI compares run files you committed and makes no model call. Fork pull requests never receive secrets, so no workflow runs a model for them.
| Target | Credential variable | Local use | CI use | Where allowed |
|---|---|---|---|---|
| No model: a Python function or an HTTP endpoint with no LLM behind it, or committed run files | none | nothing | nothing | anywhere, including fork pull requests |
| Local models (Ollama, vLLM, LM Studio, llama.cpp) | none | the server running, with the model pulled | a server on the runner, started in a workflow step | anywhere |
| Hosted APIs (Anthropic, OpenAI and the rest of Providers) | the provider's key, such as ANTHROPIC_API_KEY or OPENAI_API_KEY |
the variable in your environment | the key as a repository secret | your own repository; fork pull requests never receive it |
| Claude Code CLI | ANTHROPIC_API_KEY (api-key mode) or CLAUDE_CODE_OAUTH_TOKEN |
your Claude Code login, no variable | ANTHROPIC_API_KEY as a secret with EVALPROOF_CLI_AUTH=api-key, or CLAUDE_CODE_OAUTH_TOKEN as a secret (made by claude setup-token) |
the token: your own repository and your own CI only; use an API key for a secret shared across a team or an organisation |
| Codex CLI | OPENAI_API_KEY |
your Codex login, no variable | OPENAI_API_KEY as a secret with EVALPROOF_CLI_AUTH=api-key |
a personal Codex login only on trusted private infrastructure, never on public repos or for open source work |
evalproof init --ci api-key writes the workflow with the secret for the judge it chose, and
prints this requirement for that judge when it finishes. For Claude Code,
evalproof init --ci api-key --cli-auth oauth-token passes CLAUDE_CODE_OAUTH_TOKEN instead of
the API key and warns that it is for your own repository and CI only; Codex has no token option.
The full page adds what evalproof doctor reports.
Each is a suite file and one or two commands (use cases): gate a prompt or model change in a pull request; choose between two models or providers; find out whether your LLM judge can be trusted; use your Claude Code or Codex subscription as the judge; gate results you already have from another eval tool; check a retrieval or RAG change; hold latency and cost while you swap a model; gate in CI with no model calls and no secrets; test a non-LLM system the same way; start from test cases you already have in another tool; find the model calls, agents and tools no suite tests; test a TypeScript or JavaScript system; check the committed run before each push.
In your own repository, one command writes a starter suite, a CI workflow and an agent skill, and a second runs it:
uvx evalproof init
uvx evalproof run evals/suite.yaml --out evals/runs/latest.jsoninit picks the judge from what is on your PATH: a logged-in claude, then codex, then
Ollama, then a hosted Anthropic model with ANTHROPIC_API_KEY. It writes evals/ (the
suite, three sample cases, a README), .github/workflows/evalproof.yml (--ci keyless, the
default, makes no model calls in CI; --ci api-key runs the suite there; --ci none writes
no workflow) and a skill for coding agents at .agents/skills/evalproof/SKILL.md and
.claude/skills/evalproof/SKILL.md (--no-skill skips it). It also adds an evalproof section
to your AGENTS.md, between two markers, so any coding agent that reads that file knows which
command to run for a task, how to read the exit codes and what never to weaken
(--no-agents-md skips it; --agent-files-only writes only the section and the skill). Your
own text in that file is never changed. It prints one line per file, never overwrites one
without --force, and makes no model call. Replace the sample cases with your own: the gate
needs 20 before it will decide.
The offline example, the retrieval example and the LLM example are on the quickstart page.
The site at basantbhattarai.com.np/evalproof has:
- Getting started: install, what it needs to run, the quickstart, use cases.
- Guides: providers, testing agents, repeated trials, readiness, Agent SDK adapters, importing other tools' results, calling it from Python, CI, and troubleshooting.
- Reference: every command and flag, the suite configuration, the run file format, the exit codes, the concepts, the API and a glossary.
- About: architecture, decisions, how it relates to other tools, limits, versioning and support, security and data handling, upgrading, the roadmap, the measured run and the changelog.
Changelog · Contributing · Security policy · Code of conduct · How to cite. Releases are published to PyPI from GitHub Actions with trusted publishing, and each file carries a signed provenance attestation.