evmbench is a benchmark and agent harness for finding and exploiting smart contract bugs.

How it works | Single-skill benchmarks | Result reports | Security | Key services | Repo layout | Quickstart (local dev)

This repository contains a companion interface to the evmbench detect evaluation. The evaluation code is included directly under frontier-evals/ so the application and benchmark data can be cloned and maintained together.

Upload contract source code, select an agent, and receive a structured vulnerability report rendered in the UI.

How it works

Architecture

Frontend (Next.js)
    │
    ├─ POST /v1/jobs/start ───► Backend API (FastAPI, port 1337)
    │                               ├─► PostgreSQL (job state)
    ├─ GET  /v1/jobs/{id}           ├─► Secrets Service (port 8081)
    │                               └─► RabbitMQ (job queue)
    └─ GET  /v1/jobs/history                │
                                             ▼
                                        Instancer (consumer)
                                              │
                                    ┌─────────┴──────────┐
                                    ▼                    ▼
                              Docker backend       K8s backend (optional)
                                    │                    │
                                    └────────┬───────────┘
                                             ▼
                                      Worker container
                                        ├─► Secrets Service (fetch bundle)
                                        ├─► (optional) OAI Proxy (port 8084) ──► OpenAI API
                                        └─► Results Service (port 8083)

End-to-end flow

1. User uploads a zip of contract files via the frontend. The UI sends the archive, selected model key, and (optionally) an OpenAI API key to /v1/jobs/start.
2. The backend creates a job record in Postgres, stores a secret bundle in the Secrets Service, and publishes a message to RabbitMQ.
3. The Instancer consumes the job and starts a worker (Docker locally; Kubernetes backend is optional).
4. The worker fetches its bundle from the Secrets Service, unpacks the uploaded zip to audit/, then runs Codex in "detect-only" mode:
   • prompt: backend/worker_runner/detect.md (copied to $HOME/AGENTS.md inside the container)
   • model map: backend/worker_runner/model_map.json (maps UI model keys to Codex model IDs)
   • command wrapper: backend/worker_runner/run_codex_detect.sh
5. The agent writes submission/audit.md. The worker validates that the output contains parseable JSON with {"vulnerabilities": [...]} and then uploads it to the Results Service.
6. The frontend polls job status and renders the report with file navigation and annotations.

Single-skill benchmarks

EVMbench supports a raw baseline and exactly one local Codex skill package per worker-image build. This lets you compare a selected methodology without changing the frontend, API, database, queue, scorer, or model configuration.

The harness never downloads or vendors third-party skills. First clone or otherwise obtain a source repository yourself, inspect it, and select the specific directory that represents one package (not its repository root unless that root is itself one package). A valid package must contain exactly one SKILL.md at its root and no nested SKILL.md files. References and helper scripts may remain alongside it.

Run the raw benchmark

make prepare-skill SKILL=raw
cd backend
docker build -t evmbench/worker:latest -f docker/worker/Dockerfile .
# Start the normal stack and submit the benchmark as usual.

Raw mode stages nothing and preserves the existing EVMbench prompt exactly.

Run a benchmark for a selected skill

Repeat these steps independently for each selected QuillShield, Pashov, Cyfrin Solskill, or other compatible skill directory:

# Replace this with the single directory containing the chosen root SKILL.md.
make prepare-skill SKILL=/absolute/path/to/specific-skill
cd backend
docker build -t evmbench/worker:latest -f docker/worker/Dockerfile .
# Start the normal stack and submit the benchmark as usual.

The preparation command fails before image construction for missing, malformed, multi-skill, or symlink-containing inputs. For a staged package, the worker installs the complete package at $HOME/.codex/skills/<skill-name>, logs its name and any available source Git revision, and tells Codex to read the full SKILL.md and use its methodology. EVMbench's audit scope and JSON submission requirements continue to override any conflicting skill directions.

Reset to raw before rebuilding the baseline image. The full worker-runner details are in backend/worker_runner/README.md.

Benchmark result reports

Canonical evaluation runs can be converted into clean Markdown, JSON, and CSV bundles under frontier-evals/project/evmbench/reports/:

cd frontier-evals/project/evmbench
uv run python scripts/report_results.py \
    runs/<run-group> \
    --name my-benchmark \
    --title "My benchmark"

Multiple run groups can be supplied when a failed audit was retried separately. The latest graded attempt for each audit is reported, and earlier attempts remain included in retry and usage accounting. See the evaluation README for options and output details.

Security

evmbench runs an LLM-driven agent against uploaded, untrusted code. Treat the worker runtime (filesystem, logs, outputs) as an untrusted environment.

See SECURITY.md for the full trust model and operational guidance.

OpenAI credential handling:

• Direct BYOK (default): worker receives a plaintext OpenAI key (OPENAI_API_KEY / CODEX_API_KEY).
• Proxy-token mode (optional): worker receives an opaque token and routes requests through oai_proxy (plaintext key stays outside the worker).

Enabling proxy-token mode:

cd backend
cp .env.example .env
# set BACKEND_OAI_KEY_MODE=proxy and OAI_PROXY_AES_KEY=...
docker compose --profile proxy up -d --build

Operational note: worker runtime is bounded by default; override the max audit runtime with EVM_BENCH_CODEX_TIMEOUT_SECONDS (default: 10800 seconds).

Key services

Service	Default port	Role
backend	1337	Main API: job submission, status, history, auth
secretsvc	8081	Stores and serves per-job secret bundles (zip + key material)
resultsvc	8083	Receives worker results, validates/parses, persists to DB
oai_proxy	8084	Optional OpenAI proxy for proxy-token mode
instancer	(n/a)	RabbitMQ consumer that starts worker containers/pods
worker	(n/a)	Executes the detect-only agent and uploads results
Postgres	5432	Job state persistence
RabbitMQ	5672	Job queue

Repo layout

.
├── README.md
├── SECURITY.md
├── LICENSE
├── frontend/                 Next.js UI (upload zip, select model, view results)
├── frontier-evals/           Evaluation framework and benchmark data
│   └── project/evmbench/reports/  Generated Markdown, JSON, and CSV result bundles
├── backend/
│   ├── api/                  Main FastAPI API (jobs, auth, integration)
│   ├── instancer/            RabbitMQ consumer; starts workers (Docker/K8s)
│   ├── secretsvc/            Bundle storage service
│   ├── resultsvc/            Results ingestion + persistence
│   ├── oai_proxy/            Optional OpenAI proxy (proxy-token mode)
│   ├── prunner/              Optional cleanup of stale workers
│   ├── worker_runner/        Detect prompt + model map + Codex runner script
│   ├── docker/
│   │   ├── base/             Base image: codex, foundry, slither, node, tools
│   │   ├── backend/          Backend services image
│   │   └── worker/           Worker image + entrypoint
│   └── compose.yml           Full stack (DB/MQ + services)
└── deploy/                   Optional deployment scripts/examples

Quickstart (local dev)

Ensure Docker and Bun are available.

Build the base and worker images first (required before starting the stack):

cd backend
docker build -t evmbench/base:latest -f docker/base/Dockerfile .
docker build -t evmbench/worker:latest -f docker/worker/Dockerfile .

Start backend stack (API + dependencies):

cp .env.example .env
# For local dev, the placeholder secrets in .env.example are sufficient.
# For internet-exposed deployments, replace them with strong values.
docker compose up -d --build

Start frontend dev server:

cd frontend
bun install
bun dev

Open:

• http://127.0.0.1:3000 (frontend)
• http://127.0.0.1:1337/v1/integration/frontend (backend config endpoint)

Acknowledgments

Thank you to many folks on the OtterSec team for support, particularly with building the frontend: es3n1n, jktrn, TrixterTheTux, sahuang