agent-redteam

LLM-agent security failures are hard to reproduce and audit, so agent-redteam runs authorized adversarial probes through evidence-producing oracles and reusable input, output, and tool guardrails.

Run the offline proof

From a clone of this repository, this command installs the project and scans the deterministic fake target without an API key or network access:

python -m pip install -e ".[dev]"
agent-redteam scan --config examples/target.fake.yaml --suite smoke

The checked-in fixture currently runs six attacks and prints a terminal summary. On 2026-08-03 the command completed with attacks: 6, succeeded: 0, errors: 0, and max score: 0.0. Those numbers describe the deliberately safe fake target; they are a harness smoke test, not a security score for a real model.

What the tests prove

python -m pytest -o addopts= -q -p no:cacheprovider
ruff check .

The full local suite produced 153 passed, 1 warning in 2.94s; Ruff reported All checks passed!. The tests exercise authorization and allowlist refusals, attack selection, deterministic and semantic-judge adapters, shared budgets, baseline regression checks, input/output/tool guardrails, offline record/replay, agentic poisoned-retrieval attribution, causal-proof tamper detection, report formats, CLI behavior, and the FastAPI surface. All target/model responses in that run were local fakes or recorded fixtures.

Not implemented / not proven

• The offline result does not measure resistance of any live model or deployed agent, and the repository does not claim that passing its suite makes a system secure.
• No external benchmark currently quantifies oracle false-positive or false-negative rates.
• Live OpenAI-compatible targets and optional LLM judges require credentials and were not exercised by the test result above.
• The MCP adapter exists, but the default suite has no end-to-end MCP transport test.
• The CLI keeps real side effects disabled; agentic tests record or simulate tool effects rather than sending money, email, or other irreversible actions.

Scope of use. This is a defensive tool for testing systems you own or have written permission to assess. A run is refused unless the target config asserts authorized: true and the target host is on your allowlist. It plants synthetic canaries, never real secrets, and never touches third-party systems. See Responsible use.

Install

pip install prathamesh-agent-redteam            # core
pip install "prathamesh-agent-redteam[llm]"     # + OpenAI-compatible target & judge
pip install "prathamesh-agent-redteam[server]"  # + FastAPI server
pip install "prathamesh-agent-redteam[mcp]"     # + MCP server

Python 3.11+.

Configure a target

1. Point it at a target

Create target.yaml:

target:
  name: my-support-bot
  kind: openai_chat
  authorized: true                 # you are asserting you may test this
  allowlist: [api.openai.com]      # hosts probes may be sent to
  options:
    base_url: https://api.openai.com/v1
    model: gpt-4o-mini
    system_prompt: "You are a helpful support assistant."
run:
  suite: default
  fail_threshold: 7.0

Run the suite

export OPENAI_API_KEY=sk-...
agent-redteam scan --config target.yaml --report report.md

Exit code is non-zero when the run fails its threshold, so it drops straight into a pipeline.

Add guardrails and measure the delta

agent-redteam scan --config target.yaml --guardrails default --compare

The comparison reports the actual undefended and defended findings from that run; this README does not publish a canned improvement number.

Targets

kind	Wraps	Notes
openai_chat	any OpenAI-compatible /chat/completions	OpenAI, xAI/Grok, vLLM, Ollama
http	an arbitrary JSON HTTP agent	request/response mapped by template
callable	a local Python function	in-process agents, unit tests
fake	a scripted rule table	deterministic; used throughout the tests
fake_agent	a resettable RAG/tool agent	offline agentic POC and CI fixture

Agent and RAG testing

Chat-only probes cannot tell whether a poisoned document actually caused a tool to execute. Agentic scenarios are therefore opt-in and require an episode-aware target:

target:
  name: local-agent-poc
  kind: fake_agent
  authorized: true
run:
  suite: tag:agentic
  agentic: true
  seed: 7
  max_calls: 4

agent-redteam scan --config examples/agentic-target.yaml --agentic --json report.json
python examples/agentic_rag_poc.py

Each agentic finding includes a typed event graph, the untrusted retrieval event, the path from that event to the side effect, a clean-fixture counterfactual, a root-cause group, and machine-readable guardrail configuration recommendations. JSON reports also include a causal_proof bundle: every event is hash-chained and the poisoned trace, clean twin, and attribution claim share one SHA-256 content address. verify_causal_proof detects post-run edits without rerunning a model. The shipped POC proves an undefended poisoned retrieval reaches a simulated send_email, while ToolCallPolicy prevents its executor from running. Copy a report recommendation's config_patch into YAML and apply it directly:

agent-redteam scan --config examples/agentic-target.yaml --agentic \
  --guardrail-config examples/agentic-guardrails.yaml --compare

For a real in-process agent, wrap an async handler with CallableEpisodeTarget. Feed retrieved artifacts through EpisodeInstrumentation.retrieval_result and invoke tools only through EpisodeInstrumentation.execute_tool; this is the enforcement point that runs the existing GuardPipeline before the application's executor. The handler must consume the returned ArtifactUse.artifact, which contains any policy rewrite, and ignore blocked artifacts. Live side effects are disabled by default, and the same authorization and shared budget gate applies to episode and clean-twin calls.

Attack library

Grouped by category, each attack carries a stable id, OWASP-LLM / MITRE ATLAS references, and a fixture proving it detects a real vulnerability and is stopped by the matching guardrail.

• prompt_injection — instruction override, prefix injection, refusal suppression
• jailbreak — role-play, persona (DAN-style), hypothetical framing
• exfiltration — system-prompt leak, credential leak, markdown-image exfil channel
• tool_abuse — unauthorized tool use, argument injection, SSRF via tools
• obfuscation — base64, leetspeak, unicode homoglyph, translation smuggling
• multi_turn — crescendo / gradual escalation
• resource_exhaustion — denial-of-wallet (opt-in; costs tokens)

agent-redteam list-attacks           # full catalog with ids and references
agent-redteam scan --suite exfiltration --config target.yaml

Adaptive attacks

A static payload asks "does this fixed prompt work?" An adaptive attack asks "what works after watching how this target fails?" — an attacker model reads the target's real response and refines the next payload toward the oracle's success criterion, in a bounded loop (PAIR / Crescendo strategies). This finds target-specific bypasses a fixed corpus misses.

agent-redteam scan --config target.yaml --suite tag:adaptive \
  --adaptive --attacker-model gpt-4o-mini

Every adaptive finding records a full step-by-step trace in the JSON report (each payload the loop tried and how the target answered), so it is as auditable as a static finding. It is budget-safe and gated: the same authorization check applies, and a shared budget ledger caps target calls, attacker calls, tokens, and wall-clock — an adaptive run can never runaway-spend. The CLI prints the attacker model id and the hard caps before it starts.

Guardrails

Composable middleware that wraps any target into a defended one:

• Input — encoding normalizer, injection detector, allowlist
• Output — secret/PII scanner, canary scanner, exfil-URL blocker
• Tool — tool-call policy (allow/deny + argument schema + SSRF host checks)

from agent_redteam.guardrails import GuardPipeline, default_guardrails

defended = default_guardrails().wrap(my_target)

Scoring

risk (0-10) = base_severity(category) × success_confidence × exploitability

Each factor is orthogonal and printed in a recomputable vector, e.g. ART/C:exfiltration/B:9.5/S:0.92/E:0.8 → 7.0. A run fails if any attack meets fail_threshold or if successes regress against a saved baseline (agent-redteam baseline save|compare).

Semantic judging (optional)

Most attacks are scored by deterministic oracles (a planted canary either leaked or it didn't). For the few whose success is genuinely semantic, add an LLM-as-judge — pinned to temperature 0 and a strict JSON rubric for repeatability:

agent-redteam scan --config target.yaml --judge-model gpt-4o-mini
# --judge-base-url / --judge-key-env point it at any OpenAI-compatible endpoint

The judge is fail-safe: any transport or parse error scores the attack as not successful, so a flaky judge can never manufacture a finding.

Interfaces

• CLI — scan, list-attacks, list-guardrails, report, baseline
• FastAPI — POST /scan, GET /report/{id}, GET /attacks
• MCP — run_attack_suite and check_guardrail tools for agent platforms
• Reports — JSON (canonical), Markdown (human), JUnit XML, redacted SARIF 2.1.0

Responsible use

agent-redteam is built for authorized security testing and AI-safety research.

• Runs are refused unless the target asserts authorized: true and the host is allowlisted (loopback is implicitly allowed for local testing).
• Payloads are adversarial inputs, not weaponized exploits; canaries are synthetic tokens, never real credentials.
• No third-party systems are ever contacted, and no data is exfiltrated anywhere — the "exfil" attacks prove a channel exists by leaking a planted canary back to you, nothing more.

Use it on your own agents. Don't point it at systems you don't have permission to test.

Development

pip install -e ".[dev,llm,server,mcp]"
pytest              # full suite, no API key required (FakeTarget/FakeJudge)
ruff check .

License

MIT. See LICENSE.