Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 3 additions & 36 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)

Pisama ships 32 core heuristic detectors. They apply across frameworks including n8n, LangGraph, Dify and OpenClaw, with per-platform gating (for example `coordination` runs only on multi-agent platforms). They run locally with zero LLM cost on the heuristic tier. An archived, in-distribution run on the [TRAIL benchmark](https://arxiv.org/abs/2505.08638) reports 59.9% joint accuracy (span and category). It is not a held-out result: 144 of the 148 traces appeared in calibration material, and the published archive does not contain the prediction-level data needed to recompute joint accuracy. The public confusion counts do reproduce the reported macro-F1 (0.7535) and micro-F1 (0.7463), but those F1 values carry no precision information: the archive recorded no false positives, so they reduce to recall (micro-recall 0.5953). See the [benchmark evidence and its reproducibility boundary](https://github.com/Pisama-AI/pisama-detectors/tree/main/benchmarks).
Pisama ships heuristic detectors that apply across frameworks including n8n, LangGraph, Dify and OpenClaw, with per-platform gating (for example `coordination` runs only on multi-agent platforms). They run locally with zero LLM cost on the heuristic tier.

## Install

Expand All @@ -33,7 +33,7 @@ pisama analyze trace.json # Analyze a trace
pisama watch python my_agent.py # Watch a live agent (pip install "pisama[auto]")
pisama replay <trace-id> # Re-run detection on stored traces
pisama smoke-test --last 50 # Batch test recent traces
pisama detectors # List all 32 core detectors
pisama detectors # List all core detectors
pisama mcp-server # Start MCP server (pip install pisama[mcp])
```

Expand Down Expand Up @@ -116,7 +116,7 @@ is the recommended path for new projects, one package instead of two.

## Detectors

32 core detectors, gated per platform (n8n, LangGraph, Dify, OpenClaw and others). A representative selection:
Core detectors, gated per platform (n8n, LangGraph, Dify, OpenClaw and others). A representative selection:

| Detector | What It Catches |
|----------|----------------|
Expand All @@ -141,39 +141,6 @@ is the recommended path for new projects, one package instead of two.
| `routing` | Inputs misrouted to the wrong specialist agent |
| `mcp_protocol` | MCP tool-communication failures |

## Benchmark Results

**TRAIL** (trace-level failure detection, 148 traces). Archived April 2026 run, **in-distribution and not held out**: 144 of 148 traces appeared in calibration material.

| Method | Joint Accuracy |
|--------|---------------|
| Pisama archived run | 59.9% |

Reproducible from the archive: macro-F1 0.7535, micro-F1 0.7463 (`benchmarks/verify_report.py`).

**Read those F1 figures as a recall measurement.** The archive scored only annotated errors,
so no false positive was recordable: `fp = 0` in 14 of 14 categories, and `prediction_count`
equals `mapped_annotations` (813). Precision is therefore 1.000 by construction rather than by
measurement, and F1 collapses to `2R / (1 + R)`, carrying no information beyond recall. The
informative figure is micro-recall 0.5953: the heuristic tier alone missed roughly 40% of the
labelled failures. The same boundary is recorded in
[`benchmarks/evidence.json`](https://github.com/Pisama-AI/pisama-detectors/blob/main/benchmarks/evidence.json)
and in the [`pisama-detectors` README](https://github.com/Pisama-AI/pisama-detectors#archived-trail-platform-benchmark).

Joint accuracy is **not** recomputable from the public archive, which lacks prediction-level
data. LLM-judge baselines are omitted here because
[`trail_llm_baselines.json`](https://github.com/Pisama-AI/pisama-detectors/blob/main/benchmarks/trail_llm_baselines.json)
carries confusion counts but `"result": null` for every model, so no comparable joint-accuracy
figure exists in the artifact. See the
[reproducibility boundary](https://github.com/Pisama-AI/pisama-detectors/tree/main/benchmarks).

**Who&When** (ICML 2025, multi-agent attribution, 58 hand-crafted cases):

| Method | Agent Accuracy | Step Accuracy |
|--------|---------------|---------------|
| GPT-5.4 Mini | 60.3% | 22.4% |
| **Pisama + Sonnet 4** | **60.3%** | **24.1%** |

## Links

- [Documentation](https://docs.pisama.ai)
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "pisama"
version = "0.6.0"
version = "0.6.1"
description = "Multi-agent failure detection for production AI systems"
readme = "README.md"
license = "MIT"
Expand Down
2 changes: 1 addition & 1 deletion src/pisama/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

__version__ = _pkg_version("pisama")
except PackageNotFoundError: # running from a source checkout without an install
__version__ = "0.6.0"
__version__ = "0.6.1"

from pisama._analyze import AnalyzeResult, Issue, analyze, async_analyze
from pisama._http import PisamaAuthError
Expand Down
Loading