@icodenet/eval-dashboards

Standardized evaluation artifacts + beautiful dashboards for AI agent and LLM evals.

Docs site (start here):

Live: https://icodenet.github.io/eval-dashboards/v1/
In-repo source: docs-site/v1/index.html

Completion Check Ledger

Before marking work as finished, run the checklist in CHECKS_LEDGER.md. This applies to schema, reporters, gates, history, publish flow, and docs changes.

Emit a taxonomy-complete eval-report/v1 JSON artifact from any runner (Vitest, Jest, custom Node code, Python, etc.), then use eval-dashboards to generate reports, enforce quality gates, track history, and publish static dashboards. In this repo, eval-report/v1 means version 1 of the shared JSON contract described in docs/artifact-format.md. No platform sign-up, no vendor lock-in.

Same mental model as NYC/Istanbul for code coverage — but for LLM and agent evals. Schema-first, offline-first, runner-agnostic.

Why eval-dashboards?

Feature	@icodenet/eval-dashboards	Full-Featured Platform
Standardized Schema	✅ JSON Schema + teaching docs	⚠️ Vendor-specific
Offline Reports	✅ Pure static HTML	⚠️ Requires server
Bring Your Own Runner	✅ Vitest, Jest, custom Node, Python, etc.	❌ Must use platform's harness
CI/CD Integration	✅ GitHub Actions, Azure Pipelines examples included	⚠️ Usually provided
Open Source	✅ MIT licensed	❌ Proprietary
Zero Lock-in	✅ Artifact is just JSON; export anytime	❌ Your data is in their system

The core idea: Your eval results are valuable even without dashboards. Standardize the artifact format first, then layer beautiful UI on top—not the other way around.

Quick start

pnpm add -D @icodenet/eval-dashboards

CLI names after install:

eval-dashboards --help
evd --help

If you want a custom local alias, add it in your shell profile, for example:

alias myeval='evd'

See it work first (no runner required)

You do not need an eval runner, an API key, or a network connection to get a real dashboard. init --write scaffolds a working example artifact, so these three commands take you from empty directory to open dashboard:

# 1. Scaffold config, dataset, rubric, CI snippet and one example artifact
eval-dashboards init --preset=agent-quality --write

# 2. Generate an HTML dashboard from it
eval-dashboards report --input=.evals_output --reporter=html --report-dir=eval-report

# 3. Enforce quality gates (non-zero exit if gates fail)
eval-dashboards check --input=.evals_output --min-pass-rate=0.9 --max-new-failures=0

Step 2 prints eval-report/index.html — open it in any browser. Step 3 prints Eval gates passed. and exits 0.

If your project's package.json does not have "type": "module", step 1 scaffolds eval-dashboards.config.ts as an ESM file and every subsequent command will print a harmless Node ESM-load warning to stderr (it falls back to defaults). Either add "type": "module" to package.json, or rename the scaffolded file to eval-dashboards.config.mjs.

Use --dry-run to see exactly which files would be written before committing to anything, or --teach for a guided walkthrough that writes nothing.

Then wire in your own runner

Replace the scaffolded artifact with real output from your own eval run:

# 1. Emit a taxonomy-complete artifact from your runner
my-eval-runner --output=.evals_output/run-2026-08-05T170000Z.json

# 2. Generate an HTML dashboard
eval-dashboards report --input=.evals_output --reporter=html

# 3. Enforce quality gates (non-zero exit if gates fail)
eval-dashboards check --input=.evals_output --min-pass-rate=0.9 --max-new-failures=0

# 4. Publish to GitHub Pages
eval-dashboards publish --target=github-pages --repo=owner/repo

my-eval-runner is a placeholder for whatever produces your results — Vitest, Jest, a Node script, a Python harness. The scaffolded .evals_output/run-agent-quality-template.json is the shape it needs to emit; see docs/artifact-format.md for the full contract.

Use one file per run in .evals_output instead of overwriting a single artifact. History-preserving output enables baseline selection (rolling / champion) and trend reports.

Important concepts for reliable CI gates:

Preflight first: emit and require a lightweight preflight suite before expensive live/judge suites.
Canonical new failures: count regressions by scenario/category when one scenario can emit multiple rows.
Warning budgets: treat taxonomy warnings as explicit risk budgets, not invisible noise.
Sanitized config snapshots: include non-secret runtime context in run.configSnapshot for triage.

Common CI gate recipes:

PR policy (rolling baseline):

eval-dashboards check --input=.evals_output --baseline-strategy=rolling --allow-blocked-baseline --max-new-failures=0 --zero-critical

Main policy (champion baseline from recent history):

eval-dashboards check --input=.evals_output --baseline-strategy=champion --baseline-lookback=20 --max-new-failures=0 --zero-critical

Strict live policy with preflight + warning controls:

eval-dashboards check --input=.evals_output --require-suite-pass=preflight --new-failure-key=scenario-category --max-new-failures=0 --max-warnings=5 --max-warning-code=missing-kind:0 --fail-on-warning-code=missing-judge-model --zero-critical

Optional statistical policy (bootstrap CI on pass-rate delta vs baseline):

eval-dashboards check --input=.evals_output --baseline-strategy=rolling --statistical-mode=bootstrap --confidence-level=0.95 --bootstrap-samples=2000 --min-pass-rate-delta=0

See working examples:

Vitest: examples/vitest-evals/README.md
Jest: examples/jest-custom-reporter/README.md
Plain Node/TypeScript: examples/node-plain-eval/README.md
Python / Pytest: examples/python-pytest-evals/README.md
LangChain Evaluators: examples/langchain-evals/README.md
Agent quality preset: examples/agent-quality-preset/README.md
Concrete report-power artifacts: examples/report-power-artifacts/README.md

Visual Gallery

The HTML dashboard is fully responsive and works in both light and dark themes. View tracked fixture examples:

Light theme dashboard — Default presentation with light background
Dark theme dashboard — Dark mode with reduced eye strain

Light Theme	Dark Theme

Features Visible in Dashboard

Metric Cards (top)

Pass rate with status coloring (orange for warning, red for fail, green for pass)
Passed/total row count
New failures and new passes since previous run
Baseline compatibility status (compatible, warning, or blocked)

Pass-Rate Trend (historical view)

Mini sparkline showing pass-rate over time
Trend direction indicator: ↑ improving, ↓ regressing, → stable
Percentage change calculation comparing current to baseline

Suite Summary (breakdown by test suite)

Collapsed by default with one-line summary and expand/collapse affordance
Counts: total, passed, failed
Pass-rate bar chart per suite
Suite pills include hover tooltips with pass rate and pass/fail counts

Failing Rows (grouped view)

Rows organized hierarchically by dataset → scenario
Section collapsed by default; expand only when triaging regressions
Taxonomy completeness score (0–100%) with visual indicators
Kind badges (deterministic, agent, llm-judge, human-review)
Severity chips (low, medium, high, critical)

All Rows (complete inventory)

Same hierarchical grouping as failing rows
Section collapsed by default to keep first view focused on summary signals
Includes both passing and failing results
Full context for auditing and learning

Concrete report-power artifacts (tracked in git)

Use examples/report-power-artifacts/README.md as the entrypoint for locally openable proof artifacts:

History trends: examples/report-power-artifacts/report/history.json
Progress over runs: examples/report-power-artifacts/report/summary.json
Gate outcomes (pass + fail):
- examples/report-power-artifacts/gates/check-pass.json
- examples/report-power-artifacts/gates/check-fail.json
Row-level/detail analysis:
- examples/report-power-artifacts/report/summary.json (comparison.persistentFailures, comparison.disappeared)
- examples/report-power-artifacts/report/index.html

Inline artifact preview (GitHub-rendered images):

Progress	History

Gating	Detail analysis

Regenerate deterministically from fixture input with:

./scripts/generate-report-power-artifacts.sh
pnpm exec tsx scripts/generate-report-power-images.ts

Teach labs by delivery stage (local loop -> pre-PR -> triage -> release -> post-release) and an FDE workflow are in docs/teach-labs/.

Recommended pre-read before Teach Curriculum or Lab 01:

Langfuse 101: Evals Feedback Loop — read this first to understand the operating model (live evidence -> review -> dataset/versioning -> experiments -> release gates) and why evals are continuous, not one-off.

Schema & Taxonomy

JSON Schema

The eval-report/v1 format is documented as JSON Schema Draft 7. Use it to:

Validate artifacts in CI
Generate SDKs in any language
Discover compatible tools

# Download the schema
curl https://raw.githubusercontent.com/icodenet/eval-dashboards/main/schemas/eval-report-v1.schema.json > my-runner/eval-report-v1.schema.json

Taxonomy Teaching Guide

docs/taxonomy.md defines what a "taxonomy-complete" eval report looks like and why it matters. It includes:

Row taxonomy: Required fields (id, suite, passed), classification (kind, severity, category), evidence (input, output, turns, toolCalls, axisScores)
Suite taxonomy: Manifests, gate policies, rubric contracts, versioning
Implementation checklist: Copy-paste patterns for Vitest, Jest, plain Node
Real-world example: Full artifact with multiple suites, judges, and tool calls
FAQ: What fields are required? Optional? Can I add custom fields?

Start here to understand what to emit.

docs/schema-taxonomy-decisions.md records which setup concepts belong in the shared schema, preset files, examples, docs, adapter helpers, or host applications.

Artifact Format Reference

Your runner emits eval-report/v1 JSON:

{
  schemaVersion: 'eval-report/v1',
  run: { 
    id, generatedAt, project, team, branch, commit, buildId 
  },
  suites: [
    { id, name, total, passed, failed }
  ],
  rows: [{
    // Required
    id, suite, passed,
    
    // Classify (recommended)
    kind?,          // 'deterministic' | 'agent' | 'llm-judge' | 'human-review'
    severity?,      // 'none' | 'low' | 'medium' | 'high' | 'critical'
    category?,      // e.g., 'timeout', 'pii-leaked', 'off-topic'
    
    // Evidence (depends on kind)
    input?, output?, expected?,
    turns?,         // ConversationTurn[] for agents
    toolCalls?,     // ToolCall[] for agent actions
    judgeModel?, judgeVerdict?, judgeReasoning?,
    axisScores?,    // Record<string, number> for graded evaluations
    
    // Versioning & tracking
    datasetId?, scenarioId?, rubricId?,
    durationMs?,
  }],
  suiteManifests?: [{
    name, owner, riskArea, datasetVersion, rubricVersion,
    gate: { mode: 'blocking', thresholds: { passRate, zeroCritical } }
  }],
}

Full format documentation | JSON Schema | Taxonomy guide

Runner Adapter Helpers

If you already have an eval runner, use the public adapter helpers to turn local case results into a validated eval-report/v1 artifact:

import { writeEvalReportArtifact } from '@icodenet/eval-dashboards';

await writeEvalReportArtifact('.evals_output/run.json', {
  run: { id: process.env.BUILD_ID ?? 'local-run', project: 'my-agent' },
  cases: [
    { id: 'case-1', suite: 'answer-quality', passed: true, severity: 'none' },
    { id: 'case-2', suite: 'mcp-routing', passed: false, severity: 'high' },
  ],
});

createEvalReportArtifact(...) computes suite totals from rows and validates the artifact before returning it. writeEvalReportArtifact(...) writes the JSON and can clean the output directory when cleanOutputDir: true is passed.

Set cleanOutputDir: true only when you explicitly want a single-file snapshot workflow. Most teams should keep run history files for baseline-aware checks and trend reporting.

CLI Commands

Command	Description
`eval-dashboards report`	Generate HTML, text, Markdown, or JSON-summary dashboards from artifacts (`--profile=guardrail` for attack-focused triage)
`eval-dashboards report-index`	Generate grouped multi-report HTML index from discovered artifacts
`eval-dashboards lint`	Run fast semantic/taxonomy preflight checks before expensive eval runs
`eval-dashboards check`	Enforce pass-rate, new-failure, critical-severity, and suite-manifest gates
`eval-dashboards publish`	Publish dashboard to dir, github-pages, or Azure Storage (Azure Static Web Apps: dry-run validation only). Hard-fails on unredacted sensitive evidence unless `--redact` or `--allow-sensitive-publish` is set
`eval-dashboards history`	Build a history JSON trend file from discovered artifacts (pass-rate over time, etc.)
`eval-dashboards merge`	Merge multiple artifacts into one
`eval-dashboards teach`	Guided onboarding walkthrough (alias of `init --preset=agent-quality --teach`)
`eval-dashboards init`	Print a starter config, or scaffold preset files with `--preset=agent-quality --write`
`eval-dashboards completion`	Print or install shell completion for bash, zsh, or fish
`eval-dashboards import`	Convert third-party eval output JSON to `eval-report/v1` (Promptfoo, DeepEval, AgentEvals, Ragas, Langfuse, eval-ai-library; `openevals` alias supported)
`eval-dashboards adjudicate`	Export unresolved rows for reviewer adjudication bundles and merge reviewer verdicts back into artifacts
`eval-dashboards sign`	Hash a check-result artifact and write a detached signature (cosign keyless in CI with OIDC; honest `unavailable` fallback locally)
`eval-dashboards verify`	Re-validate a check-result artifact's digest and signature; fails closed on tampering, staleness, or a missing/unavailable signature
`eval-dashboards heartbeat-verify`	Scheduled check that a fresh, healthy gate-run heartbeat exists — makes a deleted/skipped gate step on a release detectable
`eval-dashboards org-rollup`	Render one static offline HTML overview from N published per-repo `history.json` artifacts (pass rate, critical failures, drift trend)

Help quick checks (truth-sync targets):

eval-dashboards report --help
eval-dashboards check --help
eval-dashboards publish --help
eval-dashboards import --help
eval-dashboards teach --help
eval-dashboards init --help

Shell completion quick setup:

# auto-install for current shell (works for eval-dashboards + evd)
eval-dashboards completion install

# manual install examples
eval-dashboards completion --shell=bash > ~/.eval-dashboards-completion.bash
source ~/.eval-dashboards-completion.bash

# import Promptfoo JSON output into eval-report/v1
eval-dashboards import --from=promptfoo --input=./promptfoo-results.json --out=.evals_output/import-promptfoo.json

# render guardrail-focused triage section for attack-style suites
eval-dashboards report --input=.evals_output --reporter=html --profile=guardrail --report-dir=eval-report

# export unresolved rows for reviewer adjudication
eval-dashboards adjudicate export --input=.evals_output --out=eval-report/adjudication-bundle.json

# merge reviewer verdicts back into a run artifact
eval-dashboards adjudicate import --input=.evals_output --bundle=eval-report/adjudication-bundle-reviewed.json --out=eval-report/adjudicated-run.json

Configuration

// eval-dashboards.config.ts
export default {
  input:     ['.evals_output/**/*.json'],
  reportDir: 'eval-dashboard',
  reporters: ['html', 'json-summary'],
  theme:     'dark',
  locale:    'en-GB',
  gates: {
    minPassRate:    0.9,
    maxNewFailures: 0,
    zeroCritical:   true,
  },
};

See docs/configuration.md for all options.

Themes

Three built-in themes. Switch with --theme or set in config:

eval-dashboards report --theme=dark
eval-dashboards report --theme=minimal
eval-dashboards report --theme=default

# Compare any two runs directly
eval-dashboards report --input=.evals_output --run-id=run-2026-08-03 --baseline-run-id=run-2026-07-28 --reporter=html

Bring your own brand colors:

export default {
  theme: {
    name: 'default',
    variables: {
      '--banner-bg': '#1a1a2e',
      '--accent':    '#e94560',
      '--pass':      '#0f3460',
    },
  },
};

Adopt With An Agent Skill

Already have an eval runner and just want dashboards, gates, and history without hand-wiring the adapter? Install the skill directly into your repo:

npx skills add IcodeNet/eval-dashboards -s eval-dashboards-adopt -y

This clones the skill into .agents/skills/eval-dashboards-adopt/, wired for Claude Code, Codex, GitHub Copilot, OpenCode, Hermes Agent, and other agents. Then point your coding agent at it — it inspects your existing eval command, writes a thin adapter with writeEvalReportArtifact, wires check/report into your CI, and opens a PR. See skills/eval-dashboards-adopt/SKILL.md for the full procedure.

Examples

TypeScript / Node.js:

Vitest evals example — emit artifacts from test assertions
Jest reporter example — custom reporter emitting artifacts
Plain Node example — run eval logic and emit artifacts
Taxonomy-complete fixture — template showing all recommended fields
Basic JSON trace-link fixture — row-level trace/span deep links for dashboard triage

Python:

Pytest evals example — conftest.py plugin that collects rows and writes eval-report/v1 artifacts after your pytest session
LangChain Evaluators — wrap LangChain's built-in evaluators (QA, criteria, embedding) to emit taxonomy-complete rows

CI/CD:

Versioned composite GitHub Action — uses: IcodeNet/eval-dashboards@v0.7.0 (report + gate + optional publish in one step)
GitHub Actions
GitHub Actions approval gate
GitHub Actions PR cleanup
Azure Pipelines

Adoption & Community

This project is in active development (v0.x). Core schema and API are stabilizing. We're looking for:

Release validity note: because the branch history was rewritten to match the current codebase, only the latest release tied to this cleaned history should be treated as valid for the current code state. Earlier release artifacts are superseded and should not be used as references for the present implementation.

Eval runner authors — integrate eval-dashboards as a native reporter option
Teams using custom evals — adopt the schema and share feedback
Contributors — improve HTML styling, add publishing targets, expand examples

Current metrics (targeting by 2026-Q4):

✅ Schema + taxonomy foundation complete (drift guard hardening in progress)
⏳ 5+ external runners discovering this project
⏳ 1 runner emitting taxonomy-complete artifacts
⏳ 100+ npm downloads/week

How to help:

Try one of the examples
Emit an artifact from your runner and share feedback (GitHub discussions)
Report issues or suggest improvements
Star the repo if you find it useful!

Documentation Index

Full audience-grouped index: docs/README.md. docs/*.md is the canonical source of truth; docs-site/v1 is a curated subset for external readers.

Start here:

Artifact format — the eval-report/v1 contract, field by field
Taxonomy teaching guide — what makes a "complete" eval report
Configuration — all config options

Contract & schema:

Artifact format
Taxonomy
JSON Schema — for validation and SDK generation
Schema/taxonomy decision rules
Benchmark pack templates — versioned safety/tool-routing/groundedness suite bundles

CI & governance:

Gates — quality gates and CI integration
Publishing — GitHub Pages, Azure, custom
GitHub approval-gate pattern — reviewer approvals + commit-status gating
Repo hardening — sign/verify, waivers, heartbeat, bypass tracking
CLI help snapshots — exact --help outputs tracked for truth-sync

Teaching:

Teach curriculum — detailed novice path from synthetic dataset to gates/history
Teach exercises — foundational hands-on exercises
Teach labs — delivery-stage workflow labs
Langfuse 101: Evals Feedback Loop — conceptual pre-read before curriculum/labs

Product & roadmap:

Static docs site (v1) — versioned quickstart + CLI-first onboarding
Roadmap — phases and adoption plan
Status — done vs. remaining, honest completion ledger
Comparison with NYC/Istanbul
Adoption feedback loop — weekly page-to-action checks, usage markers, and cadence signals
Docs adoption friction backlog — documentation blockers tracked with owner/severity/next action
Case study: adopting eval-dashboards in assistant-ui — first real external-repo adoption, with real bugs found and fixed

Integrations:

Integrations ("Works with" guides) — Promptfoo, DeepEval, OpenAI/AgentEvals, Anthropic methodology, Langfuse, Weave, Phoenix, Braintrust, Ragas, TruLens, Patronus, trace stacks

Reference:

Reporters — HTML, text, Markdown, JSON

See examples/ for runnable demos.

Name		Name	Last commit message	Last commit date
.github		.github
.vscode		.vscode
docs-site		docs-site
docs		docs
eval-report-dark		eval-report-dark
examples		examples
github		github
schemas		schemas
scripts		scripts
skills		skills
src		src
test		test
.editorconfig		.editorconfig
.gitignore		.gitignore
.npmignore		.npmignore
.releaserc.json		.releaserc.json
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CHECKS_LEDGER.md		CHECKS_LEDGER.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SOUL.md		SOUL.md
action.yml		action.yml
agent-checklists.md		agent-checklists.md
agent-prompts.md		agent-prompts.md
commitlint.config.cjs		commitlint.config.cjs
lefthook.yml		lefthook.yml
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
tsconfig.json		tsconfig.json
tsup.config.ts		tsup.config.ts
verify-cli-init-and-completion.sh		verify-cli-init-and-completion.sh
vitest.config.ts		vitest.config.ts

Navigation Menu

Repository files navigation

@icodenet/eval-dashboards

Completion Check Ledger

Why eval-dashboards?

Quick start

See it work first (no runner required)

Then wire in your own runner

Visual Gallery

Features Visible in Dashboard

Concrete report-power artifacts (tracked in git)

Schema & Taxonomy

JSON Schema

Taxonomy Teaching Guide

Artifact Format Reference

Runner Adapter Helpers

CLI Commands

Configuration

Themes

Adopt With An Agent Skill

Examples

Adoption & Community

Documentation Index

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

@icodenet/eval-dashboards

Completion Check Ledger

Why eval-dashboards?

Quick start

See it work first (no runner required)

Then wire in your own runner

Visual Gallery

Features Visible in Dashboard

Concrete report-power artifacts (tracked in git)

Schema & Taxonomy

JSON Schema

Taxonomy Teaching Guide

Artifact Format Reference

Runner Adapter Helpers

CLI Commands

Configuration

Themes

Adopt With An Agent Skill

Examples

Adoption & Community

Documentation Index

License

About

Topics

Resources

Code of conduct

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages