Home

EvalCI Wiki

EvalCI is a zero-infra GitHub Action that runs @eval_case quality suites on every PR and blocks merge if scores drop below your threshold.

→ Full documentation: https://synapsekit.github.io/synapsekit-docs/docs/evalci/overview

---

Quick links

Page	Description
Overview	What EvalCI does and how it works
Quickstart	Get running in under 5 minutes
Writing Eval Cases	@eval_case decorator reference
Action Reference	All inputs, outputs, and exit codes
Examples	RAG, agents, multi-provider, Slack notifications
Troubleshooting	Common errors and fixes

---

30-second setup

# .github/workflows/eval.yml
- uses: SynapseKit/evalci@v1
  with:
    path: tests/evals
    threshold: "0.80"
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

# tests/evals/eval_rag.py
from synapsekit import eval_case

@eval_case(min_score=0.80)
async def eval_my_pipeline():
    result = await pipeline.ask("What is SynapseKit?")
    return {"score": result.relevancy_score}

---

Get help

• Open an issue
• Start a discussion
• Full docs