Skip to content

promptfoo.md

Latest commit

 

History

History
39 lines (26 loc) · 1.97 KB

promptfoo.md

File metadata and controls

39 lines (26 loc) · 1.97 KB
Raw

Works with Promptfoo

What Promptfoo does well

  • Fast prompt and policy regression loops.
  • Good matrix testing across prompts, models, and variables.
  • Established workflow for red-team and guardrail assertions.

Tradeoffs

  • Output JSON shape can vary by plugin/version.
  • Promptfoo focuses on test execution; cross-run governance and publish workflows are separate concerns.

Minimal conversion path into eval-report/v1

Use the built-in import adapter.

  • Required: promptfoo JSON/JSONL export (results[] rows, results.outputs[] rows, or newline-delimited result rows).
  • Output: one eval-report/v1 artifact under .evals_output/.

Concrete command/pattern

eval-dashboards import --from=promptfoo --input=./promptfoo-results.json --out=.evals_output/import-promptfoo.json
eval-dashboards check --input=.evals_output
eval-dashboards report --input=.evals_output --reporter=html --report-dir=eval-report

The importer preserves key Promptfoo evidence fields when present:

  • gradingResult.reason / gradingResult.comment -> rows[].reason
  • latencyMs (or response.latencyMs) -> rows[].durationMs
  • metadata.sessionId -> rows[].metadata.sourceSessionId
  • missing row ids -> deterministic promptfoo-<sha1> ids derived from suite + case payload (#2 suffix for duplicates in one file), so row identity does not depend on array position
  • provider/eval error rows (error, response.error, providerResponse.error, or failureReason: 2) -> imported as failed rows with the numeric row.score or gradingResult.score preserved, defaulting to score: 0 only when neither is numeric; available error reasons are preserved

Related risks: integration risk register

Generated base IDs are independent of array position. Duplicate suffixes depend on encounter order and can change in partial exports that omit earlier matching cases. Explicit IDs are preserved unchanged; generated IDs avoid all explicit IDs in the file, including suffixed IDs.