diff --git a/docs/project-worklog.md b/docs/project-worklog.md index 2caac5d..aebd441 100644 --- a/docs/project-worklog.md +++ b/docs/project-worklog.md @@ -479,24 +479,67 @@ Deferred: - Full interactive Docker demo remains pending until Docker is available. - Real model weights remain deferred until a GPU server is prepared. -## Next Recommended Work +### 2026-07-09: Evaluation Result Insights -Continue Phase B from `docs/project-improvement-roadmap.md` with a narrower -verification and insight branch: +Branch: ```text feature/evaluation-result-insights ``` +Commit: + +```text +See the PR commit history after merge. +``` + +What changed: + +- Added tested frontend insight calculation for evaluation samples. +- Added a confusion matrix, label distribution, and sample-outcome summary to + `/admin/evaluations`. +- Improved the wrong-sample empty state for completed matching predictions. + +Why: + +- The evaluation page needed to explain model behavior, not only list runs and + aggregate metrics. +- The change keeps insight logic in a pure utility so future backend aggregation + can be compared against frontend expectations. + +Verification: + +- `npm run test` +- `npm run lint` +- `npm run build` +- `mvn -B test` + +Deferred: + +- No backend aggregation API was added in this branch. +- Real model weights remain deferred until a GPU server is prepared. +- Local headless screenshot verification was attempted with a mocked API, but + the temporary Vite process exited before Chrome could capture the page. + +## Next Recommended Work + +Continue Phase B from `docs/project-improvement-roadmap.md` with a backend +observability branch: + +```text +feature/evaluation-observability +``` + Scope: -- Add a small confusion matrix or label breakdown to `/admin/evaluations`. -- Add clearer empty states for runs with no wrong samples. +- Add structured logs around evaluation execution start, retry, completion, and + failure. +- Add lightweight timing fields or counters that make run latency explainable. +- Document how to inspect evaluation execution behavior locally and in CI. - Keep the deterministic model boundary until GPU weights are available. -- Preserve the current frontend visual style. Reason: -The project now has evaluation execution and a usable admin workbench. The next -interview-visible step is to make model quality easier to explain without +The project now has evaluation execution and frontend result explainability. The +next interview-visible step is to show production-minded observability without expanding scope into training, video detection, or heavy model operations. diff --git a/docs/superpowers/plans/2026-07-09-evaluation-result-insights.md b/docs/superpowers/plans/2026-07-09-evaluation-result-insights.md new file mode 100644 index 0000000..467418f --- /dev/null +++ b/docs/superpowers/plans/2026-07-09-evaluation-result-insights.md @@ -0,0 +1,117 @@ +# Evaluation Result Insights Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add lightweight evaluation explainability to `/admin/evaluations` without adding backend APIs or model weights. + +**Architecture:** Compute insight summaries in a tested pure TypeScript utility, then render the result in the existing React evaluation page. Keep the UI within the current dark admin style and avoid chart dependencies. + +**Tech Stack:** React 18, TypeScript, Vite, CSS Modules, Node built-in test runner. + +## Global Constraints + +- Preserve the current frontend visual style. +- Do not download model weights. +- Do not add a chart library. +- Do not add backend aggregation APIs in this branch. +- Keep counting logic outside JSX. +- Verify with `npm run test`, `npm run lint`, `npm run build`, `mvn -B test`, and `git diff --check`. + +--- + +### Task 1: Insight Utility Tests + +**Files:** +- Modify: `package.json` +- Create: `src/pages/AdminEvaluations/evaluationInsights.test.ts` + +**Interfaces:** +- Consumes future `buildEvaluationInsights(samples: EvaluationSampleResponse[]): EvaluationInsights`. +- Produces expected behavior for matrix counts, summary counts, and label distribution. + +- [ ] Update `package.json` test script to include frontend test files: + +```json +"test": "node --test --experimental-strip-types src/api/errorMessage.test.ts src/pages/AdminEvaluations/evaluationInsights.test.ts" +``` + +- [ ] Add a failing test for a mixed sample set: + +```typescript +const insights = buildEvaluationInsights([ + sample('a.jpg', 'AUTHENTIC', 'AUTHENTIC', true), + sample('b.jpg', 'AUTHENTIC', 'SYNTHETIC', false), + sample('c.jpg', 'SYNTHETIC', 'SYNTHETIC', true), + sample('d.jpg', 'UNCERTAIN', null, null, 'model unavailable'), + sample('e.jpg', 'SYNTHETIC', null, null), +]); +``` + +Expected summary: + +```text +total=5, correct=2, wrong=1, failed=1, pending=1 +``` + +- [ ] Run `npm run test`. + +Expected: FAIL because `evaluationInsights.ts` does not exist yet. + +### Task 2: Insight Utility Implementation + +**Files:** +- Create: `src/pages/AdminEvaluations/evaluationInsights.ts` + +**Interfaces:** +- Produces `LABELS`, `EvaluationInsightSummary`, `ConfusionMatrixRow`, `LabelDistributionRow`, `EvaluationInsights`, and `buildEvaluationInsights`. + +- [ ] Implement fixed labels `AUTHENTIC`, `SYNTHETIC`, `UNCERTAIN`. +- [ ] Count only samples with a `predictedLabel` in the confusion matrix. +- [ ] Count `failureReason` samples as failed. +- [ ] Count samples without `predictedLabel` and without `failureReason` as pending. +- [ ] Calculate label-distribution percentages as `count / total`, using `0` when total is `0`. +- [ ] Run `npm run test`. + +Expected: PASS. + +### Task 3: Evaluation Page Rendering + +**Files:** +- Modify: `src/pages/AdminEvaluations/index.tsx` +- Modify: `src/pages/AdminEvaluations/AdminEvaluations.module.css` + +**Interfaces:** +- Consumes `buildEvaluationInsights(detail?.samples ?? [])`. +- Renders summary cards, confusion matrix, and label distribution. + +- [ ] Import and memoize `buildEvaluationInsights`. +- [ ] Add a summary strip below metric cards. +- [ ] Add a confusion matrix with truth rows and prediction columns. +- [ ] Add a label distribution panel. +- [ ] Improve wrong-sample empty copy: + +```text +All completed predictions match the manifest labels. +``` + +- [ ] Keep responsive behavior for narrow screens. +- [ ] Run `npm run build`. + +Expected: PASS. + +### Task 4: Worklog And Verification + +**Files:** +- Modify: `docs/project-worklog.md` + +**Interfaces:** +- Records the branch, scope, verification, and deferred work. + +- [ ] Add a dated worklog entry for `feature/evaluation-result-insights`. +- [ ] Run `npm run test`. +- [ ] Run `npm run lint`. +- [ ] Run `npm run build`. +- [ ] Run `mvn -B test` in `backend-java`. +- [ ] Run `git diff --check`. +- [ ] Commit as `feat: add evaluation result insights`. +- [ ] Push and open a PR against `main`. diff --git a/docs/superpowers/specs/2026-07-09-evaluation-result-insights-design.md b/docs/superpowers/specs/2026-07-09-evaluation-result-insights-design.md new file mode 100644 index 0000000..c74c605 --- /dev/null +++ b/docs/superpowers/specs/2026-07-09-evaluation-result-insights-design.md @@ -0,0 +1,92 @@ +# Evaluation Result Insights Design + +## Purpose + +Improve `/admin/evaluations` so an interviewer can understand model behavior +from one run without reading raw sample rows. The page should explain where the +model is correct, where it confuses labels, and whether failures are prediction +errors or execution errors. + +## Scope + +This branch adds frontend-only insight rendering based on the existing +`EvaluationDetailResponse.samples` payload. + +In scope: + +- Confusion matrix for `AUTHENTIC`, `SYNTHETIC`, and `UNCERTAIN`. +- Ground-truth label distribution. +- Summary counts for total, correct, wrong, pending, and failed samples. +- Clearer empty state when a selected run has no wrong or failed samples. +- Documentation of the work in `docs/project-worklog.md`. + +Out of scope: + +- New backend aggregation APIs. +- Model weight download or GPU runtime integration. +- Charts from a new visualization dependency. +- Video, training, or dataset management workflows. + +## Architecture + +Add a focused pure utility module under `src/pages/AdminEvaluations`: + +```text +EvaluationDetailResponse.samples + -> buildEvaluationInsights(samples) + -> AdminEvaluations insight panels +``` + +The utility owns all counting logic and is tested with Node's built-in test +runner. The React page only renders the computed view model. This keeps business +math out of JSX and makes later backend/API changes easier to compare against +frontend expectations. + +## UI Behavior + +The existing visual style stays intact: + +- Use the current dark surface, border, mono labels, and compact panel rhythm. +- Render the insight section inside the selected-run detail area. +- Use text tables and small numeric cards rather than a chart library. +- Keep labels English to match the current admin page copy. + +The insight section contains: + +- A compact summary strip. +- A confusion matrix with rows as ground truth and columns as prediction. +- A label-distribution list showing sample counts and percentages. +- A wrong-sample empty state that distinguishes "no selected run" from "all + completed predictions matched". + +## Error Handling + +Samples with `failureReason` count as failed samples. Samples with no +`predictedLabel` and no `failureReason` count as pending samples. Failed and +pending samples are excluded from the confusion matrix because no final +prediction exists. + +## Testing + +Use the existing frontend `npm run test` command and add coverage for: + +- Correct confusion-matrix counting. +- Failed and pending sample classification. +- Label distribution percentages. +- Empty sample handling. + +Run the full project checks before PR: + +```powershell +npm run test +npm run lint +npm run build +cd backend-java +mvn -B test +``` + +## Approval + +The user requested continuing with the next planned branch. This design follows +the previously recommended `feature/evaluation-result-insights` scope and keeps +the project focused on interview-visible evaluation explainability. diff --git a/package.json b/package.json index 07b1d28..f4b7a2d 100644 --- a/package.json +++ b/package.json @@ -7,7 +7,7 @@ "dev": "vite", "build": "tsc -b && vite build", "lint": "eslint .", - "test": "node --test --experimental-strip-types src/api/errorMessage.test.ts", + "test": "node --test --experimental-strip-types src/api/errorMessage.test.ts src/pages/AdminEvaluations/evaluationInsights.test.ts", "preview": "vite preview" }, "dependencies": { diff --git a/src/pages/AdminEvaluations/AdminEvaluations.module.css b/src/pages/AdminEvaluations/AdminEvaluations.module.css index 594926a..59e8c7e 100644 --- a/src/pages/AdminEvaluations/AdminEvaluations.module.css +++ b/src/pages/AdminEvaluations/AdminEvaluations.module.css @@ -5,6 +5,7 @@ .shell { display: grid; + align-items: start; grid-template-columns: minmax(0, 1fr) 340px; gap: var(--space-6); } @@ -204,6 +205,127 @@ font-weight: var(--weight-medium); } +.insightGrid { + display: grid; + grid-template-columns: minmax(0, 1.25fr) minmax(260px, 0.75fr); + gap: var(--space-3); + margin-bottom: var(--space-5); +} + +.insightPanel, +.matrixPanel { + border: 1px solid var(--rule); + border-radius: var(--radius); + background: var(--canvas); + padding: 18px; +} + +.matrixPanel { + margin-bottom: var(--space-5); +} + +.insightHeader { + margin-bottom: var(--space-3); +} + +.insightHeader h3 { + margin-top: 3px; + color: var(--ink); + font-size: var(--text-lg); + font-weight: var(--weight-medium); +} + +.insightNumbers { + display: grid; + grid-template-columns: repeat(5, minmax(0, 1fr)); + gap: var(--space-2); +} + +.insightNumber { + min-width: 0; + border: 1px solid var(--rule); + border-radius: var(--radius); + padding: 12px; +} + +.insightNumber span, +.distributionRow span, +.distributionRow em, +.matrixHead, +.matrixRow strong { + color: var(--ink-3); + font-family: var(--font-mono); + font-size: var(--text-xs); +} + +.insightNumber strong { + display: block; + margin-top: 6px; + color: var(--ink); + font-family: var(--font-serif); + font-size: var(--text-xl); + font-weight: var(--weight-medium); +} + +.distributionList { + display: grid; + gap: var(--space-2); +} + +.distributionRow { + display: grid; + grid-template-columns: minmax(0, 1fr) 44px 56px; + align-items: center; + gap: var(--space-2); + min-height: 38px; + border-bottom: 1px solid var(--rule); +} + +.distributionRow:last-child { + border-bottom: 0; +} + +.distributionRow strong { + color: var(--ink); + font-family: var(--font-serif); + font-size: var(--text-lg); + font-weight: var(--weight-medium); +} + +.matrix { + display: grid; + gap: var(--space-2); + overflow-x: auto; +} + +.matrixHead, +.matrixRow { + display: grid; + grid-template-columns: minmax(150px, 1fr) repeat(3, minmax(92px, 0.42fr)); + align-items: center; + gap: var(--space-2); + min-width: 520px; +} + +.matrixHead { + padding-bottom: 8px; + border-bottom: 1px solid var(--rule); +} + +.matrixRow { + min-height: 42px; + border: 1px solid var(--rule); + border-radius: var(--radius); + padding: 10px 12px; +} + +.matrixRow span { + color: var(--ink); + font-family: var(--font-serif); + font-size: var(--text-xl); + font-weight: var(--weight-medium); +} + .empty, .error { border: 1px dashed var(--rule); @@ -226,5 +348,12 @@ .metrics { grid-template-columns: repeat(2, minmax(0, 1fr)); } -} + .insightGrid { + grid-template-columns: 1fr; + } + + .insightNumbers { + grid-template-columns: repeat(2, minmax(0, 1fr)); + } +} diff --git a/src/pages/AdminEvaluations/evaluationInsights.test.ts b/src/pages/AdminEvaluations/evaluationInsights.test.ts new file mode 100644 index 0000000..fc18761 --- /dev/null +++ b/src/pages/AdminEvaluations/evaluationInsights.test.ts @@ -0,0 +1,81 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import type { EvaluationSampleResponse, ModelLabel } from '@/api/backend'; +import { buildEvaluationInsights } from './evaluationInsights.ts'; + +function sample( + filename: string, + groundTruthLabel: ModelLabel, + predictedLabel: ModelLabel | null, + correct: boolean | null, + failureReason: string | null = null, +): EvaluationSampleResponse { + return { + sampleId: filename, + evaluationId: 'eval-1', + filename, + groundTruthLabel, + predictedLabel, + score: predictedLabel ? 0.8 : null, + latencyMs: predictedLabel ? 12 : null, + correct, + failureReason, + createdAt: '2026-07-09T00:00:00Z', + }; +} + +test('builds summary counts and confusion matrix from mixed samples', () => { + const insights = buildEvaluationInsights([ + sample('a.jpg', 'AUTHENTIC', 'AUTHENTIC', true), + sample('b.jpg', 'AUTHENTIC', 'SYNTHETIC', false), + sample('c.jpg', 'SYNTHETIC', 'SYNTHETIC', true), + sample('d.jpg', 'UNCERTAIN', null, null, 'model unavailable'), + sample('e.jpg', 'SYNTHETIC', null, null), + ]); + + assert.deepEqual(insights.summary, { + total: 5, + correct: 2, + wrong: 1, + failed: 1, + pending: 1, + completed: 3, + }); + assert.equal(insights.matrix.AUTHENTIC.AUTHENTIC, 1); + assert.equal(insights.matrix.AUTHENTIC.SYNTHETIC, 1); + assert.equal(insights.matrix.SYNTHETIC.SYNTHETIC, 1); + assert.equal(insights.matrix.UNCERTAIN.UNCERTAIN, 0); +}); + +test('builds label distribution percentages from ground-truth labels', () => { + const insights = buildEvaluationInsights([ + sample('a.jpg', 'AUTHENTIC', 'AUTHENTIC', true), + sample('b.jpg', 'AUTHENTIC', 'SYNTHETIC', false), + sample('c.jpg', 'SYNTHETIC', 'SYNTHETIC', true), + sample('d.jpg', 'UNCERTAIN', null, null, 'model unavailable'), + ]); + + assert.deepEqual( + insights.labelDistribution.map((row) => [row.label, row.count, row.ratio]), + [ + ['AUTHENTIC', 2, 0.5], + ['SYNTHETIC', 1, 0.25], + ['UNCERTAIN', 1, 0.25], + ], + ); +}); + +test('returns zeroed insights for empty samples', () => { + const insights = buildEvaluationInsights([]); + + assert.deepEqual(insights.summary, { + total: 0, + correct: 0, + wrong: 0, + failed: 0, + pending: 0, + completed: 0, + }); + assert.ok(insights.labelDistribution.every((row) => row.count === 0 && row.ratio === 0)); +}); diff --git a/src/pages/AdminEvaluations/evaluationInsights.ts b/src/pages/AdminEvaluations/evaluationInsights.ts new file mode 100644 index 0000000..0a08e30 --- /dev/null +++ b/src/pages/AdminEvaluations/evaluationInsights.ts @@ -0,0 +1,91 @@ +import type { EvaluationSampleResponse, ModelLabel } from '@/api/backend'; + +export const LABELS = ['AUTHENTIC', 'SYNTHETIC', 'UNCERTAIN'] as const satisfies readonly ModelLabel[]; + +export type ConfusionMatrix = Record>; + +export interface EvaluationInsightSummary { + total: number; + correct: number; + wrong: number; + failed: number; + pending: number; + completed: number; +} + +export interface LabelDistributionRow { + label: ModelLabel; + count: number; + ratio: number; +} + +export interface EvaluationInsights { + summary: EvaluationInsightSummary; + matrix: ConfusionMatrix; + labelDistribution: LabelDistributionRow[]; +} + +function createMatrix(): ConfusionMatrix { + return LABELS.reduce((matrix, truthLabel) => { + matrix[truthLabel] = LABELS.reduce( + (row, predictedLabel) => { + row[predictedLabel] = 0; + return row; + }, + {} as Record, + ); + return matrix; + }, {} as ConfusionMatrix); +} + +export function buildEvaluationInsights(samples: EvaluationSampleResponse[]): EvaluationInsights { + const matrix = createMatrix(); + const distributionCounts = LABELS.reduce( + (counts, label) => { + counts[label] = 0; + return counts; + }, + {} as Record, + ); + const summary: EvaluationInsightSummary = { + total: samples.length, + correct: 0, + wrong: 0, + failed: 0, + pending: 0, + completed: 0, + }; + + for (const sample of samples) { + distributionCounts[sample.groundTruthLabel] += 1; + + if (sample.failureReason) { + summary.failed += 1; + continue; + } + + if (!sample.predictedLabel) { + summary.pending += 1; + continue; + } + + summary.completed += 1; + matrix[sample.groundTruthLabel][sample.predictedLabel] += 1; + + if (sample.correct) { + summary.correct += 1; + } else { + summary.wrong += 1; + } + } + + return { + summary, + matrix, + labelDistribution: LABELS.map((label) => ({ + label, + count: distributionCounts[label], + ratio: summary.total === 0 ? 0 : distributionCounts[label] / summary.total, + })), + }; +} diff --git a/src/pages/AdminEvaluations/index.tsx b/src/pages/AdminEvaluations/index.tsx index 7bc1652..e7f3b1e 100644 --- a/src/pages/AdminEvaluations/index.tsx +++ b/src/pages/AdminEvaluations/index.tsx @@ -10,6 +10,7 @@ import { type EvaluationRunResponse, } from '@/api/backend'; import { formatDate, formatPercent, labelText, statusTone } from '@/pages/adminFormat'; +import { buildEvaluationInsights, LABELS } from './evaluationInsights'; import styles from './AdminEvaluations.module.css'; const sampleManifest = `filename,groundTruthLabel @@ -25,6 +26,15 @@ function Metric({ label, value }: { label: string; value?: number | null }) { ); } +function InsightNumber({ label, value }: { label: string; value: number }) { + return ( +
+ {label} + {value} +
+ ); +} + export function AdminEvaluations() { const [runs, setRuns] = useState([]); const [selectedId, setSelectedId] = useState(null); @@ -40,6 +50,7 @@ export function AdminEvaluations() { const selectedRun = useMemo(() => runs.find((run) => run.evaluationId === selectedId) ?? runs[0] ?? null, [runs, selectedId]); const wrongSamples = detail?.samples.filter((sample) => sample.correct === false || sample.failureReason) ?? []; + const insights = useMemo(() => buildEvaluationInsights(detail?.samples ?? []), [detail?.samples]); const refresh = async (nextSelectedId = selectedId) => { setLoading(true); @@ -201,6 +212,59 @@ export function AdminEvaluations() { +
+
+
+ Quality Split +

Sample Outcome

+
+
+ + + + + +
+
+ +
+
+ Manifest +

Label Distribution

+
+
+ {insights.labelDistribution.map((row) => ( +
+ {labelText(row.label)} + {row.count} + {formatPercent(row.ratio)} +
+ ))} +
+
+
+ +
+
+ Confusion Matrix +

Truth By Prediction

+
+
+
+ Truth \\ Prediction + {LABELS.map((label) => {labelText(label)})} +
+ {LABELS.map((truthLabel) => ( +
+ {labelText(truthLabel)} + {LABELS.map((predictedLabel) => ( + {insights.matrix[truthLabel][predictedLabel]} + ))} +
+ ))} +
+
+
Filename @@ -221,7 +285,11 @@ export function AdminEvaluations() { {sample.latencyMs === null ? 'N/A' : `${sample.latencyMs}ms`}
))} - {!wrongSamples.length ?

No wrong or failed samples for the selected run.

: null} + {!wrongSamples.length ? ( +

+ {detail ? 'All completed predictions match the manifest labels.' : 'Create or select an evaluation to inspect wrong samples.'} +

+ ) : null}