From 186375caa16b57599fc717550b99ba12ae2805fc Mon Sep 17 00:00:00 2001 From: fengting124 Date: Sat, 11 Jul 2026 14:14:11 +0800 Subject: [PATCH 1/2] docs: define real evaluation execution boundary --- docs/README.md | 3 + ...-07-11-evaluation-real-execution-design.md | 502 ++++++++++++++++++ 2 files changed, 505 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-11-evaluation-real-execution-design.md diff --git a/docs/README.md b/docs/README.md index a9044ec..b1a264e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -46,6 +46,9 @@ Use these to understand system boundaries. - [AI Application Product Boundary And Long-Term Roadmap](superpowers/specs/2026-07-11-ai-application-product-roadmap-design.md): authoritative product scope, review workflow, long-term phases, and portfolio acceptance criteria. +- [Real Evaluation Execution Boundary](superpowers/specs/2026-07-11-evaluation-real-execution-design.md): + asset-backed datasets, durable batch execution, fenced recovery, result + provenance, and Phase 2 acceptance criteria. - [Model Integration Framework](model-integration-framework.md): Java-to-Python model-service contract, model runtime configuration, and weight handling. - [Async Detection Jobs](async-detection-jobs.md): Redis-backed asynchronous diff --git a/docs/superpowers/specs/2026-07-11-evaluation-real-execution-design.md b/docs/superpowers/specs/2026-07-11-evaluation-real-execution-design.md new file mode 100644 index 0000000..b8ee1f7 --- /dev/null +++ b/docs/superpowers/specs/2026-07-11-evaluation-real-execution-design.md @@ -0,0 +1,502 @@ +# Real Evaluation Execution Boundary + +- Status: Approved design +- Owners: Project maintainers +- Last reviewed: 2026-07-11 +- Target branch: `feature/evaluation-real-execution-boundary` + +## 1. Purpose + +This specification turns the existing manifest-based evaluation demo into a +real, recoverable execution path. It defines how uploaded image assets are +assembled into an evaluation dataset, dispatched through Redis, evaluated by +the configured model service, and persisted with enough provenance to compare +results later. + +The implementation deliberately does not download model weights. Development +may use the explicitly identified heuristic runtime, but it must use the same +Java orchestration, HTTP model contract, persistence, retry, and observability +boundaries that a GPU runtime will use. + +This is the second delivery phase in +`docs/superpowers/specs/2026-07-11-ai-application-product-roadmap-design.md`. + +## 2. Current Problem + +The existing evaluation module is useful scaffolding but is not a real model +evaluation system: + +- its CSV manifest identifies samples only by display filename; +- filenames are not resolved to uploaded `media_asset` records; +- `DeterministicEvaluationModelClient` derives a score from the model ID and + filename rather than image content; +- the controller executes the complete batch inside an HTTP request; +- model calls and result writes occur inside one database transaction; +- retry skips samples with predictions, but has no lease or fencing token; +- model version, threshold, runtime identity, and dataset digest are absent; +- partial results cannot be distinguished from a complete evaluation. + +The result is a deterministic demonstration, not evidence that the model +service evaluated the declared dataset. + +## 3. Decisions + +### 3.1 Dataset identity + +An evaluation manifest references images that have already crossed the secure +upload boundary: + +```csv +assetId,groundTruthLabel +asset_abc123,AUTHENTIC +asset_def456,SYNTHETIC +``` + +The service resolves each `assetId` through `MediaAssetRepository`. It never +accepts a client-provided filesystem path. The original filename is copied as +display metadata and is never used to build a storage path. + +The canonical manifest is produced from validated rows in their submitted +order, with normalized line endings and enum labels. Its SHA-256 digest and +schema version are stored on the evaluation run. This makes dataset identity +stable without introducing a separate dataset aggregate in this phase. + +Creation constraints: + +- schema version is `1`; +- required columns are exactly `assetId` and `groundTruthLabel`; +- labels are `AUTHENTIC` or `SYNTHETIC`; +- every asset must exist before the run is created; +- duplicate asset IDs are rejected; +- the manifest contains between 1 and 500 sample rows; +- predictions, scores, latency, and paths are never accepted from the client. + +### 3.2 Execution granularity + +One durable message represents one evaluation run. A worker claims the run and +processes unresolved samples sequentially. Each sample result is persisted in +a short transaction immediately after inference. + +This provides crash recovery and partial progress without introducing the +coordination cost of one queue message per sample. Sample-level parallelism is +deferred until measured GPU throughput shows that it is needed. + +### 3.3 Source of truth + +PostgreSQL is the source of business truth. Redis Streams transports wake-up +messages and may deliver the same message more than once. Correctness must not +depend on exactly-once Redis delivery. + +### 3.4 Model boundary + +Evaluation reuses the production `ModelInferenceClient` contract. The worker +resolves a validated asset storage path and a snapshot of model endpoint and +threshold, then invokes the model outside a database transaction. + +The deterministic filename client is removed from the formal evaluation path. +The local heuristic model service remains permitted only when its response and +stored provenance clearly identify a development runtime. + +## 4. Alternatives Considered + +### 4.1 Run-level message with sample checkpoints + +Selected. It has bounded infrastructure complexity, preserves progress after a +crash, and demonstrates reliable orchestration without making queue topology +the product. + +### 4.2 One message per sample + +Rejected for this phase. It improves horizontal parallelism but requires fan +out, completion coordination, queue backpressure, event cardinality controls, +and more complex cancellation semantics. Those costs are not justified for a +500-sample local evaluation. + +### 4.3 Synchronous HTTP execution + +Rejected. It couples batch duration to request timeouts, retains long database +transactions, and makes process restart recovery ambiguous. + +### 4.4 Server-local dataset paths or ZIP upload + +Rejected. Arbitrary paths violate the upload trust boundary. ZIP ingestion +adds archive traversal, expansion limits, cleanup, and storage lifecycle work +that is independent of reliable evaluation execution. + +## 5. Architecture + +```text +EvaluationController + -> EvaluationApplicationService + -> validate model and assets + -> save run and samples + -> save EVALUATION_REQUESTED outbox event + +OutboxPublisher + -> Redis Stream + -> EvaluationJobConsumer + -> EvaluationExecutionService + -> EvaluationExecutionTransactionService.claim() + -> ModelInferenceClient.predict() outside transaction + -> EvaluationExecutionTransactionService.completeSample() + -> renewLease() when required + -> finalizeRun() +``` + +Responsibilities are separated as follows: + +- `EvaluationApplicationService`: command validation, creation, retry request, + and read models. +- `EvaluationManifestParser`: structured parsing, schema validation, + canonicalization, row limits, and digest calculation. +- `EvaluationExecutionService`: non-transactional orchestration and model + invocation. +- `EvaluationExecutionTransactionService`: locked claims, leases, fenced + sample writes, retries, and finalization in short transactions. +- `EvaluationJobConsumer`: Redis acknowledgement policy and duplicate-safe job + handoff. +- `EvaluationMetricsCalculator`: deterministic metrics over completed sample + results only. + +No component other than the transaction service owns execution-state +transitions. + +## 6. Data Model + +### 6.1 `evaluation_run` + +Add or formalize these fields: + +| Field | Purpose | +| --- | --- | +| `manifest_sha256` | Digest of the canonical validated manifest | +| `manifest_schema_version` | Manifest contract version, initially `1` | +| `model_endpoint_url` | Endpoint snapshot used by this run | +| `threshold` | Threshold snapshot used by this run | +| `runtime_id` | Runtime identity reported by the model service | +| `model_version` | Actual model version reported during execution | +| `failed_samples` | Terminal failed sample count | +| `execution_token` | Fencing token owned by the active worker | +| `lease_expires_at` | Time after which another worker may claim the run | +| `version` | JPA optimistic-lock version | + +Endpoint values are operationally sensitive and are not returned by public +read APIs. The endpoint snapshot prevents registry edits from changing a run +that is already queued. + +### 6.2 `evaluation_sample` + +Add or formalize these fields: + +| Field | Purpose | +| --- | --- | +| `asset_id` | Existing uploaded asset used for inference | +| `display_filename` | Immutable display metadata copied from the asset | +| `status` | `PENDING`, `COMPLETED`, or `FAILED` | +| `ground_truth_label` | Validated binary reference label | +| `predicted_label` | Model label for a completed sample | +| `raw_score` | Score before client normalization | +| `normalized_score` | Comparable score in the formal client contract | +| `threshold` | Threshold used for this prediction | +| `latency_ms` | Model-service latency | +| `model_version` | Version reported for this prediction | +| `runtime_id` | Runtime reported for this prediction | +| `raw_response_json` | Bounded response provenance | +| `attempt_count` | Number of inference attempts for the sample | +| `failure_code` | Stable machine-readable failure classification | +| `failure_reason` | Bounded safe diagnostic message | +| `completed_at` | Terminal result timestamp | +| `execution_token` | Run attempt that produced the terminal write | + +The database enforces uniqueness of `(evaluation_id, asset_id)` and a foreign +key from `evaluation_sample.asset_id` to `media_asset.asset_id` with deletion +restricted. Application validation still produces a stable API error before a +database constraint can fail. + +### 6.3 Migration compatibility + +Existing evaluation rows came from a filename-only demo and cannot prove asset +identity. The migration must preserve them for history but mark their schema as +legacy and non-executable. New execution endpoints reject legacy runs with the +stable error code `LEGACY_MANIFEST_UNRESOLVED`. + +No migration invents asset IDs or silently matches assets by filename. + +## 7. State Machines + +### 7.1 Run state + +```text +QUEUED -> RUNNING -> COMPLETED + -> PARTIALLY_COMPLETED + -> FAILED +FAILED -> QUEUED (explicit retry while attempts remain) +PARTIALLY_COMPLETED -> QUEUED (explicit retry of retryable failed samples) +``` + +Definitions: + +- `COMPLETED`: every sample completed successfully. +- `PARTIALLY_COMPLETED`: all samples are terminal and at least one failed, but + at least one valid prediction exists. +- `FAILED`: the run cannot produce reliable results, or no sample completed. +- `QUEUED`: a durable execution request exists or is ready to be published. +- `RUNNING`: a worker owns a non-expired lease. + +Changing dataset, model, or threshold requires a new evaluation run. Retry +never changes the identity of the original experiment. + +### 7.2 Sample state + +```text +PENDING -> COMPLETED +PENDING -> FAILED +FAILED -> PENDING (explicit retry when failure is retryable) +``` + +A `COMPLETED` sample is immutable within its evaluation run. Automatic retries +are internal attempts while the sample remains under active execution; an +exhausted sample becomes `FAILED`. + +## 8. Execution Flow + +### 8.1 Create + +Within one transaction: + +1. Parse and validate the manifest. +2. Load the selected enabled model. +3. Resolve all asset IDs and reject missing or duplicate assets. +4. Canonicalize the manifest and calculate its digest. +5. Snapshot model ID, endpoint, and threshold. +6. Save the queued run and pending samples. +7. Save an `EVALUATION_REQUESTED` outbox event. + +The API returns `202 Accepted` after the transaction commits. + +### 8.2 Claim + +The worker locks the run row in a short transaction: + +- terminal runs return `TERMINAL`; +- a run with a live lease returns `BUSY`; +- an eligible run receives a new unpredictable execution token and lease; +- the claim returns an immutable execution plan containing only required data. + +Default lease duration is five minutes and is configurable. The worker renews +the lease before half of the remaining duration is consumed. + +### 8.3 Process sample + +For each unresolved sample: + +1. Resolve its trusted `MediaAsset.storagePath`. +2. Call `ModelInferenceClient.predict()` outside a transaction. +3. Classify any exception as transient or permanent. +4. Retry transient failures with bounded exponential backoff and jitter. +5. Persist a completed or exhausted result in a short fenced transaction. + +The fenced transaction checks that the run still owns the supplied token. A +stale worker returns `STALE` and stops processing immediately. + +### 8.4 Finalize + +After all samples are terminal, a locked short transaction: + +- counts completed and failed samples; +- calculates metrics over completed samples; +- calculates coverage over all declared samples; +- validates that model runtime provenance is internally consistent; +- clears the execution lease and token; +- sets `COMPLETED`, `PARTIALLY_COMPLETED`, or `FAILED`. + +Finalization is idempotent. + +## 9. Retry And Failure Policy + +### 9.1 Failure classes + +| Class | Examples | Policy | +| --- | --- | --- | +| Transient model error | timeout, connection reset, HTTP 429 or 5xx | Up to 3 attempts with backoff | +| Permanent sample error | unsupported image, corrupt input, validation rejection | Fail sample without automatic retry | +| Contract error | malformed JSON, missing required model fields | Fail sample and expose stable code | +| Configuration error | model missing, disabled, or invalid endpoint snapshot | Fail run | +| Infrastructure error | database or Redis unavailable | Do not acknowledge message; recover by redelivery | +| Stale ownership | token mismatch after lease takeover | Reject write and stop old worker | + +Retry delays are configurable and deterministic under tests. Production uses +jitter to avoid synchronized retry bursts. + +### 9.2 Safe diagnostics + +Errors persist a stable code and a message capped at 2,048 characters. Messages +must not contain stack traces, authorization headers, model endpoint secrets, +or absolute storage paths. Full exceptions remain in protected structured +logs. + +## 10. Metrics + +The run stores or returns: + +- accuracy; +- precision; +- recall; +- F1; +- true-positive, false-positive, true-negative, and false-negative counts; +- coverage: `completedSamples / totalSamples`; +- completed and failed sample counts. + +Classification metrics are calculated only from completed samples. Coverage is +always returned beside them so incomplete data cannot appear to be a complete +benchmark. A run with zero completed samples has no classification metrics. + +## 11. API Contract + +### 11.1 Commands + +- `POST /api/evaluations`: validate, create, and queue a run; return `202` with + the created run resource. +- `POST /api/evaluations/{evaluationId}/retry`: queue retryable failed samples; + return `202`. + +The existing synchronous `/run` endpoint remains available for one documented +compatibility cycle, delegates to asynchronous dispatch, and is marked +deprecated. It must not execute inference in the request thread. + +### 11.2 Queries + +- `GET /api/evaluations`: paged summary list. +- `GET /api/evaluations/{evaluationId}`: run status, progress, metrics, + coverage, manifest digest, model provenance, and safe failure details. +- `GET /api/evaluations/{evaluationId}/samples`: paged samples with optional + `status` and `correct` filters. + +Public responses never expose model endpoint URLs or storage paths. + +### 11.3 Frontend behavior + +The existing visual style remains unchanged. The evaluation workbench: + +- creates manifests with asset IDs rather than filenames; +- treats command responses as accepted work, not completed results; +- polls with bounded intervals while a run is queued or running; +- displays completed, failed, total, and coverage independently; +- distinguishes partial completion from success; +- identifies `development/heuristic` runtime results visibly; +- paginates samples and preserves stable error-code explanations. + +No evidence, heatmap, or confidence explanation is synthesized by the UI. + +## 12. Observability + +Structured execution logs include: + +- `evaluationId`; +- `sampleId`; +- `assetId`; +- `modelId`; +- `executionToken`; +- attempt number; +- failure code. + +Micrometer metrics include: + +- evaluation runs started and completed by status; +- sample inference latency; +- completed and failed sample counts; +- model retries; +- lease takeovers; +- stale write rejections; +- outbox age and Redis stream backlog. + +OpenTelemetry boundaries are reserved around API command handling, outbox +publication, Redis consumption, and model HTTP calls. Actuator health details +must not reveal endpoints, paths, or exception stacks. + +## 13. Verification Strategy + +### 13.1 Unit tests + +- manifest schema, row bounds, duplicates, canonicalization, and digest; +- run and sample state transitions; +- failure classification and bounded backoff; +- confusion matrix, classification metrics, and coverage; +- safe failure-message truncation. + +### 13.2 Service tests + +- missing assets and disabled models are rejected atomically; +- endpoint and threshold snapshots do not change after registry edits; +- completed samples remain immutable; +- retries select only eligible failed or pending samples; +- finalization is idempotent. + +### 13.3 Concurrency and integration tests + +With PostgreSQL and Redis Testcontainers when Docker is available: + +- duplicate stream messages produce one active owner; +- a live lease prevents a second claim; +- an expired lease permits takeover; +- a stale token cannot persist a sample or finalize a run; +- outbox publication recovers after Redis unavailability; +- Flyway migration works from the current schema. + +### 13.4 Model contract tests + +WireMock covers successful inference, timeout, HTTP 429, HTTP 5xx, permanent +client error, malformed JSON, and missing provenance fields. + +### 13.5 Frontend tests + +Cover queued, running, completed, partially completed, and failed views; +polling cleanup; development-runtime disclosure; pagination; and API errors. + +Existing Java, model-service, smoke, lint, and frontend suites remain required +regression gates. + +## 14. Delivery Slices + +Implementation remains on one feature branch but is committed in reviewable +slices: + +1. migration and domain state machines; +2. versioned manifest parser and asset validation; +3. outbox event and asynchronous API commands; +4. leased, fenced execution transactions; +5. real model-client adapter and retry classification; +6. metrics, provenance, and paged query contract; +7. Redis consumer and recovery behavior; +8. frontend contract adaptation; +9. observability, documentation, and full verification. + +Each slice adds tests before implementation and leaves the branch buildable. + +## 15. Explicit Non-Goals + +This phase does not: + +- download or benchmark production model weights; +- claim SOTA accuracy; +- accept arbitrary paths or ZIP datasets; +- implement model training or hyperparameter search; +- implement sample-level distributed fan-out; +- add Kafka, a workflow engine, or new microservices; +- fabricate evidence, heatmaps, explanations, or benchmark conclusions; +- present heuristic-runtime output as production model performance. + +## 16. Acceptance Criteria + +The phase is complete when: + +1. every executable sample resolves to a trusted uploaded asset; +2. run creation and dispatch are durable across Redis interruption; +3. model HTTP calls occur outside database transactions; +4. sample progress survives Java process interruption; +5. duplicate delivery and stale workers cannot overwrite valid results; +6. retry resumes only eligible work; +7. metrics include coverage and reproducible model/dataset provenance; +8. development runtime identity is visible and truthful; +9. CPU/local and future GPU deployments share the same Java contract; +10. automated tests demonstrate the success and recovery paths. From 15b93979530e4739c6397c9eb6895bec1964fdd8 Mon Sep 17 00:00:00 2001 From: fengting124 Date: Sat, 11 Jul 2026 14:26:42 +0800 Subject: [PATCH 2/2] docs: plan real evaluation execution --- docs/README.md | 1 + .../2026-07-11-evaluation-real-execution.md | 826 ++++++++++++++++++ 2 files changed, 827 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-11-evaluation-real-execution.md diff --git a/docs/README.md b/docs/README.md index b1a264e..6b69250 100644 --- a/docs/README.md +++ b/docs/README.md @@ -61,6 +61,7 @@ These documents are useful for understanding how the project evolved. - [AIGC Forensics Platform Plan](superpowers/plans/2026-07-07-aigc-forensics-platform.md) - [Frontend API Integration Plan](superpowers/plans/2026-07-07-frontend-api-integration.md) - [Real Nonescape Runtime Plan](superpowers/plans/2026-07-08-real-nonescape-runtime.md) +- [Real Evaluation Execution Plan](superpowers/plans/2026-07-11-evaluation-real-execution.md) - [Platform Design Spec](superpowers/specs/2026-07-07-image-authenticity-platform-design.md) - [Production Foundation Design](superpowers/specs/2026-07-11-production-foundation-design.md) diff --git a/docs/superpowers/plans/2026-07-11-evaluation-real-execution.md b/docs/superpowers/plans/2026-07-11-evaluation-real-execution.md new file mode 100644 index 0000000..0458882 --- /dev/null +++ b/docs/superpowers/plans/2026-07-11-evaluation-real-execution.md @@ -0,0 +1,826 @@ +# Real Evaluation Execution Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace filename-derived evaluation results with durable, asset-backed, asynchronous model execution that survives duplicate delivery and worker restart. + +**Architecture:** PostgreSQL remains the source of truth. Evaluation creation stores validated asset-backed samples and an outbox event atomically; Redis wakes a worker that claims a leased run, invokes the shared model HTTP client outside transactions, checkpoints each sample with a fencing token, and finalizes reproducible metrics. The frontend submits work asynchronously and polls truthful progress. + +**Tech Stack:** Java 21, Spring Boot, Spring Data JPA, Flyway, PostgreSQL/H2 tests, Redis Streams, Jackson CSV, Micrometer, JUnit 5, AssertJ, Mockito, FastAPI/Pydantic, React 19, TypeScript, Node test runner. + +## Global Constraints + +- Executable manifests use schema version `1` and exactly `assetId,groundTruthLabel` columns. +- Ground-truth labels are only `AUTHENTIC` and `SYNTHETIC`. +- One run accepts 1 through 500 unique uploaded assets. +- Client-provided filesystem paths, predictions, scores, and latency are rejected. +- PostgreSQL is authoritative; Redis delivery is at least once. +- Model HTTP calls never run inside a database transaction. +- The run lease defaults to five minutes and all writes are fenced by an execution token. +- A completed sample is immutable within its run. +- Transient model failures receive at most three attempts with bounded backoff; permanent failures do not retry. +- Public APIs never return model endpoint URLs or storage paths. +- Heuristic execution is identified as `heuristic`; it is never described as production model quality. +- Do not download model weights or require Docker in this plan. Record Testcontainers cases for the later Docker-enabled verification phase. +- Preserve the existing frontend visual language and component conventions. + +--- + +## File Map + +### Evaluation domain and persistence + +- Create `backend-java/src/main/resources/db/migration/V7__add_real_evaluation_execution.sql`: legacy-safe schema migration, asset foreign key, leases, provenance, counters, and indexes. +- Modify `evaluation/domain/EvaluationStatus.java`: add `PARTIALLY_COMPLETED`. +- Create `evaluation/domain/EvaluationSampleStatus.java`: sample terminal-state vocabulary. +- Modify `evaluation/domain/EvaluationRun.java`: immutable experiment snapshot and leased state machine. +- Modify `evaluation/domain/EvaluationSample.java`: asset identity, result provenance, attempt state, and immutable completion. +- Modify both evaluation repositories: locked run lookup, paged sample lookup, and unresolved sample selection. + +### Manifest and application commands + +- Create `evaluation/service/EvaluationManifestParser.java`: RFC-style CSV parsing, validation, canonicalization, and SHA-256. +- Create `evaluation/service/ParsedEvaluationManifest.java` and `EvaluationManifestRow.java`: typed parser output. +- Modify `evaluation/service/EvaluationService.java`: create and query real runs; remove client-supplied predictions. +- Modify evaluation DTOs: asynchronous command and paged query contract. + +### Dispatch and execution + +- Extend `domain/JobOutboxEventType.java`, `domain/JobOutboxEvent.java`, `service/JobOutboxService.java`, and `service/JobOutboxPublisher.java`: `EVALUATION_REQUESTED` support. +- Create `evaluation/job/*`: typed Redis queue request/message, Redis adapter, consumer, and worker. +- Create `evaluation/service/EvaluationExecutionTransactionService.java`: claim, renew, checkpoint, fail, and finalize short transactions. +- Rewrite `evaluation/service/EvaluationExecutionService.java`: transaction-free orchestration and shared model-client calls. +- Remove `evaluation/client/DeterministicEvaluationModelClient.java` and its private evaluation-client contract. + +### Model contract, API, UI, and operations + +- Modify Java `ModelInferenceResult` and HTTP client: include `runtimeId`. +- Modify Python prediction schema and endpoint: return runtime identity from health metadata. +- Modify evaluation controller and frontend API/types/workbench: `202`, polling, coverage, provenance, and partial completion. +- Add evaluation Micrometer metrics and structured logging. +- Update runbook, capability matrix, worklog, and API documentation. + +--- + +### Task 1: Persist The Real Evaluation State Model + +**Files:** +- Create: `backend-java/src/main/resources/db/migration/V7__add_real_evaluation_execution.sql` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/domain/EvaluationSampleStatus.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/domain/EvaluationStatus.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/domain/EvaluationRun.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/domain/EvaluationSample.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/repository/EvaluationRunRepository.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/repository/EvaluationSampleRepository.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/domain/EvaluationRunTest.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/domain/EvaluationSampleTest.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/repository/EvaluationRepositoryTest.java` + +**Interfaces:** +- Produces: `EvaluationRun.claimExecution(String, Instant, Instant)`, `renewLease(String, Instant)`, `ownsExecution(String)`, `finalizeExecution(...)`. +- Produces: `EvaluationSample.complete(String, ModelInferenceResult, double, Instant)` and `fail(String, EvaluationFailure, Instant)`. +- Produces: `findByEvaluationIdForUpdate(String)` and paged/unresolved repository queries. + +- [ ] **Step 1: Write failing state-machine tests** + +```java +@Test +void expiredLeaseCanBeReclaimedAndOldTokenIsRejected() { + EvaluationRun run = queuedRun(); + assertThat(run.claimExecution("token-1", NOW, NOW.plusSeconds(30))).isTrue(); + assertThat(run.claimExecution("token-2", NOW.plusSeconds(31), NOW.plusSeconds(61))).isTrue(); + assertThat(run.ownsExecution("token-1")).isFalse(); + assertThat(run.ownsExecution("token-2")).isTrue(); +} + +@Test +void completedSampleCannotBeOverwritten() { + EvaluationSample sample = pendingSample(); + sample.complete("token-1", inference("heuristic", 0.91), 0.5, NOW); + assertThatThrownBy(() -> sample.complete("token-2", inference("onnx", 0.20), 0.5, NOW)) + .isInstanceOf(IllegalStateException.class); +} +``` + +- [ ] **Step 2: Run the focused tests and confirm RED** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationRunTest,EvaluationSampleTest test` + +Expected: compilation fails because the new state methods and enum do not exist. + +- [ ] **Step 3: Add V7 and implement the domain transitions** + +The migration must keep legacy rows nullable while making all new writes explicit: + +```sql +alter table evaluation_run add column manifest_sha256 varchar(64); +alter table evaluation_run add column manifest_schema_version integer not null default 0; +alter table evaluation_run add column model_endpoint_url varchar(1024); +alter table evaluation_run add column threshold double precision; +alter table evaluation_run add column runtime_id varchar(128); +alter table evaluation_run add column model_version varchar(128); +alter table evaluation_run add column failed_samples integer not null default 0; +alter table evaluation_run add column coverage double precision; +alter table evaluation_run add column true_positive integer; +alter table evaluation_run add column false_positive integer; +alter table evaluation_run add column true_negative integer; +alter table evaluation_run add column false_negative integer; +alter table evaluation_run add column execution_token varchar(64); +alter table evaluation_run add column lease_expires_at timestamp with time zone; +alter table evaluation_run add column version bigint not null default 0; + +alter table evaluation_sample rename column filename to display_filename; +alter table evaluation_sample add column asset_id varchar(64); +alter table evaluation_sample add column status varchar(32) not null default 'PENDING'; +alter table evaluation_sample add column raw_score double precision; +alter table evaluation_sample rename column score to normalized_score; +alter table evaluation_sample add column threshold double precision; +alter table evaluation_sample add column model_version varchar(128); +alter table evaluation_sample add column runtime_id varchar(128); +alter table evaluation_sample add column raw_response_json text; +alter table evaluation_sample add column attempt_count integer not null default 0; +alter table evaluation_sample add column failure_code varchar(64); +alter table evaluation_sample add column completed_at timestamp with time zone; +alter table evaluation_sample add column execution_token varchar(64); +alter table evaluation_sample add constraint fk_evaluation_sample_asset + foreign key (asset_id) references media_asset(asset_id) on delete restrict; +create unique index uq_evaluation_sample_run_asset + on evaluation_sample(evaluation_id, asset_id) where asset_id is not null; +create index idx_evaluation_sample_run_status + on evaluation_sample(evaluation_id, status, created_at); +``` + +Implement strict transition methods; reject blank tokens and non-positive lease windows. Map old rows to schema version `0`, which later services treat as non-executable. + +- [ ] **Step 4: Run domain and repository tests** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationRunTest,EvaluationSampleTest,EvaluationRepositoryTest test` + +Expected: PASS, including Flyway/JPA validation and the asset foreign-key test. + +- [ ] **Step 5: Commit** + +```bash +git add backend-java/src/main/resources/db/migration/V7__add_real_evaluation_execution.sql backend-java/src/main/java/com/fengting/aigcforensics/evaluation backend-java/src/test/java/com/fengting/aigcforensics/evaluation +git commit -m "feat: add durable evaluation state model" +``` + +### Task 2: Parse Versioned Asset Manifests + +**Files:** +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationManifestRow.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/ParsedEvaluationManifest.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationManifestParser.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/service/EvaluationManifestParserTest.java` +- Modify: `backend-java/pom.xml` + +**Interfaces:** +- Produces: `ParsedEvaluationManifest parse(String manifest)`. +- Produces: `record EvaluationManifestRow(String assetId, ModelLabel groundTruthLabel)`. +- Produces: `record ParsedEvaluationManifest(int schemaVersion, String sha256, String canonicalCsv, List rows)`. + +- [ ] **Step 1: Add failing parser tests** + +```java +@Test +void canonicalizesQuotedCsvAndCalculatesStableDigest() { + ParsedEvaluationManifest first = parser.parse("assetId,groundTruthLabel\r\nasset_a,AUTHENTIC\r\n"); + ParsedEvaluationManifest second = parser.parse("assetId,groundTruthLabel\nasset_a,AUTHENTIC\n"); + assertThat(first.schemaVersion()).isEqualTo(1); + assertThat(first.canonicalCsv()).isEqualTo("assetId,groundTruthLabel\nasset_a,AUTHENTIC\n"); + assertThat(first.sha256()).isEqualTo(second.sha256()); +} + +@Test +void rejectsDuplicateAssetsAndClientPredictionColumns() { + assertThatThrownBy(() -> parser.parse("assetId,groundTruthLabel,predictedLabel\nasset_a,AUTHENTIC,SYNTHETIC")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("exactly assetId,groundTruthLabel"); +} +``` + +- [ ] **Step 2: Run parser tests and confirm RED** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationManifestParserTest test` + +Expected: compilation fails because the parser types do not exist. + +- [ ] **Step 3: Implement with Apache Commons CSV** + +Add `org.apache.commons:commons-csv:1.12.0` and parse with explicit headers, duplicate-header rejection, trimming, and bounded records. Build canonical CSV from typed values, then calculate lowercase SHA-256 with `HexFormat`. + +```java +public ParsedEvaluationManifest parse(String manifest) { + List rows = parseRows(manifest); + requireSize(rows, 1, 500); + requireUniqueAssets(rows); + String canonical = canonicalize(rows); + return new ParsedEvaluationManifest(1, sha256(canonical), canonical, List.copyOf(rows)); +} +``` + +- [ ] **Step 4: Run parser tests** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationManifestParserTest test` + +Expected: PASS for CRLF normalization, quoted values, malformed rows, illegal labels, duplicate assets, and 500/501 boundaries. + +- [ ] **Step 5: Commit** + +```bash +git add backend-java/pom.xml backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service backend-java/src/test/java/com/fengting/aigcforensics/evaluation/service/EvaluationManifestParserTest.java +git commit -m "feat: validate asset-backed evaluation manifests" +``` + +### Task 3: Create Runs And Outbox Events Atomically + +**Files:** +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationService.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/dto/CreateEvaluationRequest.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/domain/JobOutboxEventType.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/domain/JobOutboxEvent.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/service/JobOutboxService.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/service/EvaluationServiceTest.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/service/JobOutboxServiceTest.java` + +**Interfaces:** +- Consumes: `EvaluationManifestParser.parse(String)` from Task 2. +- Produces: `EvaluationRunResponse createEvaluation(CreateEvaluationRequest)` with a queued run. +- Produces: `JobOutboxEvent scheduleEvaluation(String evaluationId)` and `replayEvaluation(String evaluationId)`. + +- [ ] **Step 1: Write atomic-creation tests** + +```java +@Test +void createsAssetBackedRunWithModelSnapshotAndOutboxEvent() { + EvaluationRunResponse response = service.createEvaluation(request("asset_a", "AUTHENTIC")); + assertThat(response.status()).isEqualTo(EvaluationStatus.QUEUED); + assertThat(runRepository.findByEvaluationId(response.evaluationId()).orElseThrow().getManifestSchemaVersion()).isEqualTo(1); + assertThat(outboxRepository.findByEventTypeAndAggregateId(EVALUATION_REQUESTED, response.evaluationId())).isPresent(); +} + +@Test +void missingAssetRollsBackRunSamplesAndOutbox() { + assertThatThrownBy(() -> service.createEvaluation(request("asset_missing", "SYNTHETIC"))) + .isInstanceOf(ResourceNotFoundException.class); + assertThat(runRepository.count()).isZero(); + assertThat(outboxRepository.count()).isZero(); +} +``` + +- [ ] **Step 2: Run focused tests and confirm RED** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationServiceTest,JobOutboxServiceTest test` + +Expected: FAIL because evaluation outbox support and asset resolution are absent. + +- [ ] **Step 3: Implement atomic creation and generic outbox factory** + +Load the enabled model, resolve all assets in one repository query, preserve manifest order with a map, snapshot endpoint/threshold, save pending samples, and schedule the event in the same transaction. + +```java +@Transactional +public EvaluationRunResponse createEvaluation(CreateEvaluationRequest request) { + ParsedEvaluationManifest manifest = manifestParser.parse(request.manifest()); + ModelRegistry model = requireEnabledModel(request.modelId()); + Map assets = requireAssets(manifest.rows()); + EvaluationRun run = runRepository.save(newRun(request, manifest, model)); + sampleRepository.saveAll(toSamples(run, manifest.rows(), assets)); + outboxService.scheduleEvaluation(run.getEvaluationId()); + return mapper.toRunResponse(run); +} +``` + +- [ ] **Step 4: Run service and outbox tests** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationServiceTest,JobOutboxServiceTest test` + +Expected: PASS, including disabled model, missing asset, duplicate event, and transaction rollback cases. + +- [ ] **Step 5: Commit** + +```bash +git add backend-java/src/main/java/com/fengting/aigcforensics/evaluation backend-java/src/main/java/com/fengting/aigcforensics/domain/JobOutboxEvent* backend-java/src/main/java/com/fengting/aigcforensics/service/JobOutboxService.java backend-java/src/test/java +git commit -m "feat: queue evaluation runs transactionally" +``` + +### Task 4: Add Leased And Fenced Sample Execution + +**Files:** +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionClaimStatus.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionClaim.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionOutcome.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionPlan.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationSamplePlan.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationFailure.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionTransactionService.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionTransactionServiceTest.java` +- Modify: `backend-java/src/main/resources/application.yml` +- Modify: `backend-java/src/test/resources/application-test.yml` + +**Interfaces:** +- Produces: `claim(String evaluationId)`, `renewLease(String evaluationId, String token)`, `completeSample(...)`, `failSample(...)`, and `finalizeRun(...)`. +- Produces: claim statuses `CLAIMED`, `BUSY`, `TERMINAL`, `FAILED`, and write outcome `COMPLETED`, `STALE`. +- Produces: `EvaluationExecutionOutcome` values `COMPLETED`, `PARTIALLY_COMPLETED`, `FAILED`, `BUSY`, `TERMINAL`, and `STALE`. +- Produces: `record EvaluationExecutionClaim(EvaluationExecutionClaimStatus status, EvaluationExecutionPlan plan)` with `isClaimed()` and `toOutcome()`. +- Produces: `record EvaluationFailure(String code, String safeReason, boolean retryable)`. + +- [ ] **Step 1: Write failing transaction-boundary tests** + +```java +@Test +void staleOwnerCannotCompleteSampleAfterLeaseTakeover() { + EvaluationExecutionClaim first = service.claim(EVALUATION_ID); + clock.advance(Duration.ofMinutes(6)); + EvaluationExecutionClaim second = service.claim(EVALUATION_ID); + assertThat(service.completeSample(EVALUATION_ID, SAMPLE_ID, first.plan().executionToken(), result())) + .isEqualTo(EvaluationExecutionOutcome.STALE); + assertThat(service.completeSample(EVALUATION_ID, SAMPLE_ID, second.plan().executionToken(), result())) + .isEqualTo(EvaluationExecutionOutcome.COMPLETED); +} +``` + +- [ ] **Step 2: Run the test and confirm RED** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationExecutionTransactionServiceTest test` + +Expected: compilation fails because transaction service contracts do not exist. + +- [ ] **Step 3: Implement short locked transactions** + +Use `findByEvaluationIdForUpdate`, a configurable `app.evaluation.execution.lease-duration: 5m`, and `Clock`. A claim returns trusted `Path` values from `MediaAsset`; completion/failure re-locks the run and verifies the token before touching a sample. Finalization calculates counts and metrics from terminal rows and is idempotent. + +```java +@Transactional +public EvaluationExecutionClaim claim(String evaluationId) { + EvaluationRun run = findRunForUpdate(evaluationId); + return claimEligibleRun(run, Instant.now(clock)); +} + +@Transactional +public EvaluationExecutionOutcome completeSample( + String evaluationId, + String sampleId, + String executionToken, + ModelInferenceResult result) { + EvaluationRun run = findRunForUpdate(evaluationId); + if (!run.ownsExecution(executionToken)) return EvaluationExecutionOutcome.STALE; + findSample(evaluationId, sampleId).complete(executionToken, result, run.getThreshold(), Instant.now(clock)); + return EvaluationExecutionOutcome.COMPLETED; +} + +@Transactional +public EvaluationExecutionOutcome failSample( + String evaluationId, + String sampleId, + String executionToken, + EvaluationFailure failure) { + EvaluationRun run = findRunForUpdate(evaluationId); + if (!run.ownsExecution(executionToken)) return EvaluationExecutionOutcome.STALE; + findSample(evaluationId, sampleId).fail(executionToken, failure, Instant.now(clock)); + return EvaluationExecutionOutcome.FAILED; +} +``` + +- [ ] **Step 4: Run transaction tests** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationExecutionTransactionServiceTest,EvaluationMetricsCalculatorTest test` + +Expected: PASS for live lease, takeover, stale write, immutable completion, partial completion, zero-success failure, and idempotent finalization. + +- [ ] **Step 5: Commit** + +```bash +git add backend-java/src/main/java/com/fengting/aigcforensics/evaluation backend-java/src/main/resources/application.yml backend-java/src/test +git commit -m "feat: fence evaluation execution with leases" +``` + +### Task 5: Invoke The Shared Model Contract With Bounded Retry + +**Files:** +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/client/ModelInferenceResult.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/client/HttpModelInferenceClient.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/client/ModelInferenceException.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationFailureClassifier.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationRetryPolicy.java` +- Rewrite: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionService.java` +- Delete: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/client/*` +- Modify: `model-services/nonescape-mini/app/schemas.py` +- Modify: `model-services/nonescape-mini/app/main.py` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionServiceTest.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/client/HttpModelInferenceClientTest.java` +- Test: `model-services/nonescape-mini/tests/test_api.py` + +**Interfaces:** +- Produces: `ModelInferenceResult(String runtimeId, String modelVersion, double rawScore, double normalizedScore, ModelLabel label, int latencyMs, String rawResponseJson)`. +- Produces: `EvaluationFailure classify(RuntimeException)` with code, safe reason, and retryable flag. + +- [ ] **Step 1: Write failing orchestration and contract tests** + +```java +@Test +void retriesTransientFailureThenCheckpointsSuccessfulResult() { + when(client.predict(anyString(), any())).thenThrow(new ModelInferenceException("timeout", 504)).thenReturn(result()); + assertThat(service.runEvaluation(EVALUATION_ID)).isEqualTo(EvaluationExecutionOutcome.COMPLETED); + verify(client, times(2)).predict(anyString(), any()); + verify(transactionService).completeSample(eq(EVALUATION_ID), eq(SAMPLE_ID), anyString(), eq(result())); +} + +@Test +void permanentClientFailureDoesNotRetry() { + when(client.predict(anyString(), any())).thenThrow(new ModelInferenceException("unsupported image", 400)); + service.runEvaluation(EVALUATION_ID); + verify(client).predict(anyString(), any()); +} +``` + +Python response assertion: + +```python +assert response.json()["runtimeId"] == "heuristic" +``` + +- [ ] **Step 2: Run focused Java and Python tests and confirm RED** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationExecutionServiceTest,HttpModelInferenceClientTest test` + +Run: `python -m pytest model-services/nonescape-mini/tests/test_api.py -q` + +Expected: Java compilation fails and Python assertion fails because `runtimeId` and status-aware failures are absent. + +- [ ] **Step 3: Implement orchestration outside transactions** + +`EvaluationExecutionService` must have no `@Transactional` annotation. It claims once, invokes the client using each trusted plan, renews leases between samples, stops on `STALE`, and finalizes after all sample plans reach a terminal state. + +```java +public EvaluationExecutionOutcome runEvaluation(String evaluationId) { + EvaluationExecutionClaim claim = transactions.claim(evaluationId); + if (!claim.isClaimed()) return claim.toOutcome(); + for (EvaluationSamplePlan sample : claim.plan().samples()) { + EvaluationExecutionOutcome outcome = executeSample(claim.plan(), sample); + if (outcome == EvaluationExecutionOutcome.STALE) return outcome; + transactions.renewLease(evaluationId, claim.plan().executionToken()); + } + return transactions.finalizeRun(evaluationId, claim.plan().executionToken()); +} +``` + +The Python endpoint obtains `runtime.health().runtime` and returns it as `runtimeId`. Extend `ModelInferenceException` with an optional HTTP status so 429/5xx/timeouts are retryable and 4xx/contract errors are permanent. + +- [ ] **Step 4: Run Java and Python tests** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationExecutionServiceTest,HttpModelInferenceClientTest test` + +Run: `python -m pytest model-services/nonescape-mini/tests -q` + +Expected: PASS with explicit heuristic runtime identity and bounded retry behavior. + +- [ ] **Step 5: Commit** + +```bash +git add backend-java/src model-services/nonescape-mini +git commit -m "feat: execute evaluation samples through model service" +``` + +### Task 6: Deliver Evaluation Jobs Through Redis + +**Files:** +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/config/EvaluationJobRedisProperties.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/job/EvaluationJobRequest.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/job/EvaluationJobMessage.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/job/EvaluationJobQueue.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/job/EvaluationJobConsumer.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/job/RedisEvaluationJobQueue.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/job/EvaluationJobWorker.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/service/JobOutboxPublisher.java` +- Modify: `backend-java/src/main/resources/application.yml` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/job/RedisEvaluationJobQueueTest.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/job/EvaluationJobWorkerTest.java` +- Modify: `backend-java/src/test/java/com/fengting/aigcforensics/service/JobOutboxPublisherTest.java` + +**Interfaces:** +- Produces: `enqueue(EvaluationJobRequest)`, `poll()`, and `acknowledge(EvaluationJobMessage)`. +- Consumes: `EvaluationExecutionService.runEvaluation(String)` from Task 5. + +- [ ] **Step 1: Write failing publisher, queue, and worker tests** + +```java +@Test +void acknowledgesTerminalAndCompletedOutcomesButLeavesBusyPending() { + when(consumer.poll()).thenReturn(Optional.of(message)); + when(execution.runEvaluation(EVALUATION_ID)).thenReturn(EvaluationExecutionOutcome.BUSY); + worker.pollOnce(); + verify(consumer, never()).acknowledge(any()); +} + +@Test +void duplicateEventIdIsEnqueuedOnce() { + queue.enqueue(request); + queue.enqueue(request); + assertThat(streamEntries()).hasSize(1); +} +``` + +- [ ] **Step 2: Run focused tests and confirm RED** + +Run: `mvn -f backend-java/pom.xml -Dtest=JobOutboxPublisherTest,RedisEvaluationJobQueueTest,EvaluationJobWorkerTest test` + +Expected: compilation fails because evaluation Redis contracts do not exist. + +- [ ] **Step 3: Implement the evaluation stream adapter** + +Use keys `evaluation:jobs`, group `evaluation-workers`, dead letter stream `evaluation:jobs:dead-letter`, and submitted prefix `evaluation:jobs:submitted:`. Preserve event version, occurrence time, pending claim, dead-letter, and deduplication behavior from the proven detection queue. The worker acknowledges `COMPLETED`, `PARTIALLY_COMPLETED`, `FAILED`, `TERMINAL`, and `STALE`; it leaves `BUSY` pending for later claim. + +```java +@Scheduled(fixedDelayString = "${app.evaluation.jobs.poll-delay-ms:1000}") +public void pollOnce() { + consumer.poll().ifPresent(message -> { + EvaluationExecutionOutcome outcome = executionService.runEvaluation(message.evaluationId()); + if (outcome != EvaluationExecutionOutcome.BUSY) { + consumer.acknowledge(message); + } + }); +} +``` + +```yaml +app: + evaluation: + jobs: + worker-enabled: ${APP_EVALUATION_JOBS_WORKER_ENABLED:true} + poll-delay-ms: ${APP_EVALUATION_JOBS_POLL_DELAY_MS:1000} + redis: + stream-key: ${APP_EVALUATION_JOBS_REDIS_STREAM_KEY:evaluation:jobs} + group-name: ${APP_EVALUATION_JOBS_REDIS_GROUP_NAME:evaluation-workers} + dead-letter-stream-key: ${APP_EVALUATION_JOBS_REDIS_DEAD_LETTER_STREAM_KEY:evaluation:jobs:dead-letter} +``` + +- [ ] **Step 4: Run queue tests and existing detection queue regression tests** + +Run: `mvn -f backend-java/pom.xml -Dtest=JobOutboxPublisherTest,RedisEvaluationJobQueueTest,EvaluationJobWorkerTest,RedisDetectionJobQueueTest,DetectionJobWorkerTest test` + +Expected: PASS with both job types independently routed. + +- [ ] **Step 5: Commit** + +```bash +git add backend-java/src/main/java/com/fengting/aigcforensics/config backend-java/src/main/java/com/fengting/aigcforensics/evaluation/job backend-java/src/main/java/com/fengting/aigcforensics/service/JobOutboxPublisher.java backend-java/src/main/resources backend-java/src/test +git commit -m "feat: dispatch evaluation jobs through redis" +``` + +### Task 7: Expose Asynchronous And Paged Evaluation APIs + +**Files:** +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/controller/EvaluationController.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationService.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/dto/EvaluationRunResponse.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/dto/EvaluationDetailResponse.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/dto/EvaluationSampleResponse.java` +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/dto/EvaluationPageResponse.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/controller/EvaluationControllerTest.java` + +**Interfaces:** +- Produces: `202 Accepted` create/retry commands. +- Produces: paged list and sample resources with `page`, `size`, `totalElements`, and `totalPages`. + +- [ ] **Step 1: Write failing MockMvc contract tests** + +```java +mockMvc.perform(post("/api/evaluations").contentType(APPLICATION_JSON).content(validRequest)) + .andExpect(status().isAccepted()) + .andExpect(jsonPath("$.status").value("QUEUED")); + +mockMvc.perform(get("/api/evaluations/{id}/samples", EVALUATION_ID) + .param("status", "FAILED").param("page", "0").param("size", "25")) + .andExpect(status().isOk()) + .andExpect(jsonPath("$.content").isArray()) + .andExpect(jsonPath("$.size").value(25)); +``` + +- [ ] **Step 2: Run controller tests and confirm RED** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationControllerTest test` + +Expected: FAIL because create returns 201 and query endpoints return unpaged arrays. + +- [ ] **Step 3: Implement bounded paging and compatibility dispatch** + +Default page size is 25, maximum 100. `/run` remains for one compatibility cycle, is annotated deprecated, and delegates to the same asynchronous replay command as `/retry`; neither endpoint calls model inference in the request thread. + +```java +@PostMapping +@ResponseStatus(HttpStatus.ACCEPTED) +public EvaluationRunResponse createEvaluation(@Valid @RequestBody CreateEvaluationRequest request) { + return evaluationService.createEvaluation(request); +} + +@PostMapping({"/{evaluationId}/retry", "/{evaluationId}/run"}) +@ResponseStatus(HttpStatus.ACCEPTED) +public EvaluationRunResponse retryEvaluation(@PathVariable String evaluationId) { + return evaluationService.retryEvaluation(evaluationId); +} + +@GetMapping("/{evaluationId}/samples") +public EvaluationPageResponse listSamples( + @PathVariable String evaluationId, + @RequestParam(required = false) EvaluationSampleStatus status, + @RequestParam(required = false) Boolean correct, + @RequestParam(defaultValue = "0") int page, + @RequestParam(defaultValue = "25") int size) { + return evaluationService.listSamples(evaluationId, status, correct, page, Math.min(size, 100)); +} +``` + +- [ ] **Step 4: Run controller and service tests** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationControllerTest,EvaluationServiceTest test` + +Expected: PASS for accepted commands, safe fields, paging bounds, filters, legacy-run rejection, and retry exhaustion. + +- [ ] **Step 5: Commit** + +```bash +git add backend-java/src/main/java/com/fengting/aigcforensics/evaluation backend-java/src/test/java/com/fengting/aigcforensics/evaluation +git commit -m "feat: expose asynchronous evaluation api" +``` + +### Task 8: Adapt The Evaluation Workbench + +**Files:** +- Modify: `src/api/backend.ts` +- Modify: `src/pages/AdminEvaluations/index.tsx` +- Modify: `src/pages/AdminEvaluations/index.module.css` +- Modify: `src/pages/AdminEvaluations/evaluationInsights.ts` +- Modify: `src/pages/AdminEvaluations/evaluationInsights.test.ts` +- Create: `src/pages/AdminEvaluations/evaluationPolling.ts` +- Create: `src/pages/AdminEvaluations/evaluationPolling.test.ts` +- Modify: `package.json` + +**Interfaces:** +- Consumes: asynchronous and paged API from Task 7. +- Produces: `shouldPollEvaluation(status)` and `nextEvaluationPollDelay(attempt)` pure helpers. + +- [ ] **Step 1: Write failing frontend state tests** + +```typescript +test('polls only non-terminal runs with a bounded delay', () => { + assert.equal(shouldPollEvaluation('QUEUED'), true); + assert.equal(shouldPollEvaluation('RUNNING'), true); + assert.equal(shouldPollEvaluation('PARTIALLY_COMPLETED'), false); + assert.equal(nextEvaluationPollDelay(20), 10_000); +}); + +test('does not treat partial coverage as complete benchmark evidence', () => { + const insights = buildEvaluationInsights(samples, { totalSamples: 10, completedSamples: 8 }); + assert.equal(insights.coverage, 0.8); + assert.equal(insights.isComplete, false); +}); +``` + +- [ ] **Step 2: Run frontend tests and confirm RED** + +Run: `npm test` + +Expected: compilation fails because polling helpers and coverage input do not exist. + +- [ ] **Step 3: Implement the asynchronous UI contract** + +Change the default manifest to asset IDs, remove the synchronous run assumption, poll only queued/running runs, cancel timers on selection/unmount, display progress and failed count separately, paginate samples, and render a visible `Development heuristic runtime` disclosure when `runtimeId === 'heuristic'`. Keep existing typography, colors, spacing, and panel treatment. + +```typescript +export function shouldPollEvaluation(status: EvaluationStatus): boolean { + return status === 'QUEUED' || status === 'RUNNING'; +} + +export function nextEvaluationPollDelay(attempt: number): number { + return Math.min(2_000 * 2 ** Math.min(attempt, 3), 10_000); +} + +useEffect(() => { + if (!detail || !shouldPollEvaluation(detail.status)) return undefined; + const timer = window.setTimeout( + () => void refresh(detail.evaluationId), + nextEvaluationPollDelay(pollAttempt), + ); + return () => window.clearTimeout(timer); +}, [detail?.evaluationId, detail?.status, pollAttempt]); +``` + +- [ ] **Step 4: Run frontend verification** + +Run: `npm test` + +Run: `npm run lint` + +Run: `npm run build` + +Expected: all tests pass, ESLint exits 0, and Vite production build exits 0. + +- [ ] **Step 5: Commit** + +```bash +git add src/api/backend.ts src/pages/AdminEvaluations package.json +git commit -m "feat: show truthful asynchronous evaluation progress" +``` + +### Task 9: Add Operational Evidence And Documentation + +**Files:** +- Create: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionMetrics.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionService.java` +- Modify: `backend-java/src/main/java/com/fengting/aigcforensics/evaluation/job/EvaluationJobWorker.java` +- Test: `backend-java/src/test/java/com/fengting/aigcforensics/evaluation/service/EvaluationExecutionMetricsTest.java` +- Modify: `docs/fullstack-evaluation-demo.md` +- Create: `docs/evaluation-execution.md` +- Modify: `docs/capability-matrix.md` +- Modify: `docs/project-worklog.md` +- Modify: `docs/README.md` + +**Interfaces:** +- Produces Micrometer counters/timers prefixed `evaluation.execution`. +- Produces the operator runbook and verification record. + +- [ ] **Step 1: Write failing metrics tests** + +```java +@Test +void recordsRetryLeaseTakeoverAndStaleWriteSignals() { + SimpleMeterRegistry registry = new SimpleMeterRegistry(); + EvaluationExecutionMetrics metrics = new EvaluationExecutionMetrics(registry); + metrics.recordRetry("timeout"); + metrics.recordLeaseTakeover(); + metrics.recordStaleWrite(); + assertThat(registry.get("evaluation.execution.retries").tag("code", "timeout").counter().count()).isEqualTo(1); + assertThat(registry.get("evaluation.execution.lease.takeovers").counter().count()).isEqualTo(1); + assertThat(registry.get("evaluation.execution.stale.writes").counter().count()).isEqualTo(1); +} +``` + +- [ ] **Step 2: Run metrics tests and confirm RED** + +Run: `mvn -f backend-java/pom.xml -Dtest=EvaluationExecutionMetricsTest test` + +Expected: compilation fails because the metrics facade does not exist. + +- [ ] **Step 3: Implement bounded telemetry and runbooks** + +Use low-cardinality metric tags only; IDs belong in structured logs, never metric tags. Document stream keys, lease inspection, replay, stable failure codes, heuristic disclosure, legacy manifests, and future PostgreSQL/Redis Testcontainers cases. Update the capability matrix only for behavior proven by tests in this branch. + +```java +public final class EvaluationExecutionMetrics { + private final MeterRegistry registry; + + public EvaluationExecutionMetrics(MeterRegistry registry) { + this.registry = registry; + } + + public void recordRetry(String code) { + registry.counter("evaluation.execution.retries", "code", code).increment(); + } + + public void recordLeaseTakeover() { + registry.counter("evaluation.execution.lease.takeovers").increment(); + } + + public void recordStaleWrite() { + registry.counter("evaluation.execution.stale.writes").increment(); + } +} +``` + +- [ ] **Step 4: Run the complete verification matrix** + +Run: `mvn -f backend-java/pom.xml test` + +Run: `python -m pytest model-services/nonescape-mini/tests -q` + +Run: `npm test` + +Run: `npm run lint` + +Run: `npm run build` + +Run: `git diff --check origin/main...HEAD` + +Expected: Maven reports zero failures/errors, Pytest reports all passing, Node reports all passing, lint/build exit 0, and Git reports no whitespace errors. + +- [ ] **Step 5: Commit** + +```bash +git add backend-java/src docs +git commit -m "docs: operationalize real evaluation execution" +``` + +## Deferred Docker Verification + +When Docker is available, create a separate `test/evaluation-testcontainers` +branch and add PostgreSQL/Redis Testcontainers coverage for: + +- concurrent `SELECT FOR UPDATE` claims; +- lease expiry and stale token rejection against PostgreSQL; +- outbox recovery after Redis interruption; +- Redis pending-entry takeover and dead-letter behavior; +- Flyway migration from V6 data containing legacy filename-only evaluations. + +These cases are deferred because the current environment does not provide the +agreed Docker runtime; they are not silently claimed as completed by this plan.