[anle2024]

Summary

Require every configured validation and test metric before accepting a fitted result or reference. Compare each prediction with the sampled input population, record requested/scored/skipped counts, and reject unexpected skips.

Resolve forecast coordinates from the scoring snapshot metadata. Compare timestamp values so equivalent nanosecond and microsecond storage formats match.

The four-file patch has 169 additions and 20 deletions, including an 89-line regression test file.

Change map

File-by-file change map.

Testing Done

• 34 repository CPU checks passed: python -m pytest -q -c /dev/null --confcutdir=test/scripts -p no:cacheprovider test/scripts/test_engine_e2e_case_results.py test/scripts/test_engine_e2e_case_hardware.py.
• An additional retained 117-case CPU regression suite passed against identical runner code.
• Flake8, isort and git diff --check passed.
• The wider CPU suite passed 189 tests. test_history_scan_explicitly_selects_all_commits failed because the local checkout is shallow.

GPU training and the full scale campaign were not rerun.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Caution

Review failed

The pull request is closed.

No actionable comments were generated in the recent review.

Walkthrough

The test-case runner now checks that predictions cover the requested task-specific population and have valid row counts. It requires configured metrics to be present and finite, records per-prediction counts, and checks repeated and reloaded results. New end-to-end tests cover metrics, population coverage, and forecast coordinates. Repository readiness now runs these tests.

Merge Risk: Minimal · up to f90e9

The proposed early validation does not prevent a reachable late failure. No actionable merge-blocking risk remains after normal checks.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Recent review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/kumo-engine
• Branch: main
• File: .coderabbit.yaml
• Commit: 8b4942f
• Layers: Repository YAML
• Review profile: QUIET
• Plan: Enterprise
• Run ID: b5a23ae8-ecfd-41ca-a984-7209a3febfe0

Commits

Reviewing files that changed from the base of the PR and between 8b4942f and f90e9f7.

Files selected for processing: 4

• .github/workflows/oss-readiness.yml
• packages/kumo-engine/test/cases/README.md
• packages/kumo-engine/test/cases/common.py
• test/scripts/test_engine_e2e_case_results.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

[anle2024]

Change map

The runner could accept a missing test metric or three identical outputs that each omitted the same requested input. Its fixed forecast keys also omitted the entity's configured time column.

File	Function / scope	What changed	Why
packages/kumo-engine/test/cases/common.py	prediction_table, check_metrics, prediction_counts, check_population, prediction_coordinates, check_predictions, run	Retain sampled input keys, require configured validation/test metrics, check population counts and coordinates, and resolve the forecast clock from metadata.	Reject incomplete results before pass status or reference publication. Compare exact timestamp values across storage units.
test/scripts/test_engine_e2e_case_results.py	Metric, population and forecast regressions	Add five compact CPU cases, including missing rows, foreign rows, skipped counts, reordered forecasts and timestamp precision.	Reproduce the three gaps without GPU or cloud dependencies.
.github/workflows/oss-readiness.yml	Repository CPU checks	Include the new regression file.	Run these guards in existing PR CI.
packages/kumo-engine/test/cases/README.md	Result acceptance contract	Document required metrics, complete sampled populations, zero skips and forecast coordinates.	Make the stricter reference and candidate checks clear.

Verification: 34 repository CPU checks and 117 retained author regressions passed. Flake8, isort and whitespace checks passed. The wider CPU suite has one existing shallow-history failure, recorded in the description.

Scope ends at these shared runner guards and focused regressions. Training budgets, learning thresholds and prediction tolerances stay unchanged.

[anle2024]

Superseded by #303. The replacement uses a branch in NVIDIA/kumo-engine and retains the same verified signed commit.