Skip to content

1/3 Regenerate the fixture corpus for MLPerf Endpoints v1.0 - #80

Merged
arav-agarwal2 merged 1 commit into
mainfrom
v1.0-rules/1-fixtures
Sep 11, 2026
Merged

1/3 Regenerate the fixture corpus for MLPerf Endpoints v1.0#80
arav-agarwal2 merged 1 commit into
mainfrom
v1.0-rules/1-fixtures

Conversation

@arav-agarwal2

Copy link
Copy Markdown
Collaborator

Stack 1 of 3 · main1-fixtures ← [2-checker] ← [3-docs]

Regenerates the 12 test_submissions/ trees for the v1.0 format, and commits the
script that did it.

Warning

CI is red on this PR, by design. The corpus is v1.0 while the checker reading it
is still v0.7 — 22 failures, all of that shape. It goes green in stack 2, which is
where the checker lands. Nothing merges to main until the tip is green.

How to review this

Read tests/tools/regenerate_fixtures.py and spot-check its output. Reading 9,000
lines of regenerated JSON and YAML is not a good use of anyone's time.

The script is committed and idempotent on purpose. The corpus gets hand-edited between
rounds, so a one-shot migration would be unrunnable the moment someone touched a
fixture, and the next format change would start from scratch. --check exits non-zero
if anything would change; a test in stack 2 asserts that, so idempotence is verified
rather than claimed.

What it does per tree

Per-point system description Folds the per-system system_desc_id.json and each point's run_metadata.json into a per-point system_desc.json (policies PR #119), then deletes both sources.
§8.2 renames system_availability_statuspublication_status; flat node-level accelerator_* fields nest into accelerator_info[] (§8.2.1).
tps_utilization Recomputed per Pareto curve from result_summary.json, so the stored value agrees with the measurement the checker recomputes it from.
§8.3 disclosure Every point.yaml gains the ten §8.3 fields plus shared_src, shared_docs, seed_set, target_cohort.
§4.6 seeds Rewritten under their v1.0 names from the mirrored seed set.
Metrics TPOT migrated out of run_metadata into result_summary.json in nanoseconds; a TTFT P90 interpolated from the summary's own P50/P95.

Two things it cleans up that predate this migration

Stale artifacts. results_summary.json ×10 and point_<N>.yaml ×9, left over from
the pre-r<N> naming.

Missing src/ trees. Every sub_* fixture was failing src-dir — §2.2.1 has
required src/<impl>/README.md all along — for reasons no test asserted, so every test
had to look past an error unrelated to what it was checking. Each tree now has one.

One fixture needed repair, not regeneration

valid_standardized is the corpus's must-pass tree and stops passing under v1.0.
C_max=1000 with a derived C_min=16 puts Low Concurrency at 17–26, and its points
were 16/38/88/256/512/768/1000 — nothing lands in the window. v0.7's fixed 33–42
window caught 38, which is why it was built that way. Moving that point to r20 is the
smallest change that restores coverage, and it is recorded in POINT_RENAMES with that
reasoning rather than applied silently.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Machine-generated. Review `tests/tools/regenerate_fixtures.py` and spot-check its
output rather than reading 9,000 lines of regenerated JSON and YAML.

The script is committed and idempotent on purpose: the corpus gets hand-edited
between rounds, so a one-shot migration would be unrunnable the moment someone
touched a fixture, and the next format change would start from scratch. Its
idempotence is asserted by a test, not trusted.

Per submission tree it: folds the per-system system_desc_id.json plus each point's
run_metadata.json into a per-point system_desc.json (policies PR #119) and deletes
both sources; renames system_availability_status to publication_status and nests the
flat node-level accelerator_* fields into accelerator_info[] (§8.2/§8.2.1); recomputes
tps_utilization per Pareto curve from result_summary.json so the stored value agrees
with the measurement the checker recomputes it from; adds the §8.3 disclosure and the
§4.6 seeds under their v1.0 names to every point.yaml; migrates TPOT out of
run_metadata into result_summary.json in nanoseconds and interpolates a TTFT P90.

It also clears artifacts the corpus had been carrying from layouts the spec no longer
defines — results_summary.json x10 and point_<N>.yaml x9 from the pre-r<N> naming —
and gives every tree the src/<impl>/README.md §2.2.1 has required all along. Those
fixtures were failing src-dir for reasons no test asserted, so every test had to look
past an error unrelated to what it was checking.

One fixture needed repair rather than regeneration. valid_standardized is the
must-pass tree and stops passing under v1.0: C_max=1000 with a derived C_min=16 puts
Low Concurrency at 17-26, and its points were 16/38/88/256/512/768/1000, so nothing
lands in the window. v0.7's fixed 33-42 window caught 38, which is why it was built
that way. Moving that point to r20 is the smallest change that restores coverage.

This commit is red on its own: the corpus is v1.0 while the checker reading it is
still v0.7. The checker follows in the next commit of the stack.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@arav-agarwal2
arav-agarwal2 merged commit 428596b into main Sep 11, 2026
4 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants