1/3 Regenerate the fixture corpus for MLPerf Endpoints v1.0 - #80
Merged
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
This was referenced Sep 9, 2026
Closed
arav-agarwal2
added this pull request to stack #83
September 9, 2026 16:22
anandhu-eng
approved these changes
Sep 10, 2026
Machine-generated. Review `tests/tools/regenerate_fixtures.py` and spot-check its output rather than reading 9,000 lines of regenerated JSON and YAML. The script is committed and idempotent on purpose: the corpus gets hand-edited between rounds, so a one-shot migration would be unrunnable the moment someone touched a fixture, and the next format change would start from scratch. Its idempotence is asserted by a test, not trusted. Per submission tree it: folds the per-system system_desc_id.json plus each point's run_metadata.json into a per-point system_desc.json (policies PR #119) and deletes both sources; renames system_availability_status to publication_status and nests the flat node-level accelerator_* fields into accelerator_info[] (§8.2/§8.2.1); recomputes tps_utilization per Pareto curve from result_summary.json so the stored value agrees with the measurement the checker recomputes it from; adds the §8.3 disclosure and the §4.6 seeds under their v1.0 names to every point.yaml; migrates TPOT out of run_metadata into result_summary.json in nanoseconds and interpolates a TTFT P90. It also clears artifacts the corpus had been carrying from layouts the spec no longer defines — results_summary.json x10 and point_<N>.yaml x9 from the pre-r<N> naming — and gives every tree the src/<impl>/README.md §2.2.1 has required all along. Those fixtures were failing src-dir for reasons no test asserted, so every test had to look past an error unrelated to what it was checking. One fixture needed repair rather than regeneration. valid_standardized is the must-pass tree and stops passing under v1.0: C_max=1000 with a derived C_min=16 puts Low Concurrency at 17-26, and its points were 16/38/88/256/512/768/1000, so nothing lands in the window. v0.7's fixed 33-42 window caught 38, which is why it was built that way. Moving that point to r20 is the smallest change that restores coverage. This commit is red on its own: the corpus is v1.0 while the checker reading it is still v0.7. The checker follows in the next commit of the stack. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
arav-agarwal2
force-pushed
the
v1.0-rules/1-fixtures
branch
from
September 11, 2026 16:11
7f257b5 to
3544d51
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 1 of 3 ·
main← 1-fixtures ← [2-checker] ← [3-docs]Regenerates the 12
test_submissions/trees for the v1.0 format, and commits thescript that did it.
Warning
CI is red on this PR, by design. The corpus is v1.0 while the checker reading it
is still v0.7 — 22 failures, all of that shape. It goes green in stack 2, which is
where the checker lands. Nothing merges to
mainuntil the tip is green.How to review this
Read
tests/tools/regenerate_fixtures.pyand spot-check its output. Reading 9,000lines of regenerated JSON and YAML is not a good use of anyone's time.
The script is committed and idempotent on purpose. The corpus gets hand-edited between
rounds, so a one-shot migration would be unrunnable the moment someone touched a
fixture, and the next format change would start from scratch.
--checkexits non-zeroif anything would change; a test in stack 2 asserts that, so idempotence is verified
rather than claimed.
What it does per tree
system_desc_id.jsonand each point'srun_metadata.jsoninto a per-pointsystem_desc.json(policies PR #119), then deletes both sources.system_availability_status→publication_status; flat node-levelaccelerator_*fields nest intoaccelerator_info[](§8.2.1).tps_utilizationresult_summary.json, so the stored value agrees with the measurement the checker recomputes it from.point.yamlgains the ten §8.3 fields plusshared_src,shared_docs,seed_set,target_cohort.run_metadataintoresult_summary.jsonin nanoseconds; a TTFT P90 interpolated from the summary's own P50/P95.Two things it cleans up that predate this migration
Stale artifacts.
results_summary.json×10 andpoint_<N>.yaml×9, left over fromthe pre-
r<N>naming.Missing
src/trees. Everysub_*fixture was failingsrc-dir— §2.2.1 hasrequired
src/<impl>/README.mdall along — for reasons no test asserted, so every testhad to look past an error unrelated to what it was checking. Each tree now has one.
One fixture needed repair, not regeneration
valid_standardizedis the corpus's must-pass tree and stops passing under v1.0.C_max=1000with a derivedC_min=16puts Low Concurrency at17–26, and its pointswere
16/38/88/256/512/768/1000— nothing lands in the window. v0.7's fixed33–42window caught 38, which is why it was built that way. Moving that point to
r20is thesmallest change that restores coverage, and it is recorded in
POINT_RENAMESwith thatreasoning rather than applied silently.