New folder structure

[arav-agarwal2]

Migrates the builder and the checker to the MLPerf Endpoints v1.0 submission format.

Both pipelines move together, because the failure mode this is guarding against is the
two halves disagreeing — a builder writing a bundle its own checker rejects, which is
what issue #72 turned out to be.

Important

This implements against two open endpoints_policies PRs. If either changes before
merge, the corresponding piece here needs revisiting.

• PR #119 (feat/remove_runtime_meta.json) — the file layout below.
• PR #117 (feat/add-1.0-seeds) — the seed values, mirrored into
  src/submission_checker/data/seed_sets.yaml.

Everything else follows v1.0_rules_dev @ 32f6821.

---

The format change

run_metadata.json and the per-system system_desc_id.json are both gone; the §8.2
system description now lives in every Pareto point.

<submitting_organization>/
└── <submission_id>/
    ├── src/<implementation>/README.md      # SHARED
    ├── docs/                               # SHARED
    └── results/<system>/<model_name>/r<N>/
            point.yaml            # §8.3 disclosure
            system_desc.json      # §8.2 — per point since PR #119
            result_summary.json
            accuracy_results.json
            config.yaml           # now OPTIONAL
            server_configs/

Region boundaries are now per submission

The largest behavioural change, and the one worth reviewing most closely: §5.5's
algorithm takes (C_max, C_min), and C_min is derived from the submission's own
points rather than declared. The checker previously computed regions before reading a
single point; that control flow had to invert.

compute_regions(c_max, c_min)   # was compute_regions(M)

_check_model is now two-phase — parse every point.yaml, derive C_min = min(concurrency),
compute boundaries, then run the region-dependent rules. Three calls made explicit in code:

• C_min is per (system, model) curve, not per submission — §8.5 defines a result as
  one system, one benchmark model, one dataset. Otherwise one model's smallest point would
  move another model's boundaries.
• Only parseable points contribute. A corrupt point.yaml has no trustworthy
  concurrency; including it would shift every boundary and cascade spurious failures onto
  the valid points. region-basis reports WARN on a partial basis, ERROR on none.
• C_min clamps to 32 when no point is that low, so the rest of the report stays useful
  instead of collapsing; the missing band is reported by ultra-low-concurrency-coverage.

The 10 % margin is its own region now (high_concurrency ends at C_max), and a margin
point does not satisfy High Concurrency coverage — §5.4 says the margin "does not
affect the required point distribution". That reading is one constant,
MARGIN_SATISFIES_HIGH_CONCURRENCY, so a WG ruling flips it in one line.

Region names follow §5.5/§9.1 (*_concurrency), not §8.3's stale *_throughput table.

New §9.1 checks

Rule	Checks
shared-path-resolution	shared_src / shared_docs resolve under the submission root, traversal-free
seed-set-consistency	every point records the same seed set
seed-set-membership	that set is one MLCommons published
seed-runtime-match	the RNG seeds equal the bound set's values
seed-set-adoption	§4.6's four-cohort window — SKIP + INFO, see below
target-cohort	present and of the form YYYY-MM-C0 / -C1
point-disclosure-complete	every required §8.3 field present
warmup-logs-retained	retention declared (WARN — §9.1's action is "Flag", not "Reject")
metric-consistency-tpot-p90	P90 present, finite, strictly positive
system-description-consistency	every point of a curve describes the same system
region-basis	reports the derived C_min and how many points it came from
ultra-low-concurrency-coverage	≥ 1 point at concurrency ≤ 32

Retired: run-metadata-present / run-metadata-valid (file is gone), seed-config
(hard-coded 42 — actively wrong once rotation is live; survives as seed-config-legacy
only for points that declare no seed set), and low-latency-coverage (vacuous against a
derived low_latency region, which is always 1–C_min).

Metrics: tps_per_user is now 1000 / tpot_p90_ms (was system_tps / concurrency),
and TTFT reports at P90. With run_metadata.json gone, result_summary.json is the only
TPOT source — so §9.1's "the valid per-response TPOT distribution must be non-empty" is
only verifiable as a property of the reported percentile. The rule message says so
rather than implying the samples were audited.

Seeds ship as data

v0.7 fixed every seed at 42. v1.0 rotates them, so the published sets live in
src/submission_checker/data/seed_sets.yaml and are overridable with --seed-sets FILE
or $MLPERF_ENDPOINTS_SEED_SETS — a newly published set does not need a checker release.

Bug fixes found along the way

• submissions update / add-run / remove-run were broken. prepare_pr_branch_merge
  still walked v0.7's pareto/<system>/<model>/{points,results,accuracy}/ and copied a
  top-level systems/. Against current builder output the walk silently matched nothing and
  the systems/ copy raised FileNotFoundError — so every amendment failed once the org
  directory existed on the branch, which is the normal case. Rewritten against §8.1's tree,
  preserving reviewer-edited system_desc.json across rebuilds. It had no test coverage
  for the merge body, which is why it survived the migration; now has eight.
• tps_utilization was normalised per model across systems in the builder while the
  checker recomputed it per (system, model) curve — so no two-system submission could
  match its own file.
• The results directory was named from config.yaml. §8.1 names it <model_name>,
  which is §8.2's supported-model-list name; config.yaml carries a HuggingFace path, so
  the tree got Llama-3_1-8B-Instruct where the spec asks for llama3.1-8b. Now read from
  point.yaml, matching _extract_concurrency right beside it.

---

Behaviour changes reviewers should weigh

1. config.yaml is no longer required — in the run folder or the bundle. point.yaml is
   required in both. It is still shipped whenever a run supplies it.
2. A run whose type cannot be determined now fails the build rather than defaulting to
   "performance". Defaulting was silently destructive: an accuracy run shipped without a
   config.yaml would be filed as its concurrency's performance run, collide with the real
   one, and drop its accuracy results from the bundle.
3. Model directory names change — Llama-3_1-8B-Instruct → llama3_1-8b (see above).
4. point-disclosure-complete landed as ERROR, not WARN. Full §9.1 conformance requires
   it; easy to downgrade if the first round should land softly.
5. The builder now injects shared_src / shared_docs when a point does not declare
   them. This is the one exception to Ensure config and point yamls are handled properly #72's "the builder does not derive §8.3 disclosure",
   and deliberately so: these keys carry no measurement claim, and their referent —
   src/<impl>/, the union of every run's src/ — does not exist until the builder
   assembles the bundle, so no run archive can name it. Same principle already governs
   tps_utilization. Validate if present, inject if absent, never overwrite; injection
   is append-only so the submitter's comments and key order survive.

Spec defects — judgement calls made here

Each of these is a place the spec contradicts itself; the resolution is recorded at the
code site. Worth a WG confirmation.

Conflict	Resolved as
§8.3's region enum says *_throughput; §5.5 and §9.1 say *_concurrency	follow §5.5
shared_src, shared_docs, seed_set, target_cohort required by §8.1 prose + §9.1 but deleted from §8.3's table by PR #90	treat as required
§8.2's table and §8.2.1's template are disjoint in both directions	implement the union
Availability has three spellings (publication_status / system_availability_status / availability_status)	accept all three; error only when two disagree
Five system_desc_id.json references survive PR #119 (§8.2 heading, TOC, §8.2.1, §8.5, §9.1)	read as system_desc.json
§4.3 "not per measurement point" vs §6.6 "including per measurement point"	follow §9.1 + §4.3
§5.4's Worked Example C is arithmetically wrong — gives Low 16–26, Med …–116; the algorithm and Appendix B give 17–26 and …–117	trust §5.5 + Appendix B
seedset.yaml carries no cohort keys, so §4.6's four-cohort adoption window is unevaluable	seed-set-adoption reports SKIP + INFO; enforces automatically once cohorts appear

---

Verification

• 853 tests pass; ruff, ruff format, mypy --strict over src, and sphinx-build -W
  are all clean.
• Appendix B's full 14-row table is parametrised in test_regions.py and reproduces
  exactly. This is the highest-value test here: §5.5 requires submitters to use this
  implementation to compute their boundaries, so whatever it asserts is normative for
  everyone.
• compute_regions(1024, 32) is asserted to reproduce the v0.7 boundaries
  (33–42 / 43–131 / 132–1024), which is what keeps the layout-coupled fixtures valid
  across the signature change.
• The builder→checker loop is closed end to end: a 7-point compliant curve built by the
  builder passes the real checker with zero errors, with a second test naming all 15 new
  rules individually so a regression says which one broke.
• Partial-failure paths are covered: corrupt the lowest point.yaml and C_min comes from
  the rest with region-basis WARN; corrupt them all and it is an ERROR while point-count
  still runs and the region-dependent rules are skipped rather than reported against no basis.

Fixture corpus

All 12 test_submissions/ trees regenerated by tests/tools/regenerate_fixtures.py,
committed and idempotent — a one-shot migration script would be unrunnable the moment
someone hand-edited a fixture, and the next format change would start from scratch. Its
idempotence is asserted, not trusted.

It also cleared stale artifacts already in the corpus (results_summary.json ×10,
point_<N>.yaml ×9 from the pre-r<N> naming) and gave every tree the
src/<impl>/README.md §2.2.1 has required all along — those fixtures were failing
src-dir for reasons no test asserted, and every test had to look past it.

One fixture needed repair: valid_standardized is the must-pass tree and stopped
passing under v1.0 — C_max=1000 with a derived C_min=16 puts Low Concurrency at
17–26, and its points were 16/38/88/…, so nothing landed in the window. v0.7's fixed
33–42 caught 38, which is why it was built that way. r38 → r20 is the minimal fix.

Not done

• §6.2's per-region durations are wired and warn-only — §6 is still marked WORK IN PROGRESS
  upstream.
• §6.4's query minimums stay as our MIN_QUERY_COUNT table; §6.4 gives prose, not numbers.
• result_summary.json / accuracy_results.json have no field-level schema anywhere in the
  policies repo — ours remains the de-facto contract.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[arav-agarwal2]

As we switched test fixtures, the large-scale test submissions changed. Unfortunately we can't really change how the diff looks long-term.

[arav-agarwal2]

Superseded by a 3-PR stack — same tree, split for review:

• 1/3 Regenerate the fixture corpus for MLPerf Endpoints v1.0 #80 1-fixtures → main — the 439-file regenerated corpus + its generator. CI red by design (v1.0 corpus, v0.7 checker); goes green in 2/3 Migrate the builder and checker to MLPerf Endpoints v1.0 #81.
• 2/3 Migrate the builder and checker to MLPerf Endpoints v1.0 #81 2-checker → 1-fixtures — the PR to review, 45 files of src + tests.
• 3/3 Document the v1.0 format, rules, and run-folder layout #82 3-docs → 2-checker — docs, README, pyproject.

The stack tip is byte-identical to this branch (same tree hash 70dd1c7), so nothing was dropped in the split. Suggest closing this one.