2/3 Migrate the builder and checker to MLPerf Endpoints v1.0

[arav-agarwal2]

Stack 2 of 3 · main ← [#80 1-fixtures] ← 2-checker ← [3-docs]
· This is the PR to review. The 439-file generated fixture diff is in #80.

Note

Force-pushed twice since the first review request — three commits now, not one.
submissions/github.py turned out to be unreachable, so rather than have this PR
rewrite it and a later one delete it, the migration commit no longer touches it and
the deletion is its own commit. A third commit removes submissions add-run:
endpoints_policies 7fd3e89
deleted Submission Rules §8.1 Pareto Updates, so points may no longer be added
after submission. The v1.0 migration itself is unchanged in substance.

Migrates the builder and the checker to the MLPerf Endpoints v1.0 format. 863 tests
pass; ruff, ruff format, mypy --strict over src, and sphinx-build -W are clean.

Both pipelines move together, because the failure this guards against is the two halves
disagreeing — a builder writing a bundle its own checker rejects, which is what issue #72
turned out to be. A test now builds a compliant 7-point curve and runs the real checker
over it, so that class of bug cannot reopen silently.

Important

This implements against two open endpoints_policies PRs. If either changes before
merge, the corresponding piece here needs revisiting.

• PR #119 (feat/remove_runtime_meta.json) — the file layout.
• PR #117 (feat/add-1.0-seeds) — the seed values, mirrored into data/seed_sets.yaml.

Everything else follows v1.0_rules_dev @ 32f6821.

---

Region boundaries are now per submission

The largest behavioural change, and the one worth reviewing most closely. §5.5's algorithm
takes (C_max, C_min), and C_min is derived from the submission's own points rather
than declared. The checker previously computed regions before reading a single point; that
control flow had to invert.

compute_regions(c_max, c_min)   # was compute_regions(M)

_check_model is two-phase — parse every point.yaml, derive C_min = min(concurrency),
compute boundaries, then run the region-dependent rules. Three calls made explicit in code:

• C_min is per (system, model) curve, not per submission — §8.5 defines a result as
  one system, one benchmark model, one dataset. Otherwise one model's smallest point would
  move another model's boundaries.
• Only parseable points contribute. A corrupt point.yaml has no trustworthy
  concurrency; including it would shift every boundary and cascade spurious failures onto
  the valid points. region-basis reports WARN on a partial basis, ERROR on none.
• C_min clamps to 32 when no point is that low, so the rest of the report stays useful
  instead of collapsing; the missing band is reported by ultra-low-concurrency-coverage.

The 10 % margin is its own region now (high_concurrency ends at C_max), and a margin
point does not satisfy High Concurrency coverage — §5.4 says the margin "does not affect
the required point distribution". That reading is one constant,
MARGIN_SATISFIES_HIGH_CONCURRENCY, so a WG ruling flips it in one line.

Region names follow §5.5/§9.1 (*_concurrency), not §8.3's stale *_throughput table.

New §9.1 checks

Rule	Checks
shared-path-resolution	shared_src / shared_docs resolve under the submission root, traversal-free
seed-set-consistency	every point records the same seed set
seed-set-membership	that set is one MLCommons published
seed-runtime-match	the RNG seeds equal the bound set's values
seed-set-adoption	§4.6's four-cohort window — SKIP + INFO, see below
target-cohort	present and of the form YYYY-MM-C0 / -C1
point-disclosure-complete	every required §8.3 field present
warmup-logs-retained	retention declared (WARN — §9.1's action is "Flag", not "Reject")
metric-consistency-tpot-p90	P90 present, finite, strictly positive
system-description-consistency	every point of a curve describes the same system
region-basis	reports the derived C_min and how many points it came from
ultra-low-concurrency-coverage	≥ 1 point at concurrency ≤ 32

Retired: run-metadata-present / -valid (file is gone), seed-config (hard-coded 42 —
actively wrong once rotation is live; survives as seed-config-legacy only for points that
declare no seed set), and low-latency-coverage (vacuous against a derived low_latency
region, which is always 1–C_min).

Metrics: tps_per_user is now 1000 / tpot_p90_ms (was system_tps / concurrency), and
TTFT reports at P90. With run_metadata.json gone, result_summary.json is the only TPOT
source — so §9.1's "the valid per-response TPOT distribution must be non-empty" is only
verifiable as a property of the reported percentile. The rule message says so rather than
implying the samples were audited.

Seeds ship as data. data/seed_sets.yaml, overridable with --seed-sets FILE or
$MLPERF_ENDPOINTS_SEED_SETS, so a newly published set does not need a checker release.

Two pre-existing bugs surfaced along the way

• tps_utilization was normalised per model across systems in the builder while the
  checker recomputed it per (system, model) curve — so no two-system submission could match
  its own file.
• The results directory was named from config.yaml. §8.1 names it <model_name>, which
  is §8.2's supported-model-list name; config.yaml carries a HuggingFace path, so the tree
  got Llama-3_1-8B-Instruct where the spec asks for llama3.1-8b.

---

Behaviour changes reviewers should weigh

1. config.yaml is no longer required — in the run folder or the bundle. point.yaml is
   required in both. It is still shipped whenever a run supplies it.
2. A run whose type cannot be determined now fails the build rather than defaulting to
   "performance". Defaulting was silently destructive: an accuracy run shipped without a
   config.yaml would be filed as its concurrency's performance run, collide with the real
   one, and drop its accuracy results from the bundle.
3. Model directory names change — Llama-3_1-8B-Instruct → llama3_1-8b.
4. point-disclosure-complete landed as ERROR, not WARN. Full §9.1 conformance requires
   it; easy to downgrade if the first round should land softly.
5. The builder now injects shared_src / shared_docs when a point does not declare them.
   This is the one exception to Ensure config and point yamls are handled properly #72's "the builder does not derive §8.3 disclosure", and
   deliberately so: these keys carry no measurement claim, and their referent —
   src/<impl>/, the union of every run's src/ — does not exist until the builder assembles
   the bundle, so no run archive can name it. Same principle already governs
   tps_utilization. Validate if present, inject if absent, never overwrite; injection is
   append-only, so the submitter's comments and key order survive.

Spec defects — judgement calls made here

Each is a place the spec contradicts itself; the resolution is recorded at the code site.
Worth a WG confirmation.

Conflict	Resolved as
§8.3's region enum says *_throughput; §5.5 and §9.1 say *_concurrency	follow §5.5
shared_src, shared_docs, seed_set, target_cohort required by §8.1 prose + §9.1 but deleted from §8.3's table by PR #90	treat as required
§8.2's table and §8.2.1's template are disjoint in both directions	implement the union
Availability has three spellings (publication_status / system_availability_status / availability_status)	accept all three; error only when two disagree
Five system_desc_id.json references survive PR #119 (§8.2 heading, TOC, §8.2.1, §8.5, §9.1)	read as system_desc.json
§4.3 "not per measurement point" vs §6.6 "including per measurement point"	follow §9.1 + §4.3
§5.4's Worked Example C is arithmetically wrong — gives Low 16–26, Med …–116; the algorithm and Appendix B give 17–26 and …–117	trust §5.5 + Appendix B
seedset.yaml carries no cohort keys, so §4.6's four-cohort adoption window is unevaluable	seed-set-adoption reports SKIP + INFO; enforces automatically once cohorts appear

Verification

• Appendix B's full 14-row table is parametrised in test_regions.py and reproduces
  exactly. This is the highest-value test here: §5.5 requires submitters to use this
  implementation to compute their boundaries, so whatever it asserts is normative for
  everyone.
• compute_regions(1024, 32) is asserted to reproduce the v0.7 boundaries
  (33–42 / 43–131 / 132–1024), which is what keeps the layout-coupled fixtures valid across
  the signature change.
• The builder→checker loop is closed end to end: a 7-point compliant curve built by the
  builder passes the real checker with zero errors, with a second test naming all 15 new
  rules individually so a regression says which one broke.
• Partial-failure paths are covered: corrupt the lowest point.yaml and C_min comes from
  the rest with region-basis WARN; corrupt them all and it is an ERROR while point-count
  still runs and the region-dependent rules are skipped rather than reported against no basis.

Not done

• §6.2's per-region durations are wired and warn-only — §6 is still marked WORK IN PROGRESS
  upstream.
• §6.4's query minimums stay as our MIN_QUERY_COUNT table; §6.4 gives prose, not numbers.
• result_summary.json / accuracy_results.json have no field-level schema anywhere in the
  policies repo — ours remains the de-facto contract.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[arav-agarwal2]

I'm trying out gh PR stacking - can you take a look at just this and see what you think? The hope is this should allow you us to separate changes with docs changes.

[arav-agarwal2]

Force-pushed — please re-pull before reviewing. Two commits now:

1. Migrate the builder and checker to v1.0 — unchanged in substance.
2. Delete the unreachable GitHub PR module — new.

You were right to ask about github.py. An AST walk over src/ finds zero live imports and zero live calls to it: every call site in update/add_run/remove_run/withdraw sits inside a triple-quoted string block, disabled in e722875 ("Temporarily remove github usage while testing", 2026-05-29) — 143 commits ago — and submissions create dropped PR creation outright.

I had grepped, found those call sites, and taken them for live code. So the earlier revision of this PR claimed the rewrite fixed submissions update/add-run/remove-run; it did not, nothing was failing in practice, and I've corrected that claim. The rewrite is gone and the module is deleted instead: −357 lines of source, −368 of tests, −140 of commented-out call sites, plus GitHubError whose only raiser it was.

Test count drops 863 → 825: 33 from test_github.py and 5 command tests whose whole subject was the PR flow. Everything else kept its assertions and lost only mocks of a module that no longer exists.

[arav-agarwal2]

Force-pushed again — third commit added: Remove post-submission run additions.

You were right about the rules. endpoints_policies 7fd3e89 — "Remove post-submission pareto run additions.", 2026-09-02 — deleted Submission Rules §8.1 Pareto Updates, which had allowed:

Submitters may add additional measurement points to their pareto curve during a post-submission update window of 90 days from the initial submission date.

Everything under §8 shifted up one; the TOC still lists the removed section.

CLI changes:

• submissions add-run deleted — it implemented exactly that window.
• submissions update --run-ids now rejects a list that would add a run, before anything is fetched or patched. Removals still work.
• remove-run kept: §8.1 Corrections still lets a submitter withdraw a faulty point during peer review.
• api.add_run_to_submission kept, used only to undo a failed remove-run; both it and the rollback helper now say why, since an add call surviving a commit that removes adding will otherwise look wrong.

Sharp edge now documented: withdrawn points don't count toward the 7-point minimum, and that shortfall can no longer be repaired by adding another point. The checker reports it. This is on remove-run and update --run-ids in the docs (#82).

Also fixed stale help text I left in the previous commit — remove-run, withdraw and update were all still advertising GitHub PR steps that the deleted module used to perform.

Tests 825 → 824.