3/3 Document the v1.0 format, rules, and run-folder layout by arav-agarwal2 · Pull Request #82 · mlcommons/endpoints-submission-cli

arav-agarwal2 · 2026-09-09T16:21:33Z

Stack 3 of 3 · main ← [#80 1-fixtures] ← [#81 2-checker] ← 3-docs

Documentation only — no source or test changes. 7 files.

README's structure diagram and rules tables were still describing v0.7's systems/ +
pareto/ tree and its four *_throughput coverage rules. Rewritten against §8.1's tree,
with the ~57 checks grouped by area (structure, system description, regions, measurement
points, seed binding, metrics, accuracy) so a submitter can find the rule that rejected
them.

References corrected

Some had drifted with the format; one was never right.

Was	Now
`system_desc_id.json`, `run_metadata.json`	both removed by policies PR #119
`system_info.json`	a filename the CLI has never read — it wants `system_desc.json`
seeds documented as `42`	v1.0 rotates them (§4.6)
"`config.yaml` and `point.yaml` are both required"	`config.yaml` is required in neither the run folder nor the bundle
flat `result_summary.json` in the run folder	the phase directories `mlcommons/endpoints` actually writes

The run-folder layout blocks now link to docs/endpoints-cli/reference/run-folder-layout.md
(added in #78) rather than restating it, so there is one place to keep current.

GitHub PR flow removed from the docs

Follows the module deletion in #81. The architecture reference documented github_ops.py
in its module inventory, dependency graph, two mermaid sequence diagrams, a
update_pr_branch strategy section, and the exception hierarchy — all describing code that
has been unreachable since May and is now gone. The usage docs no longer claim
submissions create opens a pull request, and no longer print a PR: … line it never
emits.

pr_url / pr_number stay documented as fields submissions get displays: the record
still carries them, the CLI just doesn't set them.

The docs described this flow far more thoroughly than the code still implemented it — the
gh install-and-authenticate prerequisites, MLPERF_SUBMISSION_REPO (which no source file
reads), four per-command pipelines whose numbered steps included cloning the repo and
pushing to a PR branch, three copies of the PR-branch file update strategy table, and
sample output printing a PR URL the CLI does not emit.

Post-submission run additions removed

Follows the command deletion in #81. endpoints_policies
7fd3e89 deleted
Submission Rules §8.1 Pareto Updates, so the add-run documentation goes with it.

Rather than only delete mentions, the rule is recorded where a reader will meet it — on
remove-run and update --run-ids, with the consequence spelled out: withdrawing a faulty
point is still permitted (§8.1 Corrections), but a submission that drops below the
7-point minimum cannot be repaired by adding another.

`pyproject.toml`

Adds package-data so data/seed_sets.yaml ships with the wheel. Without it the seed
checks would fail on an installed copy while passing from a checkout — the kind of gap that
only shows up after release.

github-actions · 2026-09-09T16:21:43Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

anandhu-eng · 2026-09-10T06:28:35Z


+A run flagged `--test` shows as `Test Run │ Yes` in `runs get`. It is **not** visible in
+`runs list`: that view is served by the API's `RunSummary` schema, which does not carry
+`is_test`, so the test marker stays dormant there until the API exposes it.


I think this is more like a TODO?

anandhu-eng

LGTM.

One point I noted: There are information being duplicated multiple places. Especially related to cli sub commands. It would be great if we could clean them

anandhu-eng · 2026-09-10T13:53:34Z

+`Accuracy + Performance`, which describes the dataset rather than this run — the build
+fails with a message naming the run.
+
+That is deliberate. Defaulting to "performance" was silently destructive: an accuracy


"defaulting to performance was" to " defaulting to performance is"?

anandhu-eng · 2026-09-10T14:13:10Z

-9. Push the merged branch to the GitHub PR.
+1. `GET /submissions/{id}` — fetch the current run list and division.
+2. **Reject the update if it would add a run** (see below); log removed runs.
+3. `PATCH /submissions/{id}` with the new `run_ids` (and any metadata fields) in a single call.


Should the wording be "updated run_ids". Feel the "new run_ids" might convey the possibility to add new points.

anandhu-eng · 2026-09-10T14:18:12Z

+**Order of operations:** DB update (`DELETE /submissions/{id}`) → delete archive (`DELETE /submissions/{id}/archive`).

-PR closure and archive deletion are best-effort — failures are reported as warnings but do not change the exit code. The submission is already `WITHDRAWN` in the database.
+Archive deletion is best-effort — a failure is reported as a warning but does not change the exit code. The submission is already `WITHDRAWN` in the database. The CLI does not close the review pull request; it no longer manages one.


it no longer manages one

I'm wondering if we should leave reference to the past? Seems the past is referenced in many part of documentation

anandhu-eng · 2026-09-10T14:29:25Z

-**If the GitHub PR step fails** the submission record and uploaded bundle still
-exist. Retry the PR step manually with `gh pr create` on the submission branch,
-then contact MLCommons to update the PR linkage.
+**If the final status PATCH fails** the submission record and uploaded bundle still


What does "final status PATCH" mean here?

anandhu-eng · 2026-09-10T14:40:45Z

-            └── accuracy/
-                ├── accuracy.txt
-                └── accuracy_result.json
+<submitting_organization>/


The directory structure seems to be documented multiple places. how about documenting it in one place and referencing in others

README's structure diagram and rules tables were still describing v0.7's systems/ + pareto/ tree and its four *_throughput coverage rules. Rewritten against §8.1's tree, with the ~57 checks grouped by area (structure, system description, regions, measurement points, seed binding, metrics, accuracy) so a submitter can find the rule that rejected them. Also corrects references that had drifted or were never right: system_desc_id.json and run_metadata.json (both removed by policies PR #119), system_info.json (a filename the CLI has never read — it wants system_desc.json), seeds documented as 42 (v1.0 rotates them per §4.6), and the claim that config.yaml is required, which it no longer is in either the run folder or the bundle. The run-folder layout blocks now show the phase directories mlcommons/endpoints actually writes, and link to docs/endpoints-cli/reference/run-folder-layout.md rather than restating it. Drops the GitHub PR flow, which the docs described far more thoroughly than the code ever still implemented: the architecture reference's module inventory, dependency graph, two sequence diagrams, update_pr_branch strategy section and GitHubError; the `gh` install-and-authenticate prerequisites in the README and getting-started guide; MLPERF_SUBMISSION_REPO, which no source file reads; four per-command pipelines whose numbered steps included cloning the repo, merging into a PR branch and pushing; three copies of the PR-branch file update strategy table; and sample output printing a PR URL the CLI does not emit. Removes the add-run documentation to match the command's deletion, and records the rule behind it where a reader will meet it — on remove-run, which is the operation that now has a sharp edge. Withdrawing a faulty point is still permitted by §8.1 Corrections, but §8 no longer provides a window for adding one, so a submission that drops below the 7-point minimum cannot be repaired. `update --run-ids` gets the same note. pyproject gains package-data so data/seed_sets.yaml ships with the wheel; without it the seed checks would fail on an installed copy while passing from a checkout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

arav-agarwal2 mentioned this pull request Sep 9, 2026

New folder structure #79

Closed

arav-agarwal2 added this pull request to stack #83 September 9, 2026 16:22

arav-agarwal2 force-pushed the v1.0-rules/3-docs branch 2 times, most recently from 79b5c31 to 0d3ddb3 Compare September 9, 2026 16:56

arav-agarwal2 mentioned this pull request Sep 9, 2026

2/3 Migrate the builder and checker to MLPerf Endpoints v1.0 #81

Merged

anandhu-eng reviewed Sep 10, 2026

View reviewed changes

anandhu-eng approved these changes Sep 10, 2026

View reviewed changes

arav-agarwal2 force-pushed the v1.0-rules/3-docs branch from 0d3ddb3 to 2ace2e5 Compare September 10, 2026 16:35

arav-agarwal2 force-pushed the v1.0-rules/3-docs branch from 2ace2e5 to 18474f6 Compare September 11, 2026 16:12

Base automatically changed from v1.0-rules/2-checker to main September 11, 2026 16:18

arav-agarwal2 force-pushed the v1.0-rules/3-docs branch from 18474f6 to 49e4ec6 Compare September 11, 2026 16:18

arav-agarwal2 merged commit f48ca84 into main Sep 11, 2026
7 checks passed

Navigation Menu

Uh oh!

3/3 Document the v1.0 format, rules, and run-folder layout - #82

3/3 Document the v1.0 format, rules, and run-folder layout#82
arav-agarwal2 merged 1 commit into
mainfrom
v1.0-rules/3-docs

arav-agarwal2 commented Sep 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

anandhu-eng Sep 10, 2026

Uh oh!

anandhu-eng left a comment

Uh oh!

anandhu-eng Sep 10, 2026

Uh oh!

anandhu-eng Sep 10, 2026

Uh oh!

anandhu-eng Sep 10, 2026

Uh oh!

anandhu-eng Sep 10, 2026

Uh oh!

anandhu-eng Sep 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

arav-agarwal2 commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

References corrected

GitHub PR flow removed from the docs

Post-submission run additions removed

pyproject.toml

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

anandhu-eng Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

anandhu-eng left a comment

Choose a reason for hiding this comment

Uh oh!

anandhu-eng Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

anandhu-eng Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

anandhu-eng Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

anandhu-eng Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

anandhu-eng Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

arav-agarwal2 commented Sep 9, 2026 •

edited

Loading

`pyproject.toml`