[docs] Benchmark suite and ruleset proposal by arekay-nv · Pull Request #468 · mlcommons/endpoints

arekay-nv · 2026-08-25T21:37:37Z

What does this PR do?

Adds a proposed Benchmark Suite spec: a git-versioned catalog of models, datasets, and per-benchmark contracts, with a cohort ruleset that freezes the valid benchmark IDs, suite version, and seed sets. A submission names only cohort, seed set, and benchmark (plus submitter-owned concurrency/endpoints); the client inlines the suite contract and the checker validates the run against it. Worked YAML examples cover DeepSeek-R1 and gpt-oss-120b (including multi-dataset accuracy, generation overrides, and gates). No code changes — this is the design source of truth for a follow-on schema/POC.

Type of change

Bug fix
New feature
Documentation update
Refactor/cleanup

Related issues

Testing

Tests added/updated
All tests pass locally
Manual testing completed

Checklist

Code follows project style
Pre-commit hooks pass
Documentation updated (if needed)

Signed-off-by: Rashid Kaleem <230885705+arekay-nv@users.noreply.github.com>

github-actions · 2026-08-25T21:37:47Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

arav-agarwal2

I don't think I can approve it just yet as it's a proposal, but my main worry is that I don't see a policies PR suggesting the changes to the submission directory and submission checker rules for this system.

arav-agarwal2 · 2026-08-31T14:37:30Z

+  - Pre-processing - executed to generate the complete configuration files, as well as perform any sanity checks (for instance current repo matches the SHA for the cohort tag ).
+  - Performance - measured execution of the performance dataset with the user specified concurrency.
+  - Accuracy - correctness checks performed over the accuracy dataset and scored by the designated scorer and aggregated to component scores. Each accuracy dataset can be run multiple times (repeats) and there can be different ways of aggregating the scores from different runs (best of K or average over K). As there can be multiple accuracy datasets, we can also specify aggregation across accuracy datasets (unweighted average), or alternatively leave the scores per dataset separate.
+  - Post-processing - performs integrity checks on the datasets (SHA verification), number of samples issued, model checkpoints. It will also ensure that the accuracy scores satisfies the gates specified in the benchmark specification. There can also be requirements on the performance phase such as minimum duration, minimum number of samples issued, time-to-first-token constraints etc.


The way I'm reading this, the document intends for the load generator to have a ton of submission-related checks.

Who is implementing these checks, and what's the goal of the submission checker given them? Is our goal to verify that these checks were made by the loadgen checker?

We already have some post-processing steps in the existing setup (steady state checks) and some expected to be added (steady state). The goal is to ensure that a submission is as close to compliance as possible and there are no issues that are overlooked (incorrect checkpoint, insufficient number of samples etc).
Submission checker would independently make those checks based on the benchmark specification which is more reliable as it is done on MLC side compared to the loadgen side checks which can be bypassed.

@arekay-nv , If we are blocking a submission due to any of the post-processing steps/checks by the submission checker, i think it would be better to mention the checks in the rules so that submitters would be aware of it. They probably would be searching for these guardrails in endpoints policies repo rather than the endpoints package repo

Hi folks, the purpose is to build clarity in the repo what is valid in a certain period of time (cohort) and this will not be captured in the policy repo

The pain point now is many new submitters or even old submitters are confused what a valid submission is, if they don't attend meeting weekly. It cannot be understood easily by reading the policy repo or the submission checker. Implementing the structure to constrain a "valid submission suite" will help us avoid many human mistakes during the submission process.

The post-submission checker should still be implemented, but they can assume the definition in these implemented ruleset and benchmark suite files, so we have one single source of truth to be quoted.

arav-agarwal2 · 2026-08-31T14:38:32Z

+  type: submission
+  submission_ref:
+    cohort:  2026-09-C0  # frozen selection (YYYY-MM-C0/C1, submission rules §4.2)
+    seed_set_id: B


Note: Target cohort should completely specify a seed set, so I'm unsure what seed set format you're expecting here. Also, what does "A" and "B" mean here? I assume it means the earlier and later available seed sets for a given submission, but it pays to be precise.

There is a cohort definiton example below that includes seeds

seed_sets: # a submission selects exactly one, by id - id: A scheduler_rng_seed: 16159082839903944936 sample_index_rng_seed: 2747215439041700203 model_seed: 42 - id: B scheduler_rng_seed: <generated at ruleset publication> sample_index_rng_seed: <generated at ruleset publication> model_seed: <generated at ruleset publication>

Just to be clear, we are not rotating seed per cohort right (would be helpful to link this to where we define how frequent the seed is changing)

I vaguely remember we discuss per 2-month or per quarter but I couldn't find it

Yes, it is a two-cohort seed rotation policy

arav-agarwal2 · 2026-08-31T14:41:52Z

+
+```
+benchmark_suite/
+├── suite.yaml             version; the loader globs the three directories below


I"m assuming this is all of the possible configuration for loadgen, which would be specified by the cohort-id.

Where would these details be stored? Do we know who has the responsibility for making sure that this is accurate?

These are independent of cohort. A cohort only defines which of these is valid for a submission into that cohort.
A task force will work independently to update the model/dataset/benchmark specs for its benchmark, and then merge those changes in. Then propose to open it up for submission at which point it will be added to the next cohort.

What does suite.yaml define? From the comment I am not too sure

For now it is just a container and contains the version of the suite schema. If we make breaking changes to it, we can then update the version.

arav-agarwal2 · 2026-08-31T14:42:33Z

+```
+benchmark_suite/
+├── suite.yaml             version; the loader globs the three directories below
+├── models/<id>.yaml       id, reference_id, tokenizer?, processor?, chat_template?,


What does refer to here? Is this a model-specific id and, if-so, where would that list be defined?

Yes, this would be a model specific id. Note that this is the list. We would not need to maintain a separate document as the models specified here are what can be used in a benchmark, which is what can be specified in a cohort for submission.

arav-agarwal2 · 2026-08-31T14:42:46Z

+├── suite.yaml             version; the loader globs the three directories below
+├── models/<id>.yaml       id, reference_id, tokenizer?, processor?, chat_template?,
+│                          metadata{}
+├── datasets/<id>.yaml     id, description, samples, source{url, sha256},


Same as models

Same as above. Datasets defined here are what can be used in a benchmark and then in a cohort.

arav-agarwal2 · 2026-08-31T14:42:51Z

+├── datasets/<id>.yaml     id, description, samples, source{url, sha256},
+│                          prompts{}, ground_truth_or_tests{}, evaluation_recipes{}?,
+│                          metadata{}
+└── benchmarks/<id>.yaml   id, family, model, api_type,


Same as models

This is one level below a cohort. A cohort will define the set of benchmarks for submission.

arav-agarwal2 · 2026-08-31T14:44:14Z

+    postprocessing:
+      gates:
+        [
+          dataset_integrity,


If this needs to be verified by the submission checker after running the benchmark, these files should be present during submission. Is there a PR in the rules repo for getting these into the submission file structure? I couldn't find any.

Do you mean dataset_integrity and peers? These are passes that will need to be implemented. But this is similar in structure to what we have for scorers/extractors for accuracy datasets.

arav-agarwal2 · 2026-08-31T14:45:11Z

+2. The front end will perform initial validation on the submission metadata to check valid cohort, benchmark membership, and verify repository versions match the remote tags for that cohort.
+3. Once validated, the specification is inlined into the yaml file to reveal an expanded configuration file which specifies exactly what needs to be executed. This ensures that the benchmark execution is correct by construction. The generated yaml file will be saved as part of the output artifacts.
+4. The generated yaml is passed on to the execution phases which will run the phases in order and produce associated artifacts. For instance, the dataloader will produce an implementation specific SHA for the data loaded which will be logged.
+5. Towards the end of the execution of the benchmark phases, post-processing step will apply the checks needed to verify submission integrity:


Is post-processing something that is done by the submission checker as created by MLC, or is this done by loadgen? If the former, we need rules added to the rules repo so everyone knows what checks are being done on submissions.

Both - the specification should be sufficient to have both independently implement a correct checker and agree on the outcomes. If they don't we need to either identify the buggy implementation and fix it or harden the specification.

arav-agarwal2 · 2026-08-31T16:40:24Z

Is this intended for 1.0? If so, this needs to be far more detailed in terms of checks and overall schema for each of the files, and there needs to be accompanying rules PRs that another organization agrees with.

arekay-nv · 2026-08-31T17:36:50Z

Is this intended for 1.0? If so, this needs to be far more detailed in terms of checks and overall schema for each of the files, and there needs to be accompanying rules PRs that another organization agrees with.

Yes, this is intended for 1.0. But this is purely an implementation change, nothing from the rules needs to change. The goal is to 1) centralize all the benchmark specification - right now we do not have a single source of truth for benchmarks in a given round/cohort 2) ensure that we can capture all aspects of what the existing rules specify to be automated and 3) enable us to add new benchmarks rapidly by updating the specification.

There are two sides to the verification of a submission - client side (performed before submission) and MLC side (after submission). Both should use the same yaml configs but should be independent so that we can more robust and identify any disambiguation.

This change is designed not have any rules PR and will be an implementation only change unless we find gaps in the specification and we can then file issues/PRs for rules.

arekay-nv · 2026-08-31T17:37:48Z

I don't think I can approve it just yet as it's a proposal, but my main worry is that I don't see a policies PR suggesting the changes to the submission directory and submission checker rules for this system.

This shouldn't affect the submission directory structure or the submission checker rules. Those should remain the same.

anandhu-eng · 2026-09-02T15:00:06Z

+
+### Folder structure:
+
+The `type: submission` entry in the yaml file above directs the front-end to load the submission details from the benchmark suite which is stored in the following folder structure:


Does the benchmark suite come along with the current endpoints repo, or are we planning it as a seperate rpository?

It will be part of the endpoint repo. That would enable us to snapshot the benchmark specification for a cohort by tagging a branch and there are no external dependencies to reproduce a run.

nv-alicheng · 2026-09-02T18:49:07Z

+```
+benchmark_suite/
+├── suite.yaml             version; the loader globs the three directories below
+├── models/<id>.yaml       id, reference_id, tokenizer?, processor?, chat_template?,


What is 'processor' here?

nv-alicheng · 2026-09-02T18:51:55Z

+      datasets:
+        - dataset: aime25
+          evaluation_recipe: mlperf_exact_match
+          generation: # sparse merge over the phase generation block


I'm iffy about sparse merging - it's bitten us in the past when doing sparse merges on configs.

nv-alicheng · 2026-09-02T18:52:19Z

+    postprocessing:
+      gates:
+        [
+          dataset_integrity,


Where would these gates be implemented?

As post-processing checks in the client for submissions.

nv-alicheng · 2026-09-02T18:53:31Z

+        top_k: 1
+        max_new_tokens: 20000
+      dataset_aggregation: { method: single_dataset }
+      thresholds:


Where do the keys in this block come from? Can we have a list of valid thresholds? When are these threshold checked and by what?

Right now they are just understood to be there - but we can expose those from the metrics implementation and use a consistent naming scheme across them. The metrics recorder would have all the computed metrics available to the checkers which can express those constraints.

nv-alicheng · 2026-09-02T18:55:00Z

+├── models/<id>.yaml       id, reference_id, tokenizer?, processor?, chat_template?,
+│                          metadata{}
+├── datasets/<id>.yaml     id, description, samples, source{url, sha256},
+│                          prompts{}, ground_truth_or_tests{}, evaluation_recipes{}?,


ground_truth_or_tests{} can be misinterpreted - can you do a similar breakdown with descriptions of each field like you did below with the benchmarks yaml?

nv-alicheng · 2026-09-02T18:57:09Z

+    samples: 30
+    source: { url: <TBD>, sha256: <TBD> }
+    prompts: { column: question }
+    ground_truth_or_tests: { column: answer }


Based on the example here, this is a dict with either the key 'column' or 'tests', which is never explicitly stated anywhere, and the name of the value implies either this is a state enum, or that the keys will be 'ground_truth' or 'tests'. Can you rename and add some docs here?

nv-alicheng · 2026-09-02T18:58:16Z

+    reference_id: openai/gpt-oss-120b
+
+# DATASETS
+datasets:


I'm curious, how would an agentic dataset be written in this yaml? Would you need to differentiate between turns and trajectories?

The way it is currently implemented, loadgen doesn't need to deal in turns. The agentic issuer is given trajectories and it would handle the issuing of turns for each trajectory before moving on.

nvzhihanj · 2026-09-02T21:04:28Z

+```yaml
+rulesets/2026-09-C0.yaml
+  cohort: 2026-09-C0                      # submission round; YYYY-MM-C0 (1st Wed) / -C1 (3rd Wed)
+  benchmark_suite_version: "1.0"


Is this 1.0 corresponds to endpoints 1.0?
Is the suite supposed to change dynamically or stay the same in 1-2 cohorts? (Asking mainly to clarify what suite.yaml specifies)

No, this is the suite schema version. Probably over-complicating things for now and we can assume there is a single version-less schema. Will take out.

Signed-off-by: Rashid Kaleem <230885705+arekay-nv@users.noreply.github.com>

Initial proposal

df8b5bf

Signed-off-by: Rashid Kaleem <230885705+arekay-nv@users.noreply.github.com>

arekay-nv requested a review from a team August 25, 2026 21:37

github-actions Bot requested a review from nvzhihanj August 25, 2026 21:37

github-actions Bot added the size/normal PR Review Policy: <=500 non-test lines & <=20 files label Aug 25, 2026

arav-agarwal2 reviewed Aug 31, 2026

View reviewed changes

anandhu-eng reviewed Sep 2, 2026

View reviewed changes

nv-alicheng reviewed Sep 2, 2026

View reviewed changes

nvzhihanj reviewed Sep 2, 2026

View reviewed changes

Sample model specs

3bd1d0b

Signed-off-by: Rashid Kaleem <230885705+arekay-nv@users.noreply.github.com>


		### Folder structure:

		The `type: submission` entry in the yaml file above directs the front-end to load the submission details from the benchmark suite which is stored in the following folder structure:

Conversation

arekay-nv commented Aug 25, 2026

What does this PR do?

Type of change

Related issues

Testing

Checklist

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

arav-agarwal2 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arav-agarwal2 commented Aug 31, 2026

Uh oh!

arekay-nv commented Aug 31, 2026

Uh oh!

arekay-nv commented Aug 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment