Conversation
The Pyxis runtime builds each container step's environment from an explicit allow-list, and two variables that srun and enroot genuinely need were missing. Both failures are invisible in the run: they happen inside a subprocess whose only report is the generic "Pyxis infrastructure failure before the command completed". SLURM_CONF. Without it the child srun falls back to /etc/slurm/slurm.conf. On a configless site that file does not exist and srun aborts with "Could not establish a configuration source"; on a multi-cluster site it exists but is a different cluster's file whose plugins are not installed locally, and srun aborts with "failed to initialize cli_filter plugin". Either way every step dies before a container is created. The remaining SLURM_* variables stay out of the allow-list deliberately: inheriting SLURM_JOB_ID / SLURM_STEP_ID is exactly what breaks a nested srun, which is why the allow-list exists. Proxy policy. enroot performs the registry pull inside the step, so it needs the same proxy configuration as the caller. A site that pins a container-cache proxy system-wide will 403 the CONNECT for any registry outside that cache's allow-list, and every per-instance image import fails with "curl: (56) CONNECT tunnel failed, response 403" -- including the SWE-bench task images this runtime is built to pull. Verified on a GB200 cluster whose enroot pins a container cache: before the change no sweb.eval.arm64 image could be imported from any node; after it, the image imports and the container starts. Tests: `test_pyxis_srun_environment_forwards_config_and_proxy_policy` asserts each of the seven newly allowed variables reaches the step (all seven fail against the previous allow-list), and `test_pyxis_srun_environment_withholds_inherited_step_identity` pins the other half of the contract -- SLURM step identity is still withheld -- so a later "just forward all SLURM_*" cannot pass unnoticed. The service README documents the allow-list and why those two entries are on it.
…he step Pyxis creates the container *inside* the `srun` step, so Enroot reads `ENROOT_TEMP_PATH` and `ENROOT_CONFIG_PATH` there and not in the service process. Neither was on the step environment allow-list, so both were silently dropped. The consequence is not a failed step, which is why it took so long to see: an operator points `ENROOT_TEMP_PATH` at a large device precisely so that unpacking a ~2.5 GB rootfs with ~16.8k hardlinks does not compete for space with the unpacked rootfs itself, the override never arrives, and the create-time temp lands back on the very device it was meant to spare. On a 20-node run this is how `/raid` reached 4.1 GB free of 527 GB, after which every subsequent container creation failed for want of space -- reported as an ordinary infrastructure failure with no mention of the setting that was discarded. This is the same class as the proxy variables already on the list and for the same structural reason: work that looks like it happens in the service actually happens in the step, and configuration that does not cross that boundary is configuration that does nothing. No other `SLURM_*` variable is added; inheriting `SLURM_JOB_ID` / `SLURM_STEP_ID` is what breaks a nested `srun` and is why the allow-list exists. Kept deliberately separate from the `SLURM_CONF` + proxy commit rather than folded into it. That commit is also PR mlcommons#452 upstream; if mlcommons#452 merges on its own and the stack drops its commit, this fix has to survive that, and it only does if it stands alone. Tests: the existing allow-list parametrisation gains both variables (each fails against the previous list), and `test_pyxis_srun_environment_withholds_inherited_step_identity` continues to pin the other half of the contract. The service README documents the allow-list as a table with the reason each load-bearing entry is on it.
…reated `PyxisEnvironment.cleanup()` never reclaimed anything. Pyxis namespaces named containers by the allocation, so `--container-name=X` inside job `N` is the Enroot container `pyxis_N_X`. cleanup() asked for `pyxis_X`, which does not exist. `enroot remove` exited non-zero, and `check=False` with `capture_output=True` discarded both the status and the message, so the failure was invisible. Symptom: nothing is reclaimed for the life of an allocation. Measured on a 20-node run -- 199 trajectory rootfs coexisting on one node and `/raid` down to 4.1 GB free of 527 GB, after which every subsequent container creation failed for want of space. It is also the origin of the "scancel doesn't reap enroot containers" folklore: `scancel` genuinely does not remove Enroot containers, but the containers here were never asked to go away in the first place, so the blame landed on SLURM. Two changes: * `enroot_container_name(job_id, name)` builds the name Pyxis actually created, and cleanup() uses it. * A non-zero `enroot remove` is logged with its stderr instead of being swallowed, so the next time this path breaks it says so. Tests: `test_pyxis_cleanup_removes_the_container_pyxis_actually_created` models an Enroot container set and asserts the created container is the one removed (it fails against the old name, which removes nothing); `test_pyxis_cleanup_reports_a_removal_that_did_not_happen` asserts the warning; `test_enroot_container_name_is_namespaced_by_job` pins the naming rule. The existing container-reuse test asserted the unnamespaced form and is corrected.
…ork queue Adds the two foundations of the distributed SWE-bench harness: - units.py: shards an instance-id list into immutable, content-addressed units. The sha256 digest covers the ordered id list, so a plan cannot be silently reused across a different run, instance list, or ordering. - queue.py: a filesystem work queue whose claim is a bare os.mkdir (never makedirs(exist_ok=True), which hands a unit to every caller). available() is plan - claims - results, so deleting a result alone does NOT requeue a unit; requeue() is the only supported path and removes the result, the claim and the attempt records together. Env faults are ledgered separately from counted attempts, and abandoning a unit publishes a terminal result AND releases the claim so claims/ and results/ never disagree.
merge_run(wq, run_id) refuses to emit an accuracy number unless every planned unit has a terminal result, none is abandoned, every unit accounts for exactly its planned instance IDS (a set comparison, never a count), the union equals the plan with no cross-shard duplicates, every plan_digest matches, and no unit carries an infra error. Refusal is a structured MergeRefusal naming the offending units and ids; there is no force flag and no partial-credit path. There is deliberately no --all: merge_run takes a required run id and treats a foreign run id or digest as a hard error, not a skip. verify_inventory() cross-checks claims, results and the id-union as independent producers, so a blind spot shared by one instrument cannot certify itself.
The merge gate refuses a bad run, and that refusal is the most important property here. But a refusal that carries no numbers is not the end of the story: somebody still has to report *something*, and with the gate silent they compute it by hand from the artifacts -- which is exactly how a run that lost 106 of 200 instances to infrastructure came to be reported as 47.0% and compared against a complete-run reference of 70.67%. It was read as a model regression. It was attrition. `assess_run()` performs the whole of the gate's arithmetic without deciding anything, and returns a `CompletenessReport`. `merge_run()` becomes the strict all-or-nothing wrapper over it and attaches the report to both `MergeResult` and `MergeRefusal`, so a caller never has to choose between "a number" and "no information". `resolved_rate` is published only when the run is *structurally complete* -- every planned instance id accounted for exactly once -- **and** zero instances were lost to infrastructure. These are two different questions and conflating them gets both wrong. An instance the model attempted and failed is a legitimate score; one our own harness dropped never had the chance. A run short of instances has the wrong denominator; a complete run that leaned on the infrastructure has the wrong provenance. Two numbers are published either way: * `conditional_resolved_rate` -- resolved over the instances that actually completed. Honest about what it measures and not comparable to a complete-run reference. * `resolved_rate_lower_bound` -- resolved over everything planned. Infrastructure losses can only ever *add* resolutions, so this bounds the truth from below even on a badly degraded run. alongside `incomplete_instance_ids`, `infra_lost_instances`, `infra_lost_unit_ids` and a `resolved_rate_withheld_reason` that says which of the two conditions failed and by how much. Ported from the banked campaign's `wq_merge.sh:7-9`: "shard_merge.py refuses to print an accuracy unless all 20 shards account for exactly their own 10 ids, and that refusal is the single most important property in this campaign." What is added here is that the refusal now shows its working. Tests: `TestCompletenessGate` covers both decision boundaries -- complete vs incomplete, and infra-lost vs genuinely-empty (a model that resolved nothing is a score, not a casualty) -- plus an abandoned unit counting as infrastructure loss, the numbers surviving a refusal, and `assess_run` not raising on the run it is describing. Against the parent commit a refusal carries no `report` and no conditional or lower-bound figure at all.
An accuracy run could complete every unit of work, exit 0, and report `N/A`. `finalize_benchmark()` scored, wrote `accuracy_results.json` with `score: null`, printed the summary, and returned -- so `main.py` exited 0 and every wrapper downstream read the run as a pass. This is not hypothetical. A distributed SWE-bench run drove all 20 of its units to terminal records; 17 were abandoned to infrastructure failures, the merge gate correctly refused, `score()` returned None -- and the sbatch wrapper wrote `disposition=run completed (driver rc=0, results=20/20)` over a run with no accuracy number at all. rc=0, all work "done", no number is the failure shape that costs whole GPU allocations, so make it loud where it originates instead of asking each caller to notice. `_require_accuracy_numbers()` runs last, after every artifact is on disk, so the failure never costs the evidence needed to diagnose it. A real number flagged `complete=False` (a partial headline) still passes: the number exists and the entry already says it is partial. A PERF-mode run owes nothing for an externally-scored dataset it never dispatched. Also count what was evaluated, not what was loaded, in the accuracy-only summary line: SWE-bench Verified loads all 500 rows and scores `num_instances` of them, so the run above printed "500 samples evaluated" directly beneath its own "unit=200" headline.
Under Pyxis, creating the container is its own piece of infrastructure work: `--container-image` makes enroot import a multi-GB SWE-bench image and slurmstepd launch a step for it. That was charged against `environment.timeout` -- the per-*command* budget, 300s in both templates, sized for `pytest`-scale work inside an already-running container -- because `PyxisSweBenchRunner._configure_environment` dropped the template's `pull_timeout: 3600` as a docker-only key and `PyxisEnvironment.__init__` had nothing else to use. A create budget must be separate from a per-command budget because the two scale with completely different things. A command's cost depends on the task; a create's cost depends on how much other work is contending for the node. Measured on an idle node, one create is ~35s and eight concurrent creates finish in 55s wall -- so 300s looks generous right up until it isn't. In the run that exposed this, four SWE-bench services drove 40 concurrent agents across 5,148 srun steps in 78 minutes, every step requesting all 144 CPUs, on a node also running four vLLM engines. Creation slowed by an order of magnitude, `subprocess.run(timeout=timeout_s + 30)` SIGKILLed the step, and 96 steps died at a uniform 5m47s-5m56s -- 330s plus step-accounting skew, against 3-11s for every ordinary command step. 17 of 20 units were lost and the run produced no accuracy number at all. The registry was never the bottleneck; step contention was. Carry `pull_timeout` through to a distinct `create_timeout_s` (default 3600) and use it for the create step only. Command steps keep `timeout`. Also stop discarding srun's own output. Both infrastructure-failure paths in `run_srun_step()` raised a fixed string and threw away the captured stream, so an import failure, an out-of-space enroot, and a step that never got resources were one indistinguishable message -- the 17 lost units above could not be told apart from their artifacts. The failure now carries srun's last 2000 characters, names the deadline it blew, and reports srun's exit code. Finally, make creation measurable while it happens rather than only afterwards in `sacct`: with `SWEBENCH_PYXIS_CREATE_TIMING_PATH` set, each create appends one JSONL record with its duration and outcome. Off by default, and a sink that cannot be written degrades to nothing -- a create that succeeded and could not be logged is still a create that succeeded.
Builds on "give Pyxis container creation its own deadline", which attached srun's own output to these failures and made them *readable*. They are still not *actionable*: nothing in the text distinguishes "the step never launched" from "the command ran and its report was lost", and only the first can be retried without risking double execution. Measured signature, from an isolated probe with no model and no GPU (20 nodes, 200 workers, 6273 ordinary shell steps): 63 steps failed, and in all 63 the status file still read `pending` -- not `started` -- with srun's output empty. The step script never ran its first line. The command provably did not execute, so re-running it cannot double-apply an edit, a removal or a test run. That is the entire safety argument, and it is only available if the status bytes are captured rather than compared and thrown away. Two changes: * `StepNotLaunched(RunnerError)` records `srun_rc`, the observed `status`, and `provable_non_execution` (`status == "pending"` and no sentinel). It subclasses `RunnerError`, so every existing `except RunnerError` is unaffected, and the status bytes now appear in the message too. * An in-band sentinel, `__MLPERF_STEP_RC__ <nonce> <rc>`, becomes the primary result channel. It travels on srun's stdout, so a step reports its outcome without depending on a readable shared filesystem -- a real failure mode of its own on a distributed one -- and it is what makes "no sentinel" half of the provability test. The nonce makes it unforgeable by the command's own output; it is stripped before the output is returned. This deliberately stops at reporting. Nothing here retries. Tests: `test_step_failure_reports_whether_non_execution_is_provable` covers the three decision points (pending / started / finished), `test_step_reports_its_return_code_in_band`, `test_step_sentinel_cannot_be_forged_by_command_output`, `test_read_step_sentinel_ignores_unrelated_output` and `test_step_not_launched_is_a_runner_error`. Against the parent commit the raised error has no `provable_non_execution`, no `srun_rc`, no `status`, and does not quote the status bytes.
reaper.py releases a stale claim only when it has no result, its heartbeat is past stale_after, AND its owner is provably gone. Liveness is a pluggable protocol: LocalProcessLiveness pairs pid with boot id so a recycled pid on a rebooted host is not read as a live owner, and SlurmStepLiveness treats a step missing from scontrol inside a live job as dead, because the job-level rule alone deadlocks the queue forever. An indeterminate probe releases NOTHING - a false reap creates two owners, duplicate results and a wrong denominator. guards.py kills a runaway graded test only under a full conjunction (RSS over threshold AND cwd inside the testbed AND a container-supervisor ancestor). Kills are by PID and refuse self and any ancestor of self; there is no pattern-kill path in the module at all, and a test greps the source to keep it that way. Each term reports its evidence count, and HealthVerdict.combine returns INDETERMINATE rather than UNHEALTHY when a term has zero evidence, so a conjunctive guard cannot collapse into its weakest clause.
… is proved The reaper returns a claim whose owner is provably gone. This is the same argument one level down: an operation that provably never ran can be run again, and one that may have run cannot. Measured signature, from an isolated probe with no model and no GPU (20 nodes, 200 workers, 6273 ordinary shell steps): 63 steps failed, and in all 63 the step's status file still read `pending` -- not `started` -- with no in-band sentinel. The step script never executed its first line. Re-running those commands cannot double-apply an edit, a removal or a test run. That is the entire safety argument, and it is why the gate is `provable_non_execution` rather than "an error happened". A failure that does not make that claim is re-raised immediately and does not consume the budget, which also makes every exception type this module has never heard of safe by default. `infra_retry` provides three things: * `retry_on_provable_non_execution(...)` -- bounded attempts, and the gate. The evidence is read as an attribute rather than an isinstance check, so the producer of the evidence and this consumer stay decoupled. * `InfraRetryLedger` -- every attempt and its outcome, appended as JSONL so a run that dies still leaves its retry history behind, and held in memory so the counters survive a ledger that cannot be written. Accounting must never be able to take a run down. * `summary()` -- `infra_retries_total`, `instances_saved_by_retry`, `infra_retries_exhausted`, the succeeded-on-attempt distribution, and `run_quality: CLEAN | OK_WITH_RETRIES | DEGRADED`. The counting is the point. The banked campaign retried environment faults without limit and without counting them (`wq_worker.sh:41` WQ_MAX_ATTEMPTS=5, `:256` "ENVIRONMENT FAULTS DO NOT CONSUME THE UNIT'S ATTEMPT BUDGET"), which is exactly why nobody knew how many there had been. A retry loop that quietly absorbs the defect it compensates for turns a broken cluster into an invisible one. Measured effect of adding this loop: RunnerError 59 -> 7 and resolve 47.0% -> 70.0% against a banked 70.67% on the identical 200 instances -- a rescue on that scale is not a clean run, and `run_quality` says so even at 200/200. This is the fleet-side half: the decision rule, the accounting and the quality verdict. The next commit applies the same rule inside the SWE-bench service, where the Pyxis step that produces `provable_non_execution` runs. Tests: `TestTheSafetyGate` covers both sides of the decision boundary (`pending` retried, `started` never retried, unfamiliar exception never retried) plus the bound; `TestAccounting` covers recovery, exhaustion, not-retryable, ledger durability and a ledger that cannot be written; `TestRunQuality` covers all three verdicts including DEGRADED on volume alone, where every operation eventually succeeded.
…aunched Wires the retry decision into the place the failure actually happens. `run_srun_step()` now re-attempts a step only when `StepNotLaunched` reports `provable_non_execution` -- the status file still `pending` and no in-band sentinel, so the step script did not run even its first line and the command definitely did not execute. A `StepNotLaunched` that reached `started`, and every other failure, is raised immediately: re-running work that may already have run can apply an edit twice, delete twice, or double a test run, and none of those announce themselves. Measured signature, from an isolated probe with no model and no GPU (20 nodes, 200 workers, 6273 ordinary shell steps): 63 steps failed and in all 63 the status file still read `pending`. Measured effect of retrying exactly those: `RunnerError` 59 -> 7 and resolve 47.0% -> 70.0% against a banked 70.67% on the identical 200 instances. Bounded by `SWEBENCH_PYXIS_STEP_RETRIES` (default 3, set 1 to disable), and every attempt and outcome is appended to `SWEBENCH_PYXIS_INFRA_RETRY_LOG` when set. The record shape is deliberately identical to `swe_bench_distributed.infra_retry.RetryRecord`, which gains `InfraRetryLedger.from_jsonl()` to read it back and publish `infra_retries_total`, `instances_saved_by_retry`, `infra_retries_exhausted` and `run_quality`. The service is an isolated subproject and must not import the benchmark client, so the two halves share a file format rather than a module -- and a test on each side pins that agreement, because if it breaks the retries stop reaching the run-level quality flag and a rescued run looks clean. A retry loop that quietly absorbs the defect it compensates for turns a broken cluster into an invisible one. That is why the accounting is not optional and why `run_quality` reports DEGRADED on volume alone. Tests: `TestStepRetry` covers both sides of the boundary (`pending` retried, `started` never retried), the bound, the recorded outcomes for recovery and exhaustion, and a log path that cannot be written. `TestReadingBackAWritten Ledger` covers the cross-process format, including a truncated final line from a run that died. An autouse fixture pins the existing single-shot tests to one attempt so they keep asserting single-shot behaviour.
…t behind One worker's infrastructure failure discarded the entire eval phase. The agent phase runs `--workers` trajectories concurrently. `_run_agent` re-raises whatever the phase raised, so a single worker that could not start its container -- or whose `srun` step never launched -- took the exception all the way out of `_run()`. That happens *before* `preds.json` is ever looked at, so the predictions every other worker had already written were never scored. Observed: a 200-instance run with 137 predictions on disk reported as a total loss, exit non-zero, no accuracy number, and had to be re-scored by hand from the retained artifacts. The GPU allocation that produced those 137 predictions was gone by then. Eval is now robust to individual worker failure: whatever predictions exist are always scored. The failure is not hidden -- * it is written to the new `agent_phase_error.txt` run artifact, with secrets redacted, and served through the existing artifact route; * it is logged at ERROR; * it is chained as `__cause__` onto the `preds.json` failure when the phase genuinely produced nothing, so an empty run still fails loudly. `RunCancelled` is explicitly not tolerated: a cancelled run is not a degraded run and must not proceed to eval. Tests: `test_run_scores_predictions_left_behind_by_a_failed_agent_phase` (the eval phase runs and the artifact is written), `test_run_still_fails_when_the_agent_phase_produced_nothing` (no false pass, cause chained), `test_run_redacts_secrets_from_the_agent_phase_error`, `test_run_does_not_tolerate_cancellation` and `test_agent_phase_error_is_a_retrievable_artifact`. The existing cleanup-after-failure test asserted that the agent error propagated verbatim and is updated to assert the chained failure instead.
The Pyxis sentinel only covers the agent phase; eval-phase error_ids were counted as real outcomes and never retried, which is what produced 24 of 25 permanently-bad runs on the source cluster. classify.py reads the SWE-bench report's error_ids and each instance's run_instance.log and classifies them through an ORDERED rule list, first match wins. The order is load-bearing: BuildImageError is checked before everything because its message embeds the other rules' needles, CONMON_EAGAIN and TEST_TIMEOUT precede WEDGE_EVAL, and PATCH_APPLY_FAILED is last. Anything unclassifiable is UNKNOWN and UNKNOWN is GENUINE, asserted by a membership test: a false bad-run costs one redo, a false retry biases the measurement toward optimism. Memory-kill markers are consumed by phase - an eval-phase kill is a genuine failure (an unbounded allocation is a failing patch), an agent-phase kill is recorded for audit only, since the agent merely gets an error observation and the instance still reaches a real outcome.
…hole report The same defect as the agent phase, one phase later. `pyxis_worker` grades each prediction in its own container concurrently, collected the per-instance failures, and then raised `RunnerError` *before* `make_run_report()`. So one wedged evaluation container threw away every other instance's grade -- the work was done, the reports were on disk, and nothing was ever written. The eval phase is now robust to individual instance failure. Whatever was graded is reported: an instance with no `report.json` is counted as an error by `make_run_report`, which is the correct and visible outcome, and is exactly what an operator needs to see. A run in which *no* instance could be evaluated still fails -- but only after the report has been written, so the run can be diagnosed from its own artifacts rather than from nothing. The losses are not hidden and, more importantly, not left as prose in a log: `eval_infra_failures.txt` lists `instance_id<TAB>error` per lost instance, is copied beside `swe_bench_results.json` and is served through the existing artifact route. That distinction is load bearing. An instance the harness dropped is not an instance the model failed, and a consumer that cannot separate them reads attrition as an accuracy regression -- which is precisely the misreport the completeness gate exists to prevent. Leaving the evidence only in a human-readable log would leave the gate unable to see it. Found while auditing which cluster-side runtime patches the package had made redundant: this one had not been, and it would have compromised the very run intended to validate the package. Tests: `test_pyxis_worker_reports_the_instances_one_bad_container_did_not_kill` (the report is produced, only the failed instance is listed), `test_pyxis_worker_still_fails_when_no_instance_could_be_evaluated` (no false pass, and the report is written first), `test_pyxis_worker_records_no_failure_file_for_a_clean_eval`, `test_eval_infra_failures_are_published_beside_the_results` and `test_eval_infra_failures_is_a_retrievable_artifact`. Four fail against the parent commit. The existing propagation test still holds: its single instance is also every instance.
A run against a remote engine with no endpoint credential makes zero
progress and never ends.
`SweBenchRunner._base_env()` auto-filled `OPENAI_API_KEY="EMPTY"` only when
the endpoint hostname was `localhost`, `127.0.0.1` or `::1`. For any other
host with `endpoint_api_key` unset it did the opposite: it *removed* the
variable. litellm then refuses to build the request locally --
litellm.AuthenticationError: Missing credentials
-- mini-swe-agent classifies that as transient and retries it every 60s,
forever. Not one request reaches the engine, nothing is logged at ERROR,
the agent processes stay alive, and the run neither progresses nor
terminates. Observed on a 20-node run against a remote GB300 engine: 200
workers, 0 requests served, no failure surfaced.
The hostname gate is the defect. An unauthenticated OpenAI-compatible
server ignores the credential value whether it is reached over loopback or
over the network, so the placeholder is correct in both cases and the
distinction only ever suppressed it where it was needed most.
Replace the pop with the placeholder. The security property that motivated
the pop is kept and made explicit: an ambient `OPENAI_API_KEY` inherited
from the service host is still never forwarded to the endpoint -- it is
overwritten rather than deleted.
Tests: `test_base_env_always_supplies_a_credential_placeholder` covers
loopback and remote hosts; the existing
`test_base_env_supplies_api_key_only_to_agent_subprocess` encoded the old
behaviour and is corrected to assert the placeholder while still proving a
configured key wins and an ambient key never leaks.
run_gates() calls assert_scale() before check() and treats GateScaleError as a gate FAILURE, never a skip. This is the code-level form of the most expensive lesson available: a tool-call gate that exercised the right operation at a 278-token prompt passed, while prompts over 2k tokens silently returned empty, and the run scored 0/80. - CheckpointIdentityGate probes /get_model_info then /v1/models and compares the served model path with == , never startswith or in: the bf16 path is a strict prefix of the fp8 path, so any substring test passes an FP8 engine as bf16. Unidentifiable or ambiguous endpoints fail closed. - ToolCallGate requires a well-formed bash tool call at a prompt of at least min_prompt_tokens measured with the server's own /tokenize, not estimated from characters. No tokenizer means the gate cannot prove its scale, so it fails. - EndpointFingerprintGate records a per-endpoint identity the dispatcher re-checks at publish time, so an engine restarted under a live client cannot yield a 0%-accuracy run that still exits rc=0.
EndpointFingerprintGate hashed the whole /v1/models payload. vLLM stamps that response with a request-time `created` field and mints a fresh `permission[].id` on every call, so two reads of one healthy, untouched engine produce two different fingerprints -- four calls, four values. The dispatcher records a fingerprint when a unit is claimed and re-reads it when the unit is published, and treats any difference as `endpoint_changed`: an infrastructure fault, which requeues the unit. With an unstable fingerprint that comparison is always true, so every unit is retried until it exhausts max_attempts, is published as abandoned, and the merge gate refuses the run. The failure costs the full agent and evaluation time of every attempt first, and reports itself as infrastructure damage rather than as a bug here. Hash only the identity-bearing fields by dropping the per-request ones. The gate still fails closed on an endpoint whose identity cannot be read at all, which is the property it exists to provide.
…dpoints
An accuracy-only run that lists more than one endpoint fails during setup:
Failed to connect to endpoint: 1 validation error for HTTPClientConfig
Value error, num_workers (1) must be a multiple of the number of
endpoint URLs (4) ... Got remainder 1.
and exits 3 before any dataset is planned or scored.
Two forced choices collide. setup_benchmark pins num_workers=1 and
max_connections=1 for every TestMode.ACC run, deliberately, so the compliance
gate's single_stream assertion holds. HTTPClientConfig separately requires
num_workers to divide the endpoint count so each endpoint gets equal workers.
One worker cannot divide four endpoints, so the run is refused.
What makes this a defect rather than a tight constraint is that the rejected
client does no work. Both SWE-bench scorers set SKIP_ENDPOINT_PHASE, so no
sample is ever issued through it -- the same run logs "Expected samples: 0"
moments earlier. A validator is rejecting a configuration on behalf of a
component that never runs, and it takes the whole run down with it.
Give the idle client a single endpoint when the run will issue nothing, so the
divisibility invariant still means what it says for runs that do issue.
Scorers that fan work out across endpoints themselves read the endpoint list
from the run's config.yaml rather than from this client, so this does not
narrow the run.
The proper fix is to skip building an issuer at all when nothing will be
issued. That requires a null issuer type, because BenchmarkSession takes a
non-None issuer, and is a larger change than this defect warrants on its own;
endpoints[:1] is the narrow form of it, not the intended end state.
Only reachable with more than one endpoint in accuracy-only mode: a
single-endpoint run divides exactly and never surfaces it.
Tests: `TestAccuracyOnlyIdleIssuer` covers the failing case (four endpoints,
one worker, zero samples -- fails against the previous code) plus the two
boundaries it must not disturb: a single-endpoint accuracy run, and an
accuracy run that will actually issue, which still receives every endpoint.
docs/evaluation/DESIGN.md records why the idle issuer exists and how narrowly
this applies.
… fleet
Wires the pieces into a scorer registered as eval_method: swe_bench_fleet.
It is a scheduler in front of the existing SWE-bench service protocol, not a
new runtime: a unit is one RunRequest over a shard, so exact instance
binding, per-instance containers, artifact allow-listing and cancellation
are reused rather than reimplemented.
preflight() runs the gates against the inference endpoints and /health
against every service, raising SetupError before a single instance is
dispatched. score() plans, dispatches with one in-flight run per service,
classifies every unit, requeues any unit with infra_error_count > 0 even
when the service reported succeeded, and takes the accuracy number only
past the merge gate - self.complete comes from the gate, never a count
heuristic.
Stall quarantine verifies effect rather than status: a service that is
/health-OK but has completed no unit within stall_timeout_s is quarantined
and its in-flight unit requeued.
Also adds scripts/swe_bench_wq.py {status,merge,requeue,reap} for
operators. reap is dry-run by default and requeue prints exactly which
result, claim and attempt records it removed, because the failure mode in
the field was an operator believing a delete had requeued something.
score() reads the run's settings back from the report directory's config.yaml,
which yaml.safe_load() returns as plain dictionaries. It then handed that
mapping to SWEBenchScorer._generation_params(), which calls .model_dump() on
it, so the fleet scorer raised
AttributeError: 'dict' object has no attribute 'model_dump'
on every run, after the plan and the work queue had been written but before a
single unit was dispatched.
Re-validate the mapping into ModelParams instead of re-implementing the field
selection here, so the fleet path and the single-service path stay in
agreement about which generation settings are forwarded to the service.
Every unit was submitted with endpoint_urls[:1], so a fleet configured with N engines sent all of its work to the first one and left the other N-1 idle. The comment justified this by noting that the service accepts exactly one endpoint per run and that the fleet's parallelism comes from running many units -- true, but it does not follow that every unit must pick the same one. Two consequences. The obvious one is a throughput ceiling: concurrency is bounded by one engine no matter how much hardware the run was given. The serious one is a measurement hazard -- a single engine's behaviour decides the whole run's accuracy, so one degraded engine is indistinguishable from a degraded model, which is precisely the confusion the endpoint fingerprint exists to prevent. Bind unit -> endpoint by shard index instead. The mapping is deterministic, so a retried unit lands on the endpoint it was originally measured against and stays comparable to its first attempt, and a run with one endpoint behaves exactly as before.
…e retry Also surface the completeness report on the refusal paths that already existed: `SWEBenchFleetScorer` writes it into the run's merge artifacts and `swe_bench_wq.py merge` prints it. A refusal is not an absence of information, and leaving the conditional rate and the lower bound unpublished is what makes somebody recompute a headline by hand from the artifacts -- which is how a run that lost 106 of 200 instances came to be reported as 47.0% accuracy.
github-actions
Bot
added
the
size/very-large
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #485.
Summary
--image-dirand--node-maprouting for generation, evaluation, and cleanupsrunenvironments safely and expose pacing, concurrency, launch grace, and retry settingssrunevidence and stderr, distinguish provable non-launches, and validate response nonces/statusThis PR intentionally contains no AGA account, partition, QOS, hostname, IP, Lustre path, campaign image/model, fixed node count,
/raidpath, or hard-coded site tuning.Validation
281 passedacross SWE-bench scorer/distributed/accuracy tests