feat(cli): added run benchmarks command to execute benchmark instances on PR

[christian-pinto]

Add nexus run benchmarks Command for PR-based Benchmark Execution

Overview

This PR introduces a new nexus run benchmarks command that automatically
discovers and executes benchmark instances from GitHub Pull Requests. The
command identifies benchmarks instances that are added or modified in the PR
ad optionally executes them using the ado cli.
The command integrates with the ADO CLI to
run benchmarks either locally or on remote Ray clusters.

It can be tested against this PR: #128

New Features

1. nexus run benchmarks Command

A new CLI command that:

• Analyzes GitHub PRs to find new or changed benchmark instances
• Automatically checks out PR code if the local repository isn't on the PR
  commit
• Executes benchmarks using the ADO CLI
• Supports both local and remote (Ray cluster) execution
• Outputs structured JSON results with execution details

Command Syntax

nexus run benchmarks --pr <pr_url> [OPTIONS]

Options

• --pr <url> (required): GitHub Pull Request URL
• --remote <path>: Execute on remote Ray cluster using specified context file
• --context <path>: Path to ADO context YAML file (samplestore context)
• --list-only: List benchmarks without executing (dry-run mode)
• --output <path>: Output results to JSON file (default: output.json)

Usage Examples

List benchmarks without executing:

nexus run benchmarks --pr https://github.com/IBM/algorithm-nexus/pull/123 --list-only

Execute benchmarks locally:

nexus run benchmarks --pr https://github.com/IBM/algorithm-nexus/pull/123

Execute on remote Ray cluster:

nexus run benchmarks \
  --pr https://github.com/IBM/algorithm-nexus/pull/123 \
  --remote config/remote-context.yaml \
  --context config/ado-context.yaml

2. Enhanced Output Format

The command generates a structured JSON output file with detailed execution
results:

{
    "instances": [
        {
            "instance_path": "packages/terratorch/models/prithvi/benchmark_instances/test_benchmark",
            "status": "success",
            "message": "Successfully created space space-727742-default and operation raysubmit_j3ZEYpEy8XLCZPSG",
            "space_id": "space-727742-default",
            "operation_id": "raysubmit_j3ZEYpEy8XLCZPSG",
            "ray_job_id": "raysubmit_j3ZEYpEy8XLCZPSG"
        },
        {
            "instance_path": "packages/terratorch/models/terramind_base_fire/benchmark_instances/test_benchmark",
            "status": "success",
            "message": "Successfully created space space-db2681-default and operation raysubmit_CqXY6ypLNkMLuBwD",
            "space_id": "space-db2681-default",
            "operation_id": "raysubmit_CqXY6ypLNkMLuBwD",
            "ray_job_id": "raysubmit_CqXY6ypLNkMLuBwD"
        }
    ],
}

Output Fields:

• instance_path: Path to the benchmark instance
• status: Execution status (success or failed)
• message: Detailed status message
• space_id: ADO discovery space identifier
• operation_id: ADO operation identifier
• ray_job_id: Ray job identifier (for remote execution tracking)
• summary: Aggregate statistics (successful/failed counts)

3. Automatic PR Checkout

The command intelligently handles repository state:

• Checks if local repository is on the PR commit
• If not, automatically clones and checks out PR code to a temporary directory
• Cleans up temporary directories after execution
• Uses GitHub CLI (gh) for seamless PR operations

4. Enhanced Metadata

Operations and spaces now include descriptive names and metadata:

Operation naming: randomwalk-pr{number}-{package}-{model}-{instance}

• Example: randomwalk-pr123-terratorch-prithvi-test_benchmark

Space naming: space-pr{number}-{package}-{model}-{instance}

• Example: space-pr123-terratorch-prithvi-test_benchmark

Metadata includes:

• PR URL
• Instance path
• Descriptive operation/space names

5. Remote Execution Support

When using --remote, the command:

• Automatically detects required benchmark packages from nexus.yaml
• Dynamically updates the remote context to install benchmark dependencies
• Ensures Ray environment has all necessary packages

[DRL-NextGen]

Checks Summary

Last run: 2026-06-11T14:37:24.346Z

Mend Unified Agent vulnerability scan found 18 vulnerabilities:

Severity	Identifier	Package	Details	Fix
❗ Critical	CVE-2025-69872	diskcache-5.6.3-py3-none-any.whl	DiskCache (python-diskcache) through 5.6.3 uses Python pickle for serialization by default. An attac... DiskCache (python-diskcache) through 5.6.3 uses Python pickle for serialization by default. An attacker with write access to the cache directory can achieve arbitrary code execution when a victim application reads from the cache.	Not Available
🔺 High	CVE-2026-44827	diffusers-0.37.1-py3-none-any.whl	Diffusers is the a library for pretrained diffusion models. Prior to 0.38.0, diffusers 0.37.0 allow... Diffusers is the a library for pretrained diffusion models. Prior to 0.38.0, diffusers 0.37.0 allows remote code execution without the trust_remote_code=True safeguard when loading pipelines from Hugging Face Hub repositories. The _resolve_custom_pipeline_and_cls function in pipeline_loading_utils.py performs string interpolation on the custom_pipeline parameter using f"{custom_pipeline}.py". When custom_pipeline is not supplied by the user, it defaults to None, which Python interpolates as the literal string "None.py". If an attacker publishes a Hub repository containing a file named None.py with a class that subclasses DiffusionPipeline, the file is automatically downloaded and executed during a standard DiffusionPipeline.from_pretrained() call with no additional keyword arguments. The trust_remote_code check in DiffusionPipeline.download() is bypassed because it evaluates custom_pipeline is not None as False (since the kwarg was never supplied), while the downstream code path that actually loads the module resolves the None value into a valid filename. An attacker can achieve silent arbitrary code execution by publishing a malicious model repository with a None.py file and a standard-looking model_index.json that references a legitimate pipeline class name, requiring only that a victim calls from_pretrained on the repository. This vulnerability is fixed in 0.38.0.	Upgrade to version diffusers - 0.38.0,diffusers - 0.38.0,https://github.com/huggingface/diffusers.git - v0.38.0
🔺 High	CVE-2026-45804	diffusers-0.37.1-py3-none-any.whl	Background This vulnerability is found in the "diffusers" package - the "transformers"-equivalent li... Background This vulnerability is found in the "diffusers" package - the "transformers"-equivalent library for diffusion models. It is found in the "DiffusionPipeline.from_pretrained" flow, which is used to load a pipeline from the HuggingFace Hub. This function has a "trust_remote_code" guard: if the repository’s "model_index.json" references a custom pipeline class defined in a ".py" file in the repo, the load is blocked unless "trust_remote_code=True" is explicitly passed: ValueError: The repository for attacker/repo contains custom code in pipeline.py which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/attacker/repo/blob/main/pipeline.py. Please pass the argument "trust_remote_code=True" to allow custom code to be run. The vulnerability allows arbitrary code execution through the custom pipeline flow from a Hub repo, with no "custom_pipeline" or "trust_remote_code" kwargs passed. The "from_pretrained" call succeeds and returns a functional pipeline. *** Naive Flow "DiffusionPipeline.from_pretrained" begins by popping all relevant arguments from "kwargs" into local variables, then calls "DiffusionPipeline.download()" to fetch the repo files: pipeline_utils.py:853 cached_folder = cls.download( pretrained_model_name_or_path, ... custom_pipeline=custom_pipeline, trust_remote_code=trust_remote_code, ... ) Inside "download()", "model_index.json" is fetched first as a standalone file via "hf_hub_download": pipeline_utils.py:1636 config_file = hf_hub_download( pretrained_model_name, cls.config_name, ... ) config_dict = cls._dict_from_json_file(config_file) This config is used to detect custom pipeline code and enforce the trust check: pipeline_utils.py:1672 if custom_pipeline is None and isinstance(config_dict["_class_name"], (list, tuple)): custom_pipeline = config_dict["_class_name"][0] load_pipe_from_hub = custom_pipeline is not None and f"{custom_pipeline}.py" in filenames if load_pipe_from_hub and not trust_remote_code: raise ValueError(...) After the check passes, "snapshot_download" then fetches all files and saves them to disk: pipeline_utils.py:1778 cached_folder = snapshot_download( pretrained_model_name, ... revision=revision, allow_patterns=allow_patterns, ... ) Back in "from_pretrained", the config is read a second time from the downloaded snapshot, and"_resolve_custom_pipeline_and_cls" reads the config to re-check if custom code needs to be loaded: pipeline_loading_utils.py:974 def _resolve_custom_pipeline_and_cls(folder, config, custom_pipeline): custom_class_name = None if os.path.isfile(os.path.join(folder, f"{custom_pipeline}.py")): custom_pipeline = os.path.join(folder, f"{custom_pipeline}.py") elif isinstance(config["_class_name"], (list, tuple)) and os.path.isfile( os.path.join(folder, f"{config['_class_name'][0]}.py") ): custom_pipeline = os.path.join(folder, f"{config['_class_name'][0]}.py") custom_class_name = config["_class_name"][1] return custom_pipeline, custom_class_name If the config points to a ".py" file, it is imported. *** The Vulnerability "hf_hub_download" and "snapshot_download" are two independent HTTP calls to the Hub, both resolving the repository’s default branch (if "revision=None") to its current HEAD at call time. There is no atomicity guarantee between them - if the repository is updated between the two calls, they will resolve to different commits and download different content, with no warning displayed to the user. The trust check in "download()" operates on the content fetched by "hf_hub_download" (commit A). The "snapshot_download" call that immediately follows can silently fetch a newer commit (commit B). The config in the newer commit will be the one parsed by "_resolve_custom_pipeline_and_cls". Therefore, it’s possible to introduce remote code into the repo between the two calls, bypassing the trust check. The race window is everything between the two Hub calls inside "download()": pipeline_utils.py:1636 config_file = hf_hub_download(...) # ← sees commit A, trust check passes ... filenames processing, pattern building, pipeline_is_cached check ... ~~~ ATTACKER PUSHES COMMIT B HERE ~~~ pipeline_utils.py:1778 cached_folder = snapshot_download(...) # ← sees commit B, downloads pipeline.py For the exploit, commit A carries a clean config with "_class_name" as a plain string, which causes "load_pipe_from_hub" to be "False" and the trust check to pass. Commit B changes "_class_name" to a list and adds "pipeline.py": Commit A - "model_index.json": { "_class_name": "FluxPipeline", "_diffusers_version": "0.31.0" } Commit B - "model_index.json": { "_class_name": ["pipeline", "FluxPipeline"], "_diffusers_version": "0.31.0" } When "from_pretrained" reads the snapshot after "download()" returns, "config["_class_name"]" is now a list, "pipeline.py" exists on disk (fetched by "snapshot_download"), and "_resolve_custom_pipeline_and_cls" resolves "custom_pipeline" to the local path of that file. "_get_pipeline_class" then imports it - with no trust check at this point in the code. *** PoC 1. Create a Hub repo with commit A’s "model_index.json" (plain string "_class_name"). 2. Run "DiffusionPipeline.from_pretrained("attacker/repo")" with a breakpoint set at "pipeline_utils.py:1778" (the "snapshot_download" call). This is for the window to be large enough to manually respond to it. 3. When execution pauses at the breakpoint, push commit B: update "model_index.json" to use a list "_class_name" and add "pipeline.py". 4. Resume execution. 5. "snapshot_download" fetches commit B; "/tmp/pwned" is written during the subsequent "_get_pipeline_class" call. *** Constraints - Does not apply when "revision" is pinned to a specific commit hash - both Hub calls resolve to the same content. - Does not apply when loading from a local directory. - If all expected files are already present in the local HF cache, "download()" returns early before reaching "snapshot_download" (line 1767 early-return), closing the race window. The exploit therefore requires a first (or forced) download. *** Exploitability The window between the two calls is very short. Local testing resulted in a window of approximately ~0.5 seconds for the attacker to push the change. This is, of course, unfeasible to accomplish for each and every new download. However, given a popular repo with many downloads per day, one may achieve statistical success by changing the repo’s state every once in a while or every few seconds, with some percentage of downloaders falling on the exact window. *** Impact The vulnerability is a silent RCE - it allows arbitrary code to be loaded through the custom pipeline flow from a Hub repo, with no "custom_pipeline" or "trust_remote_code" kwargs. The "from_pretrained" call succeeds and returns a fully functional pipeline.	Upgrade to version diffusers - 0.38.0,diffusers - 0.38.0
🔺 High	CVE-2026-44513	diffusers-0.37.1-py3-none-any.whl	Diffusers is the a library for pretrained diffusion models. Prior to 0.38.0, a trust_remote_code by... Diffusers is the a library for pretrained diffusion models. Prior to 0.38.0, a trust_remote_code bypass in DiffusionPipeline.from_pretrained allows arbitrary remote code execution despite the user passing trust_remote_code=False (or omitting it, which is the default). The vulnerability has three variants, all sharing the same root cause — the trust_remote_code gate was implemented inside DiffusionPipeline.download() rather than at the actual dynamic-module load site, so any code path that bypassed or short-circuited download() also bypassed the security check. DiffusionPipeline.from_pretrained('repoA', custom_pipeline='attacker/repoB', trust_remote_code=False) — the gate evaluated against repoA's file list rather than repoB's, so repoB's pipeline.py was loaded and executed. DiffusionPipeline.from_pretrained('/local/snapshot', custom_pipeline='attacker/repoB', trust_remote_code=False) — the local-path branch never invoked download(), so the gate was never reached and remote code from repoB executed. DiffusionPipeline.from_pretrained('/local/snapshot', trust_remote_code=False) where the snapshot contains custom component files (e.g. unet/my_unet_model.py) referenced from model_index.json — same root cause; the local path skipped download() and custom component code executed. This vulnerability is fixed in 0.38.0.	Upgrade to version diffusers - 0.38.0,diffusers - 0.38.0,https://github.com/huggingface/diffusers.git - v0.38.0
🔺 High	CVE-2026-5241	transformers-4.57.6-py3-none-any.whl	A vulnerability in the LightGlue model loading path of huggingface/transformers version 5.2.0 allows... A vulnerability in the LightGlue model loading path of huggingface/transformers version 5.2.0 allows an attacker-controlled model repository to execute arbitrary code during model initialization. The issue arises because the "trust_remote_code" parameter, intended to prevent remote code execution, is overridden by untrusted serialized configuration data in a nested code path. Specifically, when loading a LightGlue model using "AutoModel.from_pretrained()" with "trust_remote_code=False", the "LightGlueConfig" reads the "trust_remote_code" value from the untrusted "config.json" file and propagates it into nested "AutoConfig.from_pretrained()" calls. This results in the execution of attacker-provided Python modules, even when the victim explicitly disables remote code execution. The vulnerability poses a high risk for environments such as API inference servers, research notebooks, CI/CD pipelines, and model evaluation workers, potentially leading to credential theft, lateral movement, or persistence/backdoor deployment.	Upgrade to version transformers - 5.5.0,transformers - 5.5.0,https://github.com/huggingface/transformers.git - v5.5.0
🔺 High	CVE-2026-4372	transformers-4.57.6-py3-none-any.whl	A critical remote code execution vulnerability exists in all versions of the HuggingFace transformer... A critical remote code execution vulnerability exists in all versions of the HuggingFace transformers library prior to version 5.3.0. The vulnerability allows an attacker to craft a malicious "config.json" file containing the "_attn_implementation_internal" field set to an attacker-controlled HuggingFace Hub repository ID. When a victim loads this model using the standard "AutoModelForCausalLM.from_pretrained()" API, the library downloads and executes arbitrary Python code from the attacker's repository with the victim's full OS privileges. This issue arises due to unfiltered deserialization of configuration attributes, insufficient sanitization of internal fields, and unsandboxed execution of downloaded kernels. The vulnerability bypasses the "trust_remote_code" security mechanism, is invisible to the victim, and exploits the standard documented usage pattern, making it particularly severe. Users are advised to upgrade to version 5.3.0 or later to mitigate this issue.	Upgrade to version transformers - 5.3.0,transformers - 5.3.0,https://github.com/huggingface/transformers.git - v5.3.0
🔺 High	CVE-2025-14920	transformers-4.57.6-py3-none-any.whl	Hugging Face Transformers Perceiver Model Deserialization of Untrusted Data Remote Code Execution Vu... Hugging Face Transformers Perceiver Model Deserialization of Untrusted Data Remote Code Execution Vulnerability. This vulnerability allows remote attackers to execute arbitrary code on affected installations of Hugging Face Transformers. User interaction is required to exploit this vulnerability in that the target must visit a malicious page or open a malicious file. The specific flaw exists within the parsing of model files. The issue results from the lack of proper validation of user-supplied data, which can result in deserialization of untrusted data. An attacker can leverage this vulnerability to execute code in the context of the current user. Was ZDI-CAN-25423.	Not Available
🔷 Medium	CVE-2026-47155	vllm-0.19.1-cp38-abi3-manylinux_2_31_x86_64.whl	Summary vLLM's revision pinning controls do not consistently apply to all artifacts loaded for a mod... Summary vLLM's revision pinning controls do not consistently apply to all artifacts loaded for a model. A deployment that supplies "--revision" or "--code-revision" can still load dynamic code, GGUF files, image processors, retrieval side weights, or same-repository subfolder weights/config from an unpinned/default revision. This is a supply-chain integrity issue for pinned vLLM deployments. Operators can believe they are serving a reviewed model revision while vLLM resolves behavior-affecting nested or sibling artifacts outside that reviewed revision. Details The expected invariant is: «When a vLLM operator supplies a model or code revision pin, every code, config, processor, weight file, side weight, and same-repository subfolder artifact loaded as part of that model should resolve under that pin unless vLLM exposes and enforces a separate explicit pin for that artifact.» Current "main" was verified affected at commit "3795d7acf431980e62e738493f437ae2a51549da". Affected source boundaries: - "vllm/model_executor/models/registry.py:1045-1051" and ":1058-1064" - "_try_resolve_transformers()" passes "revision=model_config.revision" and "trust_remote_code=model_config.trust_remote_code", but omits "code_revision=model_config.code_revision" for external "auto_map" dynamic module imports. - "vllm/model_executor/model_loader/gguf_loader.py:58-60" - The direct-file GGUF form "repo/file.gguf" calls "hf_hub_download(repo_id=repo_id, filename=filename)" without passing "revision". - "vllm/model_executor/models/roberta.py:203-209" - BGE-M3 secondary sparse and ColBERT side weights are declared with "revision=None". - "vllm/model_executor/models/kimi_k25.py:111-114" - Kimi-K2.5 calls "cached_get_image_processor()" without passing "model_config.revision". - "vllm/model_executor/models/kimi_audio.py:92-95" - Kimi-Audio loads Whisper config from the "whisper-large-v3" subfolder without a "revision" argument. - "vllm/model_executor/models/kimi_audio.py:425-430" - Kimi-Audio declares same-repository "whisper-large-v3" secondary weights with "revision=None". - "vllm/model_executor/model_loader/default_loader.py:287-301" - The default loader preserves "model_config.revision" for the primary source, then consumes model-supplied secondary sources as declared. The strongest example is Kimi-Audio: the primary "moonshotai/Kimi-Audio-7B-Instruct" weights preserve the configured model revision, but the same-repository "whisper-large-v3" audio tower config/weights do not. A pinned Kimi-Audio deployment can therefore load the Whisper subfolder outside the audited revision. This report does not claim a "trust_remote_code=False" bypass, unauthenticated RCE, or real artifact compromise. The issue is improper propagation of explicit artifact pins across supported loader paths. Impact Affected users are operators who pin vLLM model deployments to a reviewed Hugging Face revision for safety review, provenance, rollback, or reproducibility. The impact is that the pin does not reliably describe the full set of artifacts vLLM serves. Even when the operator selects an audited revision, vLLM can resolve behavior-affecting secondary artifacts from the repository default branch or another mutable ref. Depending on the model path, the unpinned artifact can be dynamic model code, a GGUF file, an image processor, retrieval side weights, or the same-repository Kimi-Audio Whisper subfolder weights/config. This breaks the operational guarantee of a pinned deployment: "serve the exact artifact set I reviewed." A later change to an unpinned secondary artifact can alter model behavior without changing the operator's configured revision, making review, rollback, incident response, and audit records unreliable. Occurrences - "vllm/model_executor/models/kimi_k25.py" L111-L114 — Kimi-K2.5 loads its image processor with "cached_get_image_processor()" but does not pass "self.ctx.model_config.revision". The processor can therefore resolve from the default repository revision even when the model deployment is pinned. - "vllm/model_executor/models/kimi_audio.py" L425-L430 — Kimi-Audio declares same-repository "whisper-large-v3" secondary weights with "revision=None". A pinned Kimi-Audio deployment can therefore load the Whisper audio tower weights from an unpinned/default revision. - "vllm/model_executor/models/kimi_audio.py" L92-L95 — Kimi-Audio loads Whisper config from the same repository's "whisper-large-v3" subfolder without passing the top-level model revision. The config for this behavior-affecting subcomponent can be resolved outside the audited model revision. - "vllm/model_executor/models/registry.py" L1058-L1064 — The later dynamic model-class resolution repeats the same pin-decay pattern: it forwards "revision" and "trust_remote_code", but omits "code_revision". This means an operator-provided code pin is not enforced at the dynamic module loader boundary. - "vllm/model_executor/model_loader/gguf_loader.py" L58-L60 — The direct GGUF form "repo/file.gguf" calls "hf_hub_download(repo_id=repo_id, filename=filename)" without passing "model_config.revision". A deployment that pins the model revision can therefore resolve this GGUF file from the repository default revision. - "vllm/model_executor/models/registry.py" L1045-L1051 — "try_get_class_from_dynamic_module()" is called for external "auto_map" config/model classes with "revision=model_config.revision", but without forwarding "model_config.code_revision". When "--code-revision" is set, this dynamic module resolution can still fall back to the default code revision instead of the audited code revision. - "vllm/model_executor/models/roberta.py" L203-L209 — "BgeM3EmbeddingModel" creates same-repository secondary sparse/ColBERT weight sources with "revision=None". The primary model revision is not propagated to these side weights, so they can be downloaded outside the operator-selected model revision. Fixes This was fixed in: vllm-project/vllm#42616 *** Originally filed via huntr: https://huntr.com/bounties/3f1e24c0-87d2-4f6c-a705-820f380879ac. The vLLM maintainer (Russell Bryant) redirected the report to the private GHSA channel. Offline proof bundle ("vllm_artifact_pin_decay_bundle_verify.py" + "bundle-verification-20260430T143506Z.json") is available upon request.	vllm - 0.22.0
🔷 Medium	CVE-2026-44222	vllm-0.19.1-cp38-abi3-manylinux_2_31_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). From 0.6.1 to before 0.20.... vLLM is an inference and serving engine for large language models (LLMs). From 0.6.1 to before 0.20.0, there is a a Token Injection vulnerability in vLLM’s multimodal processing. Unauthenticated, text-only prompts that spell special tokens are interpreted as control. Image and video placeholder sequences supplied without matching data cause vLLM to index into empty grids during input-position computation, raising an unhandled IndexError and terminating the worker or degrading availability. Multimodal paths that rely on image_grid_thw/video_grid_thw are affected. This vulnerability is fixed in 0.20.0.	Upgrade to version vllm - 0.20.0
🔷 Medium	CVE-2026-44223	vllm-0.19.1-cp38-abi3-manylinux_2_31_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). From to before 0.20.0, th... vLLM is an inference and serving engine for large language models (LLMs). From to before 0.20.0, the extract_hidden_states speculative decoding proposer in vLLM returns a tensor with an incorrect shape after the first decode step, causing a RuntimeError that crashes the EngineCore process. The crash is triggered when any request in the batch uses sampling penalty parameters (repetition_penalty, frequency_penalty, or presence_penalty). A single request with a penalty parameter (e.g., "repetition_penalty": 1.1) is sufficient to crash the server. This vulnerability is fixed in 0.20.0.	Upgrade to version vllm - 0.20.0
🔷 Medium	CVE-2026-1839	transformers-4.57.6-py3-none-any.whl	A vulnerability in the HuggingFace Transformers library, specifically in the "Trainer" class, allows... A vulnerability in the HuggingFace Transformers library, specifically in the "Trainer" class, allows for arbitrary code execution. The "_load_rng_state()" method in "src/transformers/trainer.py" at line 3059 calls "torch.load()" without the "weights_only=True" parameter. This issue affects all versions of the library supporting "torch>=2.2" when used with PyTorch versions below 2.6, as the "safe_globals()" context manager provides no protection in these versions. An attacker can exploit this vulnerability by supplying a malicious checkpoint file, such as "rng_state.pth", which can execute arbitrary code when loaded. The issue is resolved in version v5.0.0rc3.	Upgrade to version transformers - 5.0.0rc3,https://github.com/huggingface/transformers.git - v5.0.0rc3
🔷 Medium	CVE-2026-34755	vllm-0.18.0-cp38-abi3-manylinux_2_31_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). From 0.7.0 to before 0.19.... vLLM is an inference and serving engine for large language models (LLMs). From 0.7.0 to before 0.19.0, the VideoMediaIO.load_base64() method at vllm/multimodal/media/video.py splits video/jpeg data URLs by comma to extract individual JPEG frames, but does not enforce a frame count limit. The num_frames parameter (default: 32), which is enforced by the load_bytes() code path, is completely bypassed in the video/jpeg base64 path. An attacker can send a single API request containing thousands of comma-separated base64-encoded JPEG frames, causing the server to decode all frames into memory and crash with OOM. This vulnerability is fixed in 0.19.0.	Upgrade to version vllm - 0.19.0,https://github.com/vllm-project/vllm.git - v0.19.0
🔷 Medium	CVE-2026-34756	vllm-0.18.0-cp38-abi3-manylinux_2_31_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). From 0.1.0 to before 0.19.... vLLM is an inference and serving engine for large language models (LLMs). From 0.1.0 to before 0.19.0, a Denial of Service vulnerability exists in the vLLM OpenAI-compatible API server. Due to the lack of an upper bound validation on the n parameter in the ChatCompletionRequest and CompletionRequest Pydantic models, an unauthenticated attacker can send a single HTTP request with an astronomically large n value. This completely blocks the Python asyncio event loop and causes immediate Out-Of-Memory crashes by allocating millions of request object copies in the heap before the request even reaches the scheduling queue. This vulnerability is fixed in 0.19.0.	Upgrade to version https://github.com/vllm-project/vllm.git - v0.19.0,vllm - 0.19.0,vllm - 0.19.0
🔷 Medium	CVE-2026-34753	vllm-0.18.0-cp38-abi3-manylinux_2_31_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). From 0.16.0 to before 0.19... vLLM is an inference and serving engine for large language models (LLMs). From 0.16.0 to before 0.19.0, a server-side request forgery (SSRF) vulnerability in download_bytes_from_url allows any actor who can control batch input JSON to make the vLLM batch runner issue arbitrary HTTP/HTTPS requests from the server, without any URL validation or domain restrictions. This can be used to target internal services (e.g. cloud metadata endpoints or internal HTTP APIs) reachable from the vLLM host. This vulnerability is fixed in 0.19.0.	Upgrade to version vllm - 0.19.0,https://github.com/vllm-project/vllm.git - v0.18.1
🔷 Medium	CVE-2026-7141	vllm-0.18.0-cp38-abi3-manylinux_2_31_x86_64.whl	A vulnerability was found in vllm up to 0.19.0. The affected element is the function has_mamba_layer... A vulnerability was found in vllm up to 0.19.0. The affected element is the function has_mamba_layers of the file vllm/v1/kv_cache_interface.py of the component KV Block Handler. Performing a manipulation results in uninitialized resource. It is possible to initiate the attack remotely. The attack is considered to have high complexity. The exploitability is described as difficult. The exploit has been made public and could be used. The patch is named 1ad67864c0c20f167929e64c875f5c28e1aad9fd. To fix this issue, it is recommended to deploy a patch.	Upgrade to version vllm - 0.19.1,vllm - 0.19.1,https://github.com/vllm-project/vllm.git - v0.19.1
🔷 Medium	CVE-2025-3000	torch-2.10.0-3-cp312-cp312-manylinux_2_28_x86_64.whl	A vulnerability classified as critical has been found in PyTorch 2.6.0. This affects the function to... A vulnerability classified as critical has been found in PyTorch 2.6.0. This affects the function torch.jit.script. The manipulation leads to memory corruption. It is possible to launch the attack on the local host. The exploit has been disclosed to the public and may be used.	Not Available
🔷 Medium	CVE-2026-4538	torch-2.10.0-3-cp312-cp312-manylinux_2_28_x86_64.whl	A vulnerability was identified in PyTorch 2.10.0. The affected element is an unknown function of the... A vulnerability was identified in PyTorch 2.10.0. The affected element is an unknown function of the component pt2 Loading Handler. The manipulation leads to deserialization. The attack can only be performed from a local environment. The exploit is publicly available and might be used. The project was informed of the problem early through a pull request but has not reacted yet.	Not Available
🔸 Low	CVE-2025-63396	torch-2.10.0-3-cp312-cp312-manylinux_2_28_x86_64.whl	An issue was discovered in PyTorch v2.5 and v2.7.1. Omission of profiler.stop() can cause torch.prof... An issue was discovered in PyTorch v2.5 and v2.7.1. Omission of profiler.stop() can cause torch.profiler.profile (PythonTracer) to crash or hang during finalization, leading to a Denial of Service (DoS).	Not Available

[michael-johnston]

@christian-pinto The above docs don't say which benchmarks are run or how this is decided i.e. how the discovery happens.

[christian-pinto]

@christian-pinto The above docs don't say which benchmarks are run or how this is decided i.e. how the discovery happens.

Right, good point. It's only the benchmark instances added or modified with the PR that are executed. Let me add it somewhere in the cli reference documentation.

[christian-pinto]

@christian-pinto The above docs don't say which benchmarks are run or how this is decided i.e. how the discovery happens.

Right, good point. It's only the benchmark instances added or modified with the PR that are executed. Let me add it somewhere in the cli reference documentation.

@michael-johnston it's actually already in the cli reference documentation

### Subcommands

#### nexus run benchmarks

Execute benchmarks from a GitHub Pull Request. This command identifies new or
changed benchmark instances in a PR and optionally executes them using the `ado`
CLI.

It's just here in the PR description that I do not say it.

[michael-johnston]

I'm looking for something about how a change is identified e.g. is it just a change to ado YAML, is it any change to a. model.yaml, is it any new benchmark_instance folder, ...

[AlessandroPomponio]

I wonder if it's best to require ado as part of some dependency group or something and use the pydantic models to create the operations, spaces, etc instead of hardcoding templates that are likely to become obsolete

[AlessandroPomponio]

LGTM thanks