feat(slurm): integrate Slurm execution for 1.0.0 release by nabinchha · Pull Request #1040 · NVIDIA-NeMo/DataDesigner

nabinchha · 2026-10-15T20:56:59Z

📋 Summary

Integrate the optional Slurm execution package into main for the planned 1.0.0 release. This consolidates the feature-branch implementation and documentation in one reviewable PR. Keep this PR open; do not merge until release review is complete.

🔗 Related Issue

N/A — integration of the existing Slurm feature branch and its previously reviewed changes.

🔄 Changes

✨ Added

Add the data-designer-slurm package and the version-matched data-designer[slurm] extra.
Add declarative Slurm profiles, serving image management, run planning and execution, status/cancel/retry/collection, and benchmark workflows.
Add Slurm examples, architecture notes, and user documentation.

🔧 Changed

Extend the CLI and generation invocation boundaries for the optional Slurm integration.
Update the publish workflow to build and upload the config, engine, Slurm, and interface packages at the same version.

🐛 Fixed

Incorporate feature-branch fixes for startup diagnostics, multi-model placement, shard recovery, partial outputs, and Parquet collection schema handling.

🔍 Attention Areas

Package metadata and publish order — confirm the exact-version dependency relationship and 1.0.0rc1 wheel/install flow.
Slurm run contracts and runtime — review public configuration and allocation behavior.

🧪 Testing

make test-slurm — 1,915 passed after rebasing onto main.
make check-slurm, uv lock --check, and uv pip check passed.
Unit and integration-style tests added for the Slurm package.
make test and final 1.0.0rc1 wheel/install validation are pending release review.

✅ Checklist

Follows commit message conventions.
All commits are signed off (DCO): nine existing feature-branch commits lack a Signed-off-by trailer; resolve with their authors before merge if required.
Architecture docs updated.
Release review complete; do not merge yet.

github-actions · 2026-10-15T20:59:38Z

Fern preview: https://nvidia-preview-pr-1040.docs.buildwithfern.com/nemo/datadesigner

Fern previews include the docs-website version archive with PR changes synced into latest. Notebook tutorials are rendered without execution outputs in previews.

* feat: add optional Slurm package Add the shared-namespace leaf package and publish it through the same-version data-designer[slurm] extra. Cover resolver, namespace, base-only isolation, and built-wheel installation behavior.\n\nCloses #852 Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: address Slurm package review findings Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: publish Slurm package Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: scope workspace dependency extras Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: publish Slurm before base package Upload the Slurm distribution before data-designer so the exact-version extra is resolvable as soon as the base package is published. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * docs: generalize Slurm package wording Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* feat: define public generation contract * fix: complete public generation contract * docs: clarify public invocation limits * fix: handle mixed-case config extensions Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: complete generation contract edge cases Signed-off-by: Andre Manoel <amanoel@nvidia.com> * test: make inline config regression self-contained Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: classify malformed config paths Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: normalize generation artifact errors Classify malformed inline configs independently of filename suffixes. Wrap unreadable result artifacts in the public storage error across counting and export paths. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: complete artifact error boundaries Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: tighten generation result contract Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* feat: add lazy CLI extensions Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: isolate CLI extension failures Signed-off-by: Andre Manoel <amanoel@nvidia.com> * refactor: order public CLI members first Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com> Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> Co-authored-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: add Slurm state record contracts Define strict, versioned execution, readiness, output, reconciliation, and scheduler records. Enforce cross-record identity, digest, shard-set, and terminal-state invariants with deterministic golden fixtures and built-wheel coverage. Closes #865 * fix: harden Slurm state contracts Reject partial winners and conflicting shard ownership. Require explicit versions, enforce evidence chronology and readiness precedence, and cover every state-contract path with validated fixtures. Refs #865 * fix(slurm): harden state contract invariants Hash persisted record bytes, align state identities with the resolved plan vocabulary, and preserve public model aliases. Add monotonic attempt and scheduler transition checks, including fixed accounting-lag deadlines and UNKNOWN precedence. Refs #865 * refactor(slurm): share contract primitives Make planning and runtime state consume one public immutable model family. Align shard, attempt, range, artifact, and resume workspace shapes so the configuration-plan branch can rebase without preserving duplicate Pydantic types.\n\nRefs #865

* feat: define Slurm config and plan contracts Add strict authored configuration, profile, image, client, benchmark, dependency-lock, and resolved-plan records. Validate cross-record identities and digests with sanitized single-node and multi-node golden fixtures. Closes #873 Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: tighten Slurm contract validation Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: align Slurm F3 contract seam * fix: bind client dataset to effective resume mode * fix: harden Slurm shared contracts Prevent secret material from entering persisted configuration and make nested contract collections immutable. Tighten client and benchmark semantic invariants with focused negative coverage. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: address Slurm contract review findings Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: close Slurm argument validation bypasses Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: close remaining Slurm contract gaps Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com> Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> Co-authored-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: integrate Slurm plan and state Validate plan-aware shards, readiness, attempts, and finalization through a reusable immutable context. Add deterministic goldens and negative coverage for mismatch and failure paths. Closes #880 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: reject unsubmitted planned attempts Require scheduler-bound plan validation to start only after an attempt leaves the created state. Cover created attempts that already carry scheduler metadata and verify the integration validator imports from an isolated Slurm wheel. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: harden Slurm integration join coverage Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: simplify plan state validator Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: remove validator wrapper functions Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* test: add deterministic Slurm fakes Add reusable scheduler, dependency, plugin, serving, clock, and state-transition fixtures for local and CI Slurm integration tests. Keep the harness deterministic and isolated from production lane implementations. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * test: bind Slurm fakes to contracts Promote the merged planning and finalization goldens to package-wide fixtures, exercise dependency doubles with canonical records, and add site-neutral rendered script goldens for downstream lanes. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * test: complete post-F3 Slurm fake coverage Bind rendered scripts and shared fixtures to the merged contracts, tighten scheduler command behavior, and cover retry terminal states and invalid serving restarts. Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* feat: add Slurm image registry foundation Add digest-bound client and serving inspection, atomic workspace-derived registry persistence, and verified alias/path resolution.\n\nPart of #867 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: cover image inspection boundaries Exercise the production environment reader and verify restrictive atomic registry persistence discovered during review.\n\nPart of #867 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: align image registry persistence Persist the package-owned image registry as deterministic safe YAML at the migration plan's canonical path and declare the runtime YAML dependency explicitly.\n\nPart of #867. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden image registry file boundaries Reject symlinked registry and lock files, use explicit UTF-8 persistence, and normalize malformed byte content into the package image error hierarchy.\n\nPart of #867. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: verify Slurm image wheel integration Exercise the public image service from an isolated installed wheel and assert that the wheel metadata carries the runtime PyYAML dependency.\n\nPart of #867. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: verify registry file types before use Treat only a genuinely missing registry as empty and reject non-regular registry or lock nodes before reading, locking, or changing permissions.\n\nPart of #867. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: rollback stale image registrations Reverify image bytes while registry mutation locks remain held and restore the previous snapshot if final verification fails. Part of #867 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix image registry review findings Require Data Designer in client image inspections, verify artifacts before publishing registry updates, harden storage directory boundaries, and normalize recursive YAML failures. * harden image registry contracts - Keep the image-registry foundation internal until the public facade lands\n- Require a complete client package inventory and unambiguous registry schema\n- Close SQSH identity races and exercise publication behavior through owned APIs * strengthen image registry test coverage Exercise persisted semantic invariants and exact-byte hashes. Split record, store, and service coverage by ownership. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * address image registry review feedback Bind registry transactions to verified directory descriptors, centralize immutable image fact comparisons, and verify internal image modules from the installed wheel. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * clarify image test helper names Use explicit get prefixes for value-producing test helpers while preserving action verbs for creation, inspection, and filesystem operations. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * split registry filesystem helpers Move descriptor-bound filesystem operations into a focused internal module while keeping registry transactions and YAML semantics together. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: add Slurm command client and renderer Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: tighten Slurm launcher boundaries Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: use native sacct output fields Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: require expanded Slurm array queries Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: sanitize Slurm command diagnostics Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: reject invalid visible GPU memory Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: hash rendered inputs without manifests Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: make Slurm output decoding stable Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: validate finite command timeouts Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden Slurm boundary inputs Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: reject malformed Slurm GRES lists Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: isolate submitted Slurm environments Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: verify launcher wheel import Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: pin batch host tool path Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: cover Slurm launcher boundaries Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: correlate Slurm query responses Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden Slurm command boundaries Preserve the caller's PATH as the only ambient lookup input for default Slurm commands while continuing to isolate all other environment variables. Normalize oversized numeric scheduler fields into the launcher parse-error boundary. Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: fall back from empty command path Use the platform default search path when the ambient PATH is absent or empty so bare Slurm executables remain resolvable. Part of #868 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: align Slurm launcher semantics * fix Slurm job observation contracts Support both ordinary jobs and array-task observations while preserving accounting-lag semantics. Bound scheduler numeric fields and diagnostics, and honor unthrottled plan arrays when concurrency is omitted. * refine Slurm launcher boundaries - Internalize implementation APIs and clarify transient types - Harden parsing, rendering, and process error normalization - Extend fakes and executable checksum coverage Part of #868 * fix Slurm selector filtering ownership * simplify Slurm launcher structure --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: resolve Slurm serving deployments Add pure, package-owned vLLM compatibility and deterministic process, readiness, backend, and logical endpoint resolution. Validate topology, launch inputs, and network identities with single-node and multi-node golden coverage. Closes #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden Slurm serving contracts Revalidate persisted launch policy inputs, require canonical process ordering, and normalize resolver errors. Reject vLLM CLI aliases and transports that could override compiler-owned topology or endpoints. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: close Slurm serving validation gaps Preserve independent single-node expert-parallel replicas while rejecting cross-node EP. Block attached compiler-owned vLLM topology aliases, require canonical runtime versions, and cover the boundaries with regression tests. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden vLLM launch ownership Reject unmodeled topology, plugin, LoRA, speculative-model, and environment overrides at authored and persisted boundaries. Validate JSON arguments, secret references, readiness paths, and compatibility version lengths. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: bind serving context to plan Derive serving placement and client endpoint records from the same resolved plan so callers cannot mix contracts across plans. Reject vLLM API authentication until the logical endpoint models credential forwarding. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: align serving with migration contracts - enforce plan-owned environment and endpoint behavior - limit compatibility to tested vLLM release series - split shared and vLLM configuration from run records - cover multi-group ordering and launch delays Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs: clarify option flag extraction Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: share argument and duration helpers Use action-named shared helpers for option extraction and duration conversion while keeping vLLM flag canonicalization backend-specific. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: simplify vLLM compatibility contract Track the tested runtime range and contract revision without repeating baseline vLLM or package-owned behaviors as capability booleans. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: make vLLM resolution version agnostic Preserve inspected runtime versions as digest-bound provenance without hard-coded support gates or redundant compatibility records. Continue to validate persisted text and compiler-owned vLLM arguments. Refs #866 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: treat runtime version as provenance Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: name vLLM serving module explicitly Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: remove serving context wrapper Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: centralize Slurm scalar types Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: use str for model aliases Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: clarify serving replica indices * fix: reject abbreviated vLLM flags * refactor: simplify Slurm serving contracts - make vLLM resolution consume placement inputs before plan assembly - keep serving internals backend-specific and remove speculative exports - clarify validation ownership, secret redaction, and digest semantics * fix: bind Slurm serving resolution inputs - Join placement and endpoint claims through one planner-owned context.\n- Normalize inconsistent resolved records to the serving error boundary.\n- Split resolver, record, and launch-policy tests by ownership. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * remove empty serving package initializer * fix serving contract validation boundaries Derive deployment and endpoint inputs from one resolved plan and centralize topology arithmetic. Reject plaintext secrets in persisted builder and vLLM argument payloads, normalize resolver errors, and require canonical argv tokens. * fix option argument validation gaps --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: compile deterministic Slurm run plans Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: enforce Slurm planning invariants Reject unsupported multi-shard semantics and managed output collisions at the effective configuration boundary. Normalize config errors and cover the corrected Big Iron field disposition. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: harden Slurm plan resolution Preserve partial early-shutdown intent and hide secret values in validation errors. Reject unsafe sharding semantics, require exact deployment coverage, and bind sourced payload identity. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: bind Slurm effective plan inputs Preserve all authored early-shutdown controls and reject direct effective invocation drift. Bind shard partition digests to deterministic persisted JSON bytes. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: align Slurm planning boundaries * fix: unify Slurm builder artifact identity * fix: validate Slurm compiler inputs * fix: complete Slurm plan resolution defaults * fix: address plan compiler review feedback * fix: preserve safe validation paths Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: preserve union validation paths Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* feat: add Slurm service foundation Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: harden Slurm service contracts Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: bind Slurm service results Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: clarify Slurm service boundaries Separate run planning from attempt rendering so retries can reuse an immutable plan. Keep injected implementation seams private and align public service failures with the Data Designer error hierarchy. Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: tighten Slurm service contracts * test: fix Slurm seam export guard * fix: define Slurm service boundary contracts * fix: expose Slurm service dependency contracts --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* fix: harden Slurm image publication Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: close Slurm image publication races - Hash SQSH bytes outside the global registry lock.\n- Preserve recovery state when commit synchronization fails.\n- Bind lifecycle artifacts to verified directory descriptors. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat: add Slurm state writer Persist immutable run intent and shard manifests with restrictive descriptor-bound storage, locks, digest validation, and atomic publication. Validate monotonic attempt and readiness updates across concurrent writers, and cover crash recovery, unsafe filesystem state, tampering, and fresh-process reloads. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden Slurm state persistence Bind loaded records back to persisted plan and location invariants, harden lock and file handling against replacement and hardlink races, and avoid duplicate immutable context reads. Add adversarial, concurrency, interruption-recovery, import, and performance regression coverage. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: recover interrupted state publication Repair a committed immutable record when an interruption leaves its package-owned temporary hard link behind. Add regression coverage for the exact post-publication interruption window. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: converge concurrent state recovery Treat a temporary link already removed by a recovery reader as a successful immutable publication. Cover the publisher-reader cleanup race directly. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: cover Slurm state reader errors Exercise invalid and missing shard, attempt, ordinal, and readiness inputs through the public state-writer API. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: retry concurrent state replacement Let descriptor-bound readers retry valid atomic replacements, including replacements that race interrupted-publication cleanup. Continue rejecting unsafe record types and bound the retry loop. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: compose Slurm state persistence Separate validated reads and descriptor-bound storage from the public mutation workflows. Close concurrent attempt publication, retry durability, exact-byte reading, and interrupted recovery races with focused regression coverage. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: shorten state mutation workflows Keep error translation at public boundaries and move lock-held happy paths into small helpers. No state method now exceeds 30 lines. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: expose state collaborators within package Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: harden shard-local state boundaries Keep attempt and readiness mutations scoped to their target shard, move persisted plan validation into the state layer, and share descriptor-safe filesystem primitives across state and image storage. Make the schema-v1 no-migration policy explicit and freeze compatibility fixtures. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: align state boundary invariants Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): add allocation-local client worker * fix(slurm): address client worker review feedback * fix(slurm): harden client worker boundaries * fix(slurm): harden client filesystem boundary

* feat: finalize Slurm shard winners Lease shard dataset workspaces under reviewed locks. Validate client and candidate records plus output file digests. Atomically publish one immutable winner and seal retries. Part of #869 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: unwind interrupted dataset locks Ensure an opened shard context closes when resume-lock acquisition is interrupted by any BaseException. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: rebind verified candidate files Reject candidate path replacement after digesting an opened output descriptor and before winner publication. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test: cover nested candidate paths Exercise valid nested output verification and rejection of a symlinked intermediate directory. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: validate winner retry timestamps Reject non-UTC publication timestamps before the existing-winner idempotency path can ignore them. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: classify invalid winners as corruption Normalize semantically invalid persisted winner chains as corruption across load, finalize, attempt, and workspace APIs. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: compose shard winner finalization Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor: compose finalization validation Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: harden candidate publication boundaries Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: bind finalization publication Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix: publish allocation results before success Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): wire M2 services and CLI Add package-owned execution, status, cancellation, and image service wiring. Resolve client dependencies and expose the M2 operations through the Slurm CLI. Part of #874 Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): reject swapped SQSH file types Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): clarify unavailable operations Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): resolve review findings --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* fix(slurm): close one-node M2 integration * fix(slurm): address integration review findings * fix(slurm): address smoke and review findings * docs(slurm): align status reconciliation help * fix(slurm): honor nested runtime mounts * fix(slurm): allow exclusive GPU allocations * fix(slurm): accept exclusive job GPU visibility * fix(slurm): load plans through nested mounts * fix(slurm): validate logical client paths * fix(slurm): resolve client artifacts by mount * fix(slurm): map serving model paths * fix(slurm): avoid idle GPU reaper * fix(slurm): isolate shared-node service ports * fix(slurm): validate allocation-local endpoints

* feat: reconcile persisted Slurm state Persist normalized scheduler observations and compose fresh-process run, shard, attempt, readiness, generation, and winner status. Preserve bounded accounting lag and immutable terminal evidence for status and benchmark refresh consumers. Part of #869 Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): harden state reconciliation Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): bound preemption reconciliation Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): bound public preemption status Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * feat: add Slurm retry and deterministic collection (#915) * feat(slurm): add retry and deterministic collection Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): recover ambiguous submissions Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): preserve collection snapshots Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): bind retries and collection ownership Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): preserve scheduler path on retry Render retry allocations with the same profile-configured Slurm command path as initial generation attempts so cluster-local srun and scontrol binaries remain discoverable. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): isolate attempt observation clocks Apply persisted timestamp floors per scheduler identity so a clock-ahead sibling cannot expire another attempt's preemption-requeue window. Persist each active attempt with its own reconciled observation time. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): isolate plugin-aware client generation Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor(slurm): expose client worker boundary Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor(slurm): compose client process boundary Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): refresh client overlay inventory Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test(slurm): exercise plugin worker handoff Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* harden Slurm release artifacts Add sanitized credential and infrastructure scans for source, fixtures, runtime archives, logs, and wheels. Include Apache license text in the Slurm wheel and redact sensitive scheduler diagnostics. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix Slurm security audit gaps Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * harden adjacent secret redaction Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * make Slurm redaction fail closed Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * redact underscore secret options Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * harden Slurm security audit boundaries Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix control-obscured Slurm secrets Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix control-split secret redaction Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix whitespace-control secret suffixes Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * harden archive content classification Detect explicitly supplied ZIP and tar inputs from their content when filenames use opaque suffixes. Reject unsupported tar special members while continuing to allow ordinary directories. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * close Slurm security boundary gaps Handle serialized authorization headers and Unicode whitespace without exposing credential suffixes. Keep public-artifact scanning fail-closed for non-regular files and unreadable ZIP members, and narrow test sentinels to complete values. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * redact complete Authorization diagnostics Treat an Authorization field as credential-bearing through its line boundary even when its value is quoted. This keeps control-obscured suffixes from surviving the protected redaction pass. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): compose distributed runtime launch Layer multi-node topology and follower-failure handling onto the production allocation bootstrap introduced by #929. Preserve the one-node path while adding host-scoped steps, coordinated node workers, remote readiness probes, and distributed preflight coverage. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * map distributed model paths Apply the resolved container-mount mapping to absolute model paths in both coordinated multi-node workers and remote-node serving commands. Cover differing host and container roots in the bootstrap manifest regression. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * adapt retry bootstrap coverage Pass the one-node allocation layout required by the distributed runtime manifest after composing the merged retry coverage onto the new shared base. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix distributed launch preflight Select the multi-process vLLM executor whenever pipeline or tensor parallelism spans more than one worker. Carry mapped absolute model paths through the node-worker contract so every assigned node verifies accessibility before launch. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): add profile setup and image lifecycle wiring * fix(slurm): address profile and lifecycle review * fix(slurm): require terminal cancellation evidence

* feat(slurm): wire retry and merge commands * fix(slurm): harden retry and collection actions * fix(slurm): preserve retry confirmation intent * fix(slurm): recover active retry previews * fix(slurm): address retry and merge review feedback

* feat(slurm): add deterministic benchmark workflows * fix(slurm): correct benchmark concurrency and accounting * refactor(slurm): flatten benchmark CLI callbacks

* fix: use allocation-local scratch for Slurm runtime * fix: propagate staged runtime to Slurm endpoints * fix: launch retry identity through Slurm * fix: expose staged runtime to retries * fix: isolate retry verification state * fix: preserve logical retry plan references

* docs(slurm): add public workflow and release checks Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): address release review findings Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): harden release validation Signed-off-by: Andre Manoel <amanoel@nvidia.com> * docs(slurm): sanitize coverage source Signed-off-by: Andre Manoel <amanoel@nvidia.com> * docs(slurm): fix walkthrough findings Signed-off-by: Andre Manoel <amanoel@nvidia.com> * docs(slurm): harden onboarding guidance Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): preserve allocation failure diagnostics * docs(slurm): clarify verified onboarding behavior Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* fix(slurm): show CLI submission progress Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): keep progress failures from stopping submission Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * feat(slurm): add opt-in image job log following Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): preserve and bound followed log output Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): flush final and carriage-return logs Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): improve CLI output and image logs Make operator-facing Slurm results readable while preserving opt-in JSON for automation. Surface bounded, redacted sbatch rejection details and print status/log guidance after successful generation submission. Follow image lifecycle logs by default in interactive terminals, with explicit flags to enable or disable streaming. Update the Slurm docs and cover both output modes and submission diagnostics in tests. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): harden CLI guidance and output Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): discover scheduler tools for jobs Capture the submit-host Slurm executable directory in the immutable run plan so controlled batch PATHs can invoke srun and scontrol without a profile override. Fail before submission with an actionable diagnostic when those tools are unavailable, and document the optional override. * fix(slurm): preserve scheduler command symlinks Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): isolate single-node serving replicas Run multiple vLLM replicas inside one GPU-owning Slurm step so overlapping steps cannot be assigned the same device. Preserve the existing local vLLM command for each process and cover TP=1 and TP=4 assignments. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): bind shared serving worker to node address Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): reserve CPUs across GPU sockets Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): keep local replicas on loopback Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): support multiple GPU partitions Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): validate every selected GPU partition Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): create profile parent directories Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): verify new profile directories Keep profile creation bound to verified directory descriptors and restore private permissions when a restrictive umask is active. Update the profile guide to reflect automatic parent creation. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): harden profile directory setup Normalize unsupported permission operations and sync newly created directory metadata before publishing the profile catalog. Cover replacement and error-boundary behavior. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): support Linux directory chmod Use an opened directory for permission repair when Linux rejects nofollow fchmodat. Handle a restrictive umask through a Linux path descriptor while keeping directory traversal bound to verified descriptors. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): show vLLM startup log path Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): honor JSON output with newer Typer Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): refresh serving log guidance across executions Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): align native retry and user-site policy Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): preserve delayed replica startup budget Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): cancel proxy requests and catch timeouts Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix(slurm): isolate user-site path inspection Signed-off-by: Andre Manoel <amanoel@nvidia.com> * test(slurm): guard proxy and readiness compatibility Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com>

#1012) * fix(slurm): clarify config defaults and inert fields Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs(slurm): keep run config guidance concise Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor(slurm): remove unused run config flags Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs(slurm): clarify benchmark model concurrency sweep Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): confirm image removal and allow artifact deletion Show an explicit removal result in human output. Allow opt-in deletion only for unshared package-owned OCI SQSH artifacts, with safe failure reporting and documentation. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * feat(slurm): timestamp image job progress Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * feat(slurm): clarify image add progress Report deterministic image job phases and reduce idle status noise. Omit blank followed-log lines and show elapsed time after registration. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): simplify image details output Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): clarify image add and list results Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): clarify image command help Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): sanitize image list runtime Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): explain missing builder sources Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): show serving startup progress in allocation logs Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): point allocation logs to client generation progress Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): choose Parquet partitions at merge Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): preserve collection intent on retry Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Avoid writing a duplicate Parquet export when the engine batches can be used directly as a bounded candidate. Keep the export path for schema drift or too many batch files. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Give each node-local vLLM replica its own tokenizer cache so concurrent startup cannot read or remove another replica’s partially downloaded vocabulary. Preserve an explicitly configured cache and clean generated caches after the replicas stop. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* feat(slurm): merge runs by ID with default output Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor(slurm): name default merge directory Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test(slurm): align merge destination coverage Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): read only committed attempt results Ignore client-owned partial result files when an attempt has no bound candidate. Failed shards can then be reconciled and reported without treating their incomplete output as a finalized result. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * accept nonempty partial Slurm shards Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): analyze nonempty partial benchmarks Use actual winning rows for benchmark throughput and capacity estimates. Update the benchmark result contract and Slurm docs for partial winners. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): merge reordered nested shard schemas Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): preserve saved collection plans Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* fix(slurm): benchmark one array task per case Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): analyze saved benchmark manifests Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): validate benchmark versions lazily Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs: clarify saved benchmark rates Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * feat(slurm): default benchmark records adaptively Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs: fix Slurm example references Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * docs: clarify Slurm topologies and shards Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

Add opt-in shared-node GPU quotas and explicit GPU offsets for colocating distinct models, including multiple models on one GPU. Coordinate readiness before overlapping launches and support typed vLLM IO processor plugins. Signed-off-by: Andre Manoel <amanoel@nvidia.com>

Signed-off-by: Andre Manoel <amanoel@nvidia.com>

* fix(slurm): harden run lifecycle and diagnostics Validate built-in builder settings before allocation while deferring plugin and compute-local inputs. Protect image artifacts referenced by run plans and verify reused client overlays. Correct readiness, progress, benchmark, and multi-node behaviors found during branch review. Add regression coverage and update Slurm docs. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): address review findings Poll readiness across all started servers so a slow deployment cannot consume another deployment timeout. Protect images referenced by custom run IDs and retry transient batch metadata failures in progress reporting. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * fix(slurm): narrow artifact locks and validate plans Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

* refactor(slurm): simplify preflight and readiness tests Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * test(slurm): reuse readiness command helper Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> * refactor(slurm): simplify run preparation ownership Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com> --------- Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

andreatnvidia and others added 29 commits October 15, 2026 15:08

feat: add Slurm image lifecycle jobs (#896)

5834766

feat: add one-node Slurm runtime (#909)

6feda83

feat: add allocation-local Slurm client worker (#911)

66a6f54

* feat(slurm): add allocation-local client worker * fix(slurm): address client worker review feedback * fix(slurm): harden client worker boundaries * fix(slurm): harden client filesystem boundary

feat(slurm): add profile setup and image lifecycle wiring (#933)

d0dd27f

* feat(slurm): add profile setup and image lifecycle wiring * fix(slurm): address profile and lifecycle review * fix(slurm): require terminal cancellation evidence

feat(slurm): add deterministic benchmark workflows (#932)

4931532

* feat(slurm): add deterministic benchmark workflows * fix(slurm): correct benchmark concurrency and accounting * refactor(slurm): flatten benchmark CLI callbacks

nabinchha and others added 21 commits October 15, 2026 15:09

fix(slurm): support Enroot 3.5 OCI imports (#988)

72990b4

Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

fix(slurm): leave host patterns empty by default (#1014)

c98045e

Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

fix(slurm): reuse final batch files for candidates (#1023)

b8c38e0

Avoid writing a duplicate Parquet export when the engine batches can be used directly as a bounded candidate. Keep the export path for schema drift or too many batch files. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>

nabinchha force-pushed the feat/slurm-execution branch from f404976 to a498342 Compare October 15, 2026 21:09

nabinchha and others added 4 commits October 15, 2026 15:24

fix: recognize locked dependency licenses

f4b94a0

fix(slurm): deduplicate runtime inventory paths

737e13c

Signed-off-by: Andre Manoel <amanoel@nvidia.com>

fix(slurm): sync checked examples with documentation

56a35a4

Signed-off-by: Andre Manoel <amanoel@nvidia.com>

chore: merge main into Slurm integration

d76383a

Signed-off-by: Andre Manoel <amanoel@nvidia.com>

nabinchha changed the title ~~feat(slurm): integrate Slurm execution for 1.0.0rc1~~ feat(slurm): integrate Slurm execution for 1.0.0 release Oct 16, 2026

nabinchha added 2 commits October 16, 2026 14:59

github-actions Bot mentioned this pull request Oct 19, 2026

Agentic CI: Issue & PR Triage Tracker #562

Open

Navigation Menu

Repository navigation

feat(slurm): integrate Slurm execution for 1.0.0 release - #1040

feat(slurm): integrate Slurm execution for 1.0.0 release#1040
nabinchha wants to merge 60 commits into
mainfrom
feat/slurm-execution

nabinchha commented Oct 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nabinchha commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📋 Summary

🔗 Related Issue

🔄 Changes

✨ Added

🔧 Changed

🐛 Fixed

🔍 Attention Areas

🧪 Testing

✅ Checklist

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nabinchha commented Oct 15, 2026 •

edited

Loading