Skip to content

Add Slurm backend#4009

Merged
un-def merged 4 commits into
masterfrom
issue_4008_slurm_backend
Jul 8, 2026
Merged

Add Slurm backend#4009
un-def merged 4 commits into
masterfrom
issue_4008_slurm_backend

Conversation

@un-def

@un-def un-def commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Closes: #4008

@un-def un-def requested review from peterschmidt85 and r4victor July 6, 2026 13:05
r4victor
r4victor reviewed Jul 7, 2026
View reviewed changes

@r4victor r4victor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@un-def, Have you tested running fat ML images on Slurm? I think enroot may take quite a long time to convert images to squashfs. I'd bump get_provisioning_timeout(), e.g. from 10m to 20m.

r4victor
r4victor reviewed Jul 7, 2026
View reviewed changes
Comment thread src/dstack/_internal/core/backends/slurm/client.py Outdated
if not cpus_raw:
cpus_raw = node_dict.get("cputot")
if not cpus_raw:
raise SlurmClientError("Failed to detect CPU count")

@r4victor r4victor Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueError?

@un-def

un-def commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

Have you tested running fat ML images on Slurm? I think enroot may take quite a long time to convert images to squashfs. I'd bump get_provisioning_timeout(), e.g. from 10m to 20m.

Good idea. I haven't tested really large images, but even dstackai/base takes substantial amount of time, in my experience.

@un-def un-def merged commit 39a62c6 into master Jul 8, 2026
24 checks passed
@un-def un-def deleted the issue_4008_slurm_backend branch July 8, 2026 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@r4victor r4victor r4victor approved these changes
@peterschmidt85 peterschmidt85 Awaiting requested review from peterschmidt85

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Support Slurm backend

2 participants

@un-def @r4victor