diff --git a/mkdocs/blog/posts/benchmark-amd-containers-and-partitions.md b/mkdocs/blog/posts/benchmark-amd-containers-and-partitions.md index 8b945aaba..d98f3523e 100644 --- a/mkdocs/blog/posts/benchmark-amd-containers-and-partitions.md +++ b/mkdocs/blog/posts/benchmark-amd-containers-and-partitions.md @@ -215,7 +215,7 @@ taskset -c 0-31 ./ib_write_bw -d rocep28s0 -F -a --report_gbits -q $SERVER_IP_AD #### Containers -For the container experiments we used a `dstack`’s [distributed task](../../docs/concepts/tasks.md#distributed-tasks). +For the container experiments we used a `dstack`’s [multi-node task](../../docs/concepts/tasks.md#nodes). **RCCL tests** diff --git a/mkdocs/blog/posts/digitalocean-and-amd-dev-cloud.md b/mkdocs/blog/posts/digitalocean-and-amd-dev-cloud.md index 7893356f4..b13b8f67d 100644 --- a/mkdocs/blog/posts/digitalocean-and-amd-dev-cloud.md +++ b/mkdocs/blog/posts/digitalocean-and-amd-dev-cloud.md @@ -139,7 +139,7 @@ $ dstack apply -f examples/models/gpt-oss/120b.dstack.yml > If you prefer to use bare-metal clusters with `dstack`, you can create an [SSH fleet](../../docs/concepts/fleets.md#ssh-fleets). -> This way, you’ll be able to run [distributed tasks](../../docs/concepts/tasks.md#distributed-tasks) efficiently across the cluster. +> This way, you’ll be able to run [multi-node tasks](../../docs/concepts/tasks.md#nodes) efficiently across the cluster. !!! info "What's next?" 1. Check [Quickstart](../../docs/quickstart.md) diff --git a/mkdocs/blog/posts/gpu-health-checks.md b/mkdocs/blog/posts/gpu-health-checks.md index 84746ed90..d2d957975 100644 --- a/mkdocs/blog/posts/gpu-health-checks.md +++ b/mkdocs/blog/posts/gpu-health-checks.md @@ -51,7 +51,7 @@ A healthy instance is ready for workloads. A warning means you should monitor it This release focuses on passive checks using DCGM background health checks. These run continuously and do not interrupt workloads. -For active checks today, you can run [NCCL/RCCL tests](../../docs/examples/clusters/nccl-rccl-tests.md) as a [distributed task](../../docs/concepts/tasks.md#distributed-tasks) to verify GPU-to-GPU communication and bandwidth across a fleet. Active tests like these can reveal network or interconnect issues that passive monitoring might miss. More built-in support for active diagnostics is planned. +For active checks today, you can run [NCCL/RCCL tests](../../docs/examples/clusters/nccl-rccl-tests.md) as a [multi-node task](../../docs/concepts/tasks.md#nodes) to verify GPU-to-GPU communication and bandwidth across a fleet. Active tests like these can reveal network or interconnect issues that passive monitoring might miss. More built-in support for active diagnostics is planned. ## Supported backends diff --git a/mkdocs/blog/posts/hotaisle.md b/mkdocs/blog/posts/hotaisle.md index 0d531710d..98cc6bf5d 100644 --- a/mkdocs/blog/posts/hotaisle.md +++ b/mkdocs/blog/posts/hotaisle.md @@ -103,7 +103,7 @@ $ dstack apply -f .dstack.yml Currently, `dstack` supports 1xGPU Hot Aisle VMs. Support for 8xGPU VMs will be added once Hot Aisle supports it. > If you prefer to use Hot Aisle’s bare-metal 8-GPU clusters with dstack, you can create an [SSH fleet](../../docs/concepts/fleets.md#ssh-fleets). -> This way, you’ll be able to run [distributed tasks](../../docs/concepts/tasks.md#distributed-tasks) efficiently across the cluster. +> This way, you’ll be able to run [multi-node tasks](../../docs/concepts/tasks.md#nodes) efficiently across the cluster. !!! info "What's next?" 1. Check [Quickstart](../../docs/quickstart.md) diff --git a/mkdocs/blog/posts/kubernetes-beta.md b/mkdocs/blog/posts/kubernetes-beta.md index 143f82dfd..834deeed3 100644 --- a/mkdocs/blog/posts/kubernetes-beta.md +++ b/mkdocs/blog/posts/kubernetes-beta.md @@ -128,12 +128,12 @@ Dev environments support many [different options](../../docs/concepts/dev-enviro ## Running distributed training -Distributed training can be performed in `dstack` using [distributed tasks](../../docs/concepts/tasks.md#distributed-tasks). +Distributed training can be performed in `dstack` using [multi-node tasks](../../docs/concepts/tasks.md#nodes). The configuration is similar to a dev environment, except it runs across multiple nodes. ### Creating a cluster fleet -Before running a distributed task, create a fleet with `placement` set to `cluster`: +Before running a multi-node task, create a fleet with `placement` set to `cluster`:
@@ -172,11 +172,11 @@ Provisioning...
-Once the fleet is created, you can run distributed tasks on it. +Once the fleet is created, you can run multi-node tasks on it. ### NCCL tests example -Below is an example of using distributed tasks to run NCCL tests. +Below is an example of using multi-node tasks to run NCCL tests. It also demonstrates how to use mpirun with `dstack`:
diff --git a/mkdocs/blog/posts/nebius-in-dstack-sky.md b/mkdocs/blog/posts/nebius-in-dstack-sky.md index 1f911f98d..b22d26176 100644 --- a/mkdocs/blog/posts/nebius-in-dstack-sky.md +++ b/mkdocs/blog/posts/nebius-in-dstack-sky.md @@ -101,7 +101,7 @@ $ dstack apply -f my-cluster.dstack.yml
-Once the fleet is ready, you can run [distributed tasks](../../docs/concepts/tasks.md#distributed-tasks). +Once the fleet is ready, you can run [multi-node tasks](../../docs/concepts/tasks.md#nodes). `dstack` automatically configures drivers, networking, and fast GPU-to-GPU interconnect. To learn more, see the [clusters](../../docs/examples/clusters/nebius.md) guide. diff --git a/mkdocs/blog/posts/nebius.md b/mkdocs/blog/posts/nebius.md index 5fab31022..c78daa99b 100644 --- a/mkdocs/blog/posts/nebius.md +++ b/mkdocs/blog/posts/nebius.md @@ -101,7 +101,7 @@ $ dstack apply -f .dstack.yml The new `nebius` backend supports CPU and GPU instances, [fleets](../../docs/concepts/fleets.md), -[distributed tasks](../../docs/concepts/tasks.md#distributed-tasks), and more. +[multi-node tasks](../../docs/concepts/tasks.md#nodes), and more. > Support for [network volumes](../../docs/concepts/volumes.md#network-volumes) and accelerated cluster interconnects is coming soon. diff --git a/mkdocs/blog/posts/nvidia-dynamo.md b/mkdocs/blog/posts/nvidia-dynamo.md index 1d2e63d66..4c2854490 100644 --- a/mkdocs/blog/posts/nvidia-dynamo.md +++ b/mkdocs/blog/posts/nvidia-dynamo.md @@ -29,7 +29,7 @@ Dynamo brings together disaggregated serving, intelligent routing, KV cache mana ## PD disaggregation with dstack -To deploy NVIDIA Dynamo with PD disaggregation, define a [service](../../docs/concepts/services.md) with three [replica groups](../../docs/concepts/services.md#replicas-and-scaling): +To deploy NVIDIA Dynamo with PD disaggregation, define a [service](../../docs/concepts/services.md) with three [replica groups](../../docs/concepts/services.md#replica-groups): - a Dynamo router - prefill workers diff --git a/mkdocs/blog/posts/sglang-router.md b/mkdocs/blog/posts/sglang-router.md index 028e3a23d..99c826a13 100644 --- a/mkdocs/blog/posts/sglang-router.md +++ b/mkdocs/blog/posts/sglang-router.md @@ -121,7 +121,7 @@ $ dstack apply -f gateway.dstack.yml Once the gateway has a hostname, update your domain’s DNS settings by adding a record for `*.`. -After that, if you configure [replicas and scaling](../../docs/concepts/services.md#replicas-and-scaling), the gateway will automatically scale the number of replicas and route traffic across them. +After that, if you configure [replicas and scaling](../../docs/concepts/services.md#scaling), the gateway will automatically scale the number of replicas and route traffic across them. ### Router diff --git a/mkdocs/blog/posts/volumes-on-runpod.md b/mkdocs/blog/posts/volumes-on-runpod.md index 08f2e1912..579cc9f4e 100644 --- a/mkdocs/blog/posts/volumes-on-runpod.md +++ b/mkdocs/blog/posts/volumes-on-runpod.md @@ -131,7 +131,7 @@ In this case, `dstack` attaches the specified volume to each new replica. This e once, reducing cold start time in proportion to the model size. A notable feature of Runpod is that volumes can be attached to multiple containers simultaneously. This capability is -particularly useful for auto-scalable services or distributed tasks. +particularly useful for auto-scalable services or multi-node tasks. Using [volumes](../../docs/concepts/volumes.md) not only optimizes inference cold start times but also enhances the efficiency of data and model checkpoint loading during training and fine-tuning. diff --git a/mkdocs/docs/concepts/fleets.md b/mkdocs/docs/concepts/fleets.md index 9e85b157e..630f23b2e 100644 --- a/mkdocs/docs/concepts/fleets.md +++ b/mkdocs/docs/concepts/fleets.md @@ -136,7 +136,7 @@ Backend fleets support [many options](../reference/dstack.yml/fleet.md); see som Both [backend fleets](#backend-fleet) and [SSH fleets](#ssh-fleet) allow the `placement` property to be set to `cluster`. -This property ensures that instances are interconnected. This is required for running [distributed tasks](tasks.md#distributed-tasks). +This property ensures that instances are interconnected. This is required for running [multi-node tasks](tasks.md#nodes). === "Backend fleets" Backend fleets allow to provision interconnected clusters across supported backends. @@ -405,8 +405,8 @@ For example, with 8 GPUs, 128 CPUs, and 2TB RAM, setting `blocks` to `8` gives e Set `blocks` to `auto` to match the number of blocks to the number of GPUs. -!!! info "Distributed tasks" - Distributed tasks require exclusive access to all host resources and therefore must use all blocks on each node. +!!! info "Multi-node tasks" + Multi-node tasks require exclusive access to all host resources and therefore must use all blocks on each node. ### SSH config diff --git a/mkdocs/docs/concepts/services.md b/mkdocs/docs/concepts/services.md index f9f260408..23285b14b 100644 --- a/mkdocs/docs/concepts/services.md +++ b/mkdocs/docs/concepts/services.md @@ -24,7 +24,6 @@ The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `dev.dstack.yml` name: qwen36 image: lmsysorg/sglang:v0.5.10.post1 - commands: - | sglang serve \ @@ -45,6 +44,9 @@ The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `dev.dstack.yml` path: /root/.cache optional: true + # Number of replicas, or a range to enable scaling + replicas: 1 + resources: shm_size: 16GB gpu: H100:4 @@ -82,6 +84,9 @@ The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `dev.dstack.yml` path: /root/.cache optional: true + # Number of replicas, or a range to enable scaling + replicas: 1 + resources: cpu: 52.. memory: 896GB.. @@ -92,8 +97,13 @@ The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `dev.dstack.yml` -The first startup on MI300X can take longer while SGLang compiles ROCm -kernels. + The first startup on MI300X can take longer while SGLang compiles ROCm + kernels. + + + +!!! info "Replicas" + [`replicas`](../reference/dstack.yml/service.md#replicas) can be a number or, if you use a [gateway](#gateway), a range to enable [scaling](#scaling). If omitted, it defaults to `1`. To run a service, pass the configuration to [`dstack apply`](../reference/cli/dstack/apply.md): @@ -143,198 +153,78 @@ The request and response format depends on the serving framework used by the service. Even for OpenAI-compatible endpoints, the format may vary slightly across frameworks. -If [authorization](#authorization) is not disabled, the service endpoint requires the `Authorization` header with `Bearer `. - -## Configuration options - - - -### Gateway - -Here are cases where a service may need a [gateway](gateways.md): - -* To use [auto-scaling](#replicas-and-scaling) or [rate limits](#rate-limits) -* To enable HTTPS for the endpoint and map it to your domain -* If your service requires WebSockets -* If your service cannot work with a [path prefix](#path-prefix) - - - -If you want `dstack` to explicitly validate that a gateway is used, you can set the [`gateway`](../reference/dstack.yml/service.md#gateway) property in the service configuration to `true`. In this case, `dstack` will raise an error during `dstack apply` if a default gateway is not created. - -You can also set the `gateway` property to the name of a specific gateway, if required. - -If you have a [gateway](gateways.md) created, the service endpoint will be accessible at `https://./`: - -
- -```shell -$ curl https://llama31.example.com/v1/chat/completions \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer <user token>' \ - -d '{ - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct", - "messages": [ - { - "role": "user", - "content": "Compose a poem that explains the concept of recursion in programming." - } - ] - }' -``` - -
- -### Replicas and scaling - -By default, `dstack` runs a single replica of the service. -You can configure the number of replicas as well as the auto-scaling rules. - -=== "NVIDIA" - -
- - ```yaml - type: service - name: qwen36-service - - image: lmsysorg/sglang:v0.5.10.post1 - - commands: - - | - sglang serve \ - --model-path Qwen/Qwen3.6-27B \ - --host 0.0.0.0 \ - --port 30000 \ - --tp $DSTACK_GPUS_NUM \ - --reasoning-parser qwen3 \ - --mem-fraction-static 0.8 \ - --context-length 262144 - - port: 30000 - model: Qwen/Qwen3.6-27B - - volumes: - # Optional instance volume for model and runtime caches - - instance_path: /root/.cache - path: /root/.cache - optional: true - - resources: - shm_size: 16GB - gpu: H100:4 - - replicas: 1..2 - scaling: - metric: rps - target: 1 - ``` - -
- -=== "AMD" - -
- - ```yaml - type: service - name: qwen36-service - - image: lmsysorg/sglang:v0.5.10-rocm720-mi30x - - commands: - - | - sglang serve \ - --model-path Qwen/Qwen3.6-27B \ - --host 0.0.0.0 \ - --port 30000 \ - --tp $DSTACK_GPUS_NUM \ - --reasoning-parser qwen3 \ - --mem-fraction-static 0.8 \ - --context-length 262144 - - port: 30000 - model: Qwen/Qwen3.6-27B - - volumes: - # Optional instance volume for model and runtime caches - - instance_path: /root/.cache - path: /root/.cache - optional: true - - resources: - cpu: 52.. - memory: 896GB.. - shm_size: 16GB - disk: 450GB.. - gpu: MI300X:4 - - replicas: 1..2 - scaling: - metric: rps - target: 1 - ``` - -
+??? info "Authorization" + If [authorization](#authorization) is not disabled, the service endpoint requires the `Authorization` header with `Bearer `. -The [`replicas`](../reference/dstack.yml/service.md#replicas) property can be a number or a range. +## Replica groups -The [`metric`](../reference/dstack.yml/service.md#metric) property of [`scaling`](../reference/dstack.yml/service.md#scaling) only supports the `rps` metric (requests per second). In this -case `dstack` adjusts the number of replicas (scales up or down) automatically based on the load. +A service can define multiple replica groups. Each group has its own `replicas` count (or range), +`resources`, `commands`, and `scaling` rules. -Setting the minimum number of replicas to `0` allows the service to scale down to zero when there are no requests. +With replica groups, you can define an inference service with separate router and worker +groups: the router can have CPU-only `resources` and different `commands` than the workers, since +both are set per group. -> The `scaling` property requires creating a [gateway](gateways.md). +### Router - +Router sits in front of inference backend workers (SGLang, vLLM, TensorRT-LLM) and decides +which worker serves each request based on routing policy. -### Replica groups +`dstack` supports two router implementations: -A service can define multiple replica groups. Each group has its own `replicas` count (or range), -`resources`, `commands`, and `scaling` rules. For a common use case, see -[PD disaggregation](#pd-disaggregation). +* `sglang` — runs [Shepherd Model Gateway (SMG)](https://lightseek.org/smg/) +* `dynamo` — runs the [NVIDIA Dynamo frontend](https://docs.nvidia.com/dynamo/dev/kubernetes/kv-aware-routing/using-the-dynamo-frontend/)
```yaml type: service -name: llama-8b-service +name: qwen38-flash-next +image: lmsysorg/sglang:qwen38flashnext -image: lmsysorg/sglang:v0.5.10.post1 env: - - MODEL_ID=deepseek-ai/DeepSeek-R1-Distill-Llama-8B + - HF_TOKEN + - MODEL_ID=Qwen/Qwen3.8-Flash-Next groups: - - replicas: 1..2 - scaling: - metric: rps - target: 10 + - replicas: 1 commands: + - pip install smg - | - python -m sglang.launch_server \ - --model-path $MODEL_ID \ + smg launch \ + --host 0.0.0.0 \ --port 8000 \ - --trust-remote-code + --prefill-policy cache_aware resources: - gpu: 48GB + cpu: 4 + router: + type: sglang - - replicas: 1..4 - scaling: - metric: rps - target: 5 + - replicas: 2 commands: - | python -m sglang.launch_server \ --model-path $MODEL_ID \ - --port 8000 \ - --trust-remote-code + --tp $DSTACK_GPUS_NUM \ + --ep $DSTACK_GPUS_NUM \ + --mem-fraction-static 0.85 \ + --chunked-prefill-size 8192 \ + --linear-attn-prefill-backend flashinfer \ + --linear-attn-decode-backend flashinfer \ + --mamba-ssm-dtype bfloat16 \ + --reasoning-parser auto \ + --host 0.0.0.0 \ + --port 8000 resources: - gpu: 24GB + gpu: H200:4 port: 8000 -model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B +model: Qwen/Qwen3.8-Flash-Next + +probes: + - type: http + url: /health + interval: 15s ```
@@ -343,7 +233,7 @@ model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B > Properties such as `regions`, `port`, `env` and some other cannot be configured per replica group. This support is coming soon. -??? info "Accessing replica IPs" +??? info "Internal replica IPs" Commands in any group can reference the internal IP address of any replica in the run via `${{ groups[i].replicas[j].IP_ADDRESS }}`, where `i` is the index of the group in `groups` and `j` is the index of the replica within that group. @@ -395,10 +285,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`: router: type: sglang - - replicas: 1..4 - scaling: - metric: rps - target: 3 + - replicas: 4 commands: - | python -m sglang.launch_server \ @@ -410,10 +297,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`: resources: gpu: H200 - - replicas: 1..8 - scaling: - metric: rps - target: 2 + - replicas: 8 commands: - | python -m sglang.launch_server \ @@ -476,10 +360,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`: router: type: dynamo - - replicas: 1..4 - scaling: - metric: rps - target: 3 + - replicas: 4 python: "3.12" nvcc: true commands: @@ -505,10 +386,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`: resources: gpu: H200 - - replicas: 1..8 - scaling: - metric: rps - target: 2 + - replicas: 8 python: "3.12" nvcc: true commands: @@ -581,10 +459,7 @@ groups: router: type: sglang - - replicas: 1..2 - scaling: - metric: rps - target: 300 + - replicas: 2 commands: - | python3 -m sglang.launch_server \ @@ -607,10 +482,7 @@ groups: cpu: 96.. memory: 512GB.. - - replicas: 1..4 - scaling: - metric: rps - target: 300 + - replicas: 4 commands: - | python3 -m sglang.launch_server \ @@ -658,29 +530,85 @@ volumes: While the prefill and decode replicas run on GPUs, the router replica requires a CPU instance in the same cluster. -### Authorization +## Gateway -By default, the service enables authorization, meaning the service endpoint requires a `dstack` user token. -This can be disabled by setting `auth` to `false`. +Here are cases where a service may need a [gateway](gateways.md): -
+* To use [auto-scaling](#scaling) or [rate limits](#rate-limits) +* To enable HTTPS for the endpoint and map it to your domain +* If your service requires WebSockets +* If your service cannot work with a [path prefix](#path-prefix) + + + +If you want `dstack` to explicitly validate that a gateway is used, you can set the [`gateway`](../reference/dstack.yml/service.md#_gateway) property in the service configuration to `true`. In this case, `dstack` will raise an error during `dstack apply` if a default gateway is not created. + +You can also set the `gateway` property to the name of a specific gateway, if required. + +If you have a [gateway](gateways.md) created, the service endpoint will be accessible at `https://./`: + +
+ +```shell +$ curl https://llama31.example.com/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -H 'Authorization: Bearer <user token>' \ + -d '{ + "model": "meta-llama/Meta-Llama-3.1-8B-Instruct", + "messages": [ + { + "role": "user", + "content": "Compose a poem that explains the concept of recursion in programming." + } + ] + }' +``` + +
+ +## Configuration options + +### Environment variables + +
```yaml type: service -name: http-server-service - -# Disable authorization -auth: false +name: llama-2-7b-service python: 3.12 +env: + - HF_TOKEN + - MODEL=NousResearch/Llama-2-7b-chat-hf commands: - - python3 -m http.server + - uv pip install vllm + - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 + +resources: + gpu: 24GB ```
+> If you don't assign a value to an environment variable (see `HF_TOKEN` above), +`dstack` will require the value to be passed via the CLI or set in the current process. + +??? info "System environment variables" + The following environment variables are available in any run by default: + + | Name | Description | + |-------------------------|--------------------------------------------------| + | `DSTACK_RUN_NAME` | The name of the run | + | `DSTACK_REPO_ID` | The ID of the repo | + | `DSTACK_GPUS_NUM` | The total number of GPUs in the run | + | `DSTACK_WORKING_DIR` | The working directory of the run | + | `DSTACK_REPO_DIR` | The directory where the repo is mounted (if any) | + + + ### Probes Configure one or more HTTP probes to periodically check the health of the service. @@ -760,46 +688,46 @@ Probes are executed for each service replica while the replica is `running`. A p See the [reference](../reference/dstack.yml/service.md#probes) for more probe configuration options. -### Path prefix { #path-prefix } +### Model -If your `dstack` project doesn't have a [gateway](gateways.md), services are hosted with the -`/proxy/services///` path prefix in the URL. -When running web apps, you may need to set some app-specific settings -so that browser-side scripts and CSS work correctly with the path prefix. +If the service runs a model with an OpenAI-compatible interface, you can set the [`model`](#model) property to make the model accessible through `dstack`'s chat UI on the `Models` page. +In this case, `dstack` will use the service's `/v1/chat/completions` service. -
+When `model` is set, `dstack` automatically configures [`probes`](#probes) to verify model health. +To customize or disable this, set `probes` explicitly. -```yaml -type: service -name: dash -gateway: false +### Scaling -auth: false -# Do not strip the path prefix -strip_prefix: false +To scale a service automatically, set [`replicas`](#replicas) to a range and configure +[`scaling`](../reference/dstack.yml/service.md#scaling). `dstack` then adjusts the number of +replicas within that range based on the load. -env: - # Configure Dash to work with a path prefix - # Replace `main` with your dstack project name - - DASH_ROUTES_PATHNAME_PREFIX=/proxy/services/main/dash/ +
-commands: - - uv pip install dash - # Assuming the Dash app is in your repo at app.py - - python app.py +```yaml +type: service +image: my-app:latest +port: 80 -port: 8050 +replicas: 1..4 +scaling: + metric: rps + target: 3 ```
-By default, `dstack` strips the prefix before forwarding requests to your service, -so to the service it appears as if the prefix isn't there. This allows some apps -to work out of the box. If your app doesn't expect the prefix to be stripped, -set [`strip_prefix`](../reference/dstack.yml/service.md#strip_prefix) to `false`. +[`metric`](../reference/dstack.yml/service.md#metric) currently supports only `rps` (requests per +second). `target` is the RPS a single replica should handle, so with `target: 3`, a load of 12 RPS +scales the service to 4 replicas. -If your app cannot be configured to work with a path prefix, you can host it -on a dedicated domain name by setting up a [gateway](gateways.md). +Setting the minimum number of replicas to `0` allows the service to scale down to zero when there are no requests. + +Each [replica group](#replica-groups) can set its own `replicas` range and `scaling` rules, so +worker groups scale independently. For now, a group with [`router`](#router) must have +`replicas: 1` and cannot be scaled. + +> The `scaling` property requires creating a [gateway](gateways.md). ### Rate limits @@ -850,91 +778,29 @@ Limits apply to the whole service (all replicas) and per client (by IP). Clients
-### Model - -If the service runs a model with an OpenAI-compatible interface, you can set the [`model`](#model) property to make the model accessible through `dstack`'s chat UI on the `Models` page. -In this case, `dstack` will use the service's `/v1/chat/completions` service. +### Authorization -When `model` is set, `dstack` automatically configures [`probes`](#probes) to verify model health. -To customize or disable this, set `probes` explicitly. - -### Resources - -If you specify memory size, you can either specify an explicit size (e.g. `24GB`) or a -range (e.g. `24GB..`, or `24GB..80GB`, or `..80GB`). +By default, the service enables authorization, meaning the service endpoint requires a `dstack` user token. +This can be disabled by setting `auth` to `false`. -
+
```yaml type: service -name: llama31-service +name: http-server-service + +# Disable authorization +auth: false python: 3.12 -env: - - HF_TOKEN - - MODEL_ID=meta-llama/Meta-Llama-3.1-8B-Instruct - - MAX_MODEL_LEN=4096 + commands: - - uv pip install vllm - - | - vllm serve $MODEL_ID - --max-model-len $MAX_MODEL_LEN - --tensor-parallel-size $DSTACK_GPUS_NUM + - python3 -m http.server port: 8000 - -resources: - # 16 or more x86_64 cores - cpu: 16.. - # 2 GPUs of 80GB - gpu: 80GB:2 - - # Minimum disk size - disk: 200GB ```
-The `cpu` property lets you set the architecture (`x86` or `arm`) and core count — e.g., `x86:16` (16 x86 cores), `arm:8..` (at least 8 ARM cores). -If the architecture is not set, `dstack` allows any architecture supported by the `image`, or `x86` if no `image` is set. -Since the default `dstack` image only supports `x86`, requesting `arm` requires setting `image` and is not compatible with `docker: true`. - -The `gpu` property lets you specify vendor, model, memory, and count — e.g., `nvidia` (one NVIDIA GPU), `A100` (one A100), `A10G,A100` (either), `A100:80GB` (one 80GB A100), `A100:2` (two A100), `24GB..40GB:2` (two GPUs with 24–40GB), `A100:40GB:2` (two 40GB A100s). - -If vendor is omitted, `dstack` infers it from the model or defaults to `nvidia`. - - - -??? info "Shared memory" - If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure - `shm_size`, e.g. set it to `16GB`. - -> If you’re unsure which offers (hardware configurations) are available from the configured backends, use the -> [`dstack offer`](../reference/cli/dstack/offer.md#list-gpu-offers) command to list them. - - ### Docker #### Default image @@ -961,28 +827,27 @@ port: 8000
-#### NVCC +??? info "NVCC" + By default, the base Docker image doesn’t include `nvcc`, which is required for building custom CUDA kernels. + If you need `nvcc`, set the [`nvcc`](../reference/dstack.yml/dev-environment.md#nvcc) property to true. -By default, the base Docker image doesn’t include `nvcc`, which is required for building custom CUDA kernels. -If you need `nvcc`, set the [`nvcc`](../reference/dstack.yml/dev-environment.md#nvcc) property to true. - - + -
+
-```yaml -type: service -name: http-server-service + ```yaml + type: service + name: http-server-service -python: 3.12 -nvcc: true + python: 3.12 + nvcc: true -commands: - - python3 -m http.server -port: 8000 -``` + commands: + - python3 -m http.server + port: 8000 + ``` -
+
#### Custom image @@ -1003,6 +868,9 @@ If you want, you can specify your own Docker image via `image`.
+!!! info "No commands" + If `commands` are not specified, `dstack` runs `image`’s entrypoint (or fails if none is set). + #### Docker in Docker Set `docker` to `true` to enable the `docker` CLI in your service, e.g., to run Docker images or use Docker Compose. @@ -1065,69 +933,6 @@ resources: gpu: H100:1 ``` -### `dstack` inside `dstack` - -Set `dstack` to `true` when a service needs to use the dstack CLI. dstack configures the server and -current project automatically. To run authenticated commands, pass `DSTACK_TOKEN` explicitly. - -
- -```yaml -type: service -image: dstackai/dstack -dstack: true -port: 8000 -env: - - DSTACK_TOKEN -commands: - - dstack ps - - python -m http.server 8000 -``` - -
- -> Besides inspecting runs, you can submit new runs with `dstack apply` and attach to them with `dstack attach`. - -### Environment variables - -
- -```yaml -type: service -name: llama-2-7b-service - -python: 3.12 - -env: - - HF_TOKEN - - MODEL=NousResearch/Llama-2-7b-chat-hf -commands: - - uv pip install vllm - - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 -port: 8000 - -resources: - gpu: 24GB -``` - -
- -> If you don't assign a value to an environment variable (see `HF_TOKEN` above), -`dstack` will require the value to be passed via the CLI or set in the current process. - -??? info "System environment variables" - The following environment variables are available in any run by default: - - | Name | Description | - |-------------------------|--------------------------------------------------| - | `DSTACK_RUN_NAME` | The name of the run | - | `DSTACK_REPO_ID` | The ID of the repo | - | `DSTACK_GPUS_NUM` | The total number of GPUs in the run | - | `DSTACK_WORKING_DIR` | The working directory of the run | - | `DSTACK_REPO_DIR` | The directory where the repo is mounted (if any) | - - - ### Working directory If `working_dir` is not specified, it defaults to the working directory set in the Docker image. For example, the [default image](#default-image) uses `/dstack/run` as its working directory. @@ -1351,6 +1156,152 @@ The local path can be either relative to the configuration file or absolute. Currently, you can configure up to one repo per run configuration. +### Resources + +If you specify memory size, you can either specify an explicit size (e.g. `24GB`) or a +range (e.g. `24GB..`, or `24GB..80GB`, or `..80GB`). + +
+ +```yaml +type: service +name: llama31-service + +python: 3.12 +env: + - HF_TOKEN + - MODEL_ID=meta-llama/Meta-Llama-3.1-8B-Instruct + - MAX_MODEL_LEN=4096 +commands: + - uv pip install vllm + - | + vllm serve $MODEL_ID + --max-model-len $MAX_MODEL_LEN + --tensor-parallel-size $DSTACK_GPUS_NUM +port: 8000 + +resources: + # 16 or more x86_64 cores + cpu: 16.. + # 2 GPUs of 80GB + gpu: 80GB:2 + + # Minimum disk size + disk: 200GB +``` + +
+ +The `cpu` property lets you set the architecture (`x86` or `arm`) and core count — e.g., `x86:16` (16 x86 cores), `arm:8..` (at least 8 ARM cores). +If the architecture is not set, `dstack` allows any architecture supported by the `image`, or `x86` if no `image` is set. +Since the default `dstack` image only supports `x86`, requesting `arm` requires setting `image` and is not compatible with `docker: true`. + +The `gpu` property lets you specify vendor, model, memory, and count — e.g., `nvidia` (one NVIDIA GPU), `A100` (one A100), `A10G,A100` (either), `A100:80GB` (one 80GB A100), `A100:2` (two A100), `24GB..40GB:2` (two GPUs with 24–40GB), `A100:40GB:2` (two 40GB A100s). + +If vendor is omitted, `dstack` infers it from the model or defaults to `nvidia`. + + + +??? info "Shared memory" + If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure + `shm_size`, e.g. set it to `16GB`. + +> If you’re unsure which offers (hardware configurations) are available from the configured backends, use the +> [`dstack offer`](../reference/cli/dstack/offer.md#list-gpu-offers) command to list them. + + +### Spot policy + +By default, `dstack` uses on-demand instances. However, you can change that +via the [`spot_policy`](../reference/dstack.yml/service.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`. + +### `dstack` inside `dstack` + +Set `dstack` to `true` when a service needs to use the dstack CLI. dstack configures the server and +current project automatically. To run authenticated commands, pass `DSTACK_TOKEN` explicitly. + +
+ +```yaml +type: service +image: dstackai/dstack +dstack: true +port: 8000 +env: + - DSTACK_TOKEN +commands: + - dstack ps + - python -m http.server 8000 +``` + +
+ +> Besides inspecting runs, you can submit new runs with `dstack apply` and attach to them with `dstack attach`. + +### Path prefix { #path-prefix } + +If your `dstack` project doesn't have a [gateway](gateways.md), services are hosted with the +`/proxy/services///` path prefix in the URL. +When running web apps, you may need to set some app-specific settings +so that browser-side scripts and CSS work correctly with the path prefix. + +
+ +```yaml +type: service +name: dash +gateway: false + +auth: false +# Do not strip the path prefix +strip_prefix: false + +env: + # Configure Dash to work with a path prefix + # Replace `main` with your dstack project name + - DASH_ROUTES_PATHNAME_PREFIX=/proxy/services/main/dash/ + +commands: + - uv pip install dash + # Assuming the Dash app is in your repo at app.py + - python app.py + +port: 8050 +``` + +
+ +By default, `dstack` strips the prefix before forwarding requests to your service, +so to the service it appears as if the prefix isn't there. This allows some apps +to work out of the box. If your app doesn't expect the prefix to be stripped, +set [`strip_prefix`](../reference/dstack.yml/service.md#strip_prefix) to `false`. + +If your app cannot be configured to work with a path prefix, you can host it +on a dedicated domain name by setting up a [gateway](gateways.md). + ### Retry policy By default, if `dstack` can't find capacity, or the service exits with an error, or the instance is interrupted, the run will fail. @@ -1380,11 +1331,6 @@ If one replica of a multi-replica service fails with retry enabled, !!! info "Retry duration" The duration period is calculated as a run age for `no-capacity` event and as a time passed since the last `interruption` and `error` for `interruption` and `error` events. -### Spot policy - -By default, `dstack` uses on-demand instances. However, you can change that -via the [`spot_policy`](../reference/dstack.yml/service.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`. - ### Utilization policy Sometimes it’s useful to track whether a service is fully utilizing all GPUs. While you can check this with diff --git a/mkdocs/docs/concepts/tasks.md b/mkdocs/docs/concepts/tasks.md index d35f9d3c9..6c4d36b8a 100644 --- a/mkdocs/docs/concepts/tasks.md +++ b/mkdocs/docs/concepts/tasks.md @@ -15,7 +15,7 @@ A task allows you to run arbitrary commands on one or more nodes. They are best First, define a task configuration as a YAML file. The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `dev.dstack.yml` are both acceptable). -[//]: # (TODO: Make tabs - single machine & distributed tasks & web app) +[//]: # (TODO: Make tabs - single machine & multi-node tasks & web app)
@@ -74,37 +74,9 @@ Launching `axolotl-train`... `dstack apply` automatically provisions instances and runs the task. -## Configuration options - -!!! info "No commands" - If `commands` are not specified, `dstack` runs `image`’s entrypoint (or fails if none is set). - -### Ports - -A task can configure ports. In this case, if the task is running an application on a port, `dstack apply` -will securely allow you to access this port from your local machine through port forwarding. - -
- -```yaml -type: task -name: streamlit-hello + -python: 3.12 - -commands: - - uv pip install streamlit - - streamlit hello -ports: - - 8501 -``` - -
- -When running it, `dstack apply` forwards `8501` port to `localhost:8501`, enabling secure access to the running -application. - -### Distributed tasks +## Nodes By default, a task runs on a single node. However, you can run it on a cluster of nodes by specifying `nodes`. @@ -141,7 +113,7 @@ resources:
!!! info "Cluster placement" - To submit a distributed task, you must create at least one fleet with a [cluster placement](fleets.md#cluster-placement). + To submit a multi-node task, you must create at least one fleet with a [cluster placement](fleets.md#cluster-placement). Jobs on each node communicate using their private IP addresses. Use `DSTACK_MASTER_NODE_IP`, `DSTACK_NODES_IPS`, `DSTACK_NODE_RANK`, and other [system environment variables](#system-environment-variables) for inter-node communication. @@ -221,165 +193,184 @@ the index of the node within that group. > `groups[0].nodes[0]` is the run's master node — it is what `DSTACK_MASTER_NODE_IP` resolves to. -Currently, only `resources`, `commands`, and `ports` can be configured per node group. [`groups`](../reference/dstack.yml/task.md#groups) and top-level `nodes` are mutually exclusive.Support for other properties is coming soon. +Currently, only `resources`, `commands`, and `ports` can be configured per node group. [`groups`](../reference/dstack.yml/task.md#_groups) and top-level `nodes` are mutually exclusive. Support for other properties is coming soon. + +> See the [Ray+RAGEN](../examples/training/ray-ragen.md) example for running a Ray cluster, +and the [NCCL/RCCL tests](../examples/clusters/nccl-rccl-tests.md) example for running `mpirun` with node groups. -??? info "PD disaggregation" - While PD disaggregaton is mostly used with [services](services.md#pd-disaggregation), it also possible to run it as tasks. The example below runs a CPU - router (`groups[0]`, the master) and GPU workers. `startup_order: workers-first` - instructs `dstack` to start prefill and decode workers before the router. +??? info "Prototyping services" + Node groups are the task-level equivalent of a service's [replica groups](services.md#replica-groups): + as with replica groups, each node group defines its own resources, commands, and ports. + + The example below uses node groups to prototype a service. One group runs a router node and the + other group runs two worker nodes. The workers are registered with the router by referencing + their IP addresses. Since the router needs its workers to be listening before it can register + them, `startup_order: workers-first` instructs `dstack` to start the worker groups before the + master node. + + > In a service, worker registration happens automatically when the router group sets + > [`router`](services.md#router) to `type: sglang`.
```yaml type: task - name: prefill-decode - image: lmsysorg/sglang:v0.5.10.post1 + name: qwen38-flash-next + image: lmsysorg/sglang:qwen38flashnext + env: - HF_TOKEN - - MODEL_ID=zai-org/GLM-4.5-Air-FP8 + - MODEL_ID=Qwen/Qwen3.8-Flash-Next startup_order: workers-first groups: - # Router (CPU) — master node; wires prefill + decode by IP + # Router - nodes: 1 commands: - pip install smg - | - echo "prefill=${{ groups[1].nodes[0].IP_ADDRESS }}" - echo "decode=${{ groups[2].nodes[0].IP_ADDRESS }}" smg launch \ - --pd-disaggregation \ - --prefill http://${{ groups[1].nodes[0].IP_ADDRESS }}:8000 8998 \ - --decode http://${{ groups[2].nodes[0].IP_ADDRESS }}:8000 \ - --prefill-policy cache_aware \ - --host 0.0.0.0 --port 8000 + --worker-urls \ + http://${{ groups[1].nodes[0].IP_ADDRESS }}:8000 \ + http://${{ groups[1].nodes[1].IP_ADDRESS }}:8000 \ + --policy cache_aware \ + --host 0.0.0.0 \ + --port 8000 ports: - 8000 resources: cpu: 4 - - - nodes: 1 - commands: - - | - python -m sglang.launch_server \ - --model-path $MODEL_ID \ - --disaggregation-mode prefill \ - --disaggregation-transfer-backend nixl \ - --host 0.0.0.0 --port 8000 \ - --disaggregation-bootstrap-port 8998 - resources: - gpu: H200 - - - nodes: 1 + + # Workers + - nodes: 2 commands: - | python -m sglang.launch_server \ --model-path $MODEL_ID \ - --disaggregation-mode decode \ - --disaggregation-transfer-backend nixl \ - --host 0.0.0.0 --port 8000 + --tp $DSTACK_GPUS_PER_NODE \ + --ep $DSTACK_GPUS_PER_NODE \ + --mem-fraction-static 0.85 \ + --chunked-prefill-size 8192 \ + --linear-attn-prefill-backend flashinfer \ + --linear-attn-decode-backend flashinfer \ + --mamba-ssm-dtype bfloat16 \ + --reasoning-parser auto \ + --host 0.0.0.0 \ + --port 8000 resources: - gpu: H200 + gpu: H200:4 ```
-> See the [Ray+RAGEN](../examples/training/ray-ragen.md) example for running a Ray cluster, -and the [NCCL/RCCL tests](../examples/clusters/nccl-rccl-tests.md) example for running `mpirun` with node groups. + > The [`dstack-prototyping`](https://skills.sh/dstackai/dstack/dstack-prototyping) skill is designed specifically to help agents use tasks when prototyping a service. + > Install it along with the [`dstack`](https://skills.sh/dstackai/dstack/dstack) skill via `npx skills add dstackai/dstack`. -### Resources +## Retry policy -When you specify a resource value like `cpu` or `memory`, -you can either use an exact value (e.g. `24GB`) or a -range (e.g. `24GB..`, or `24GB..80GB`, or `..80GB`). +By default, if `dstack` can't find capacity, or the task exits with an error, or the instance is interrupted, +the run will fail. -
+If you'd like `dstack` to automatically retry, configure the +[retry](../reference/dstack.yml/task.md#retry) property accordingly: + + + +
```yaml type: task -name: trl-sft +name: train python: 3.12 -env: - - MODEL=Qwen/Qwen2.5-0.5B - - DATASET=stanfordnlp/imdb - commands: - - uv pip install trl - - | - trl sft \ - --model_name_or_path $MODEL --dataset_name $DATASET \ - --num_processes $DSTACK_GPUS_PER_NODE - -resources: - # 16 or more x86_64 cores - cpu: 16.. - # 200GB or more RAM - memory: 200GB.. - # 4 GPUs from 40GB to 80GB - gpu: 40GB..80GB:4 - # Shared memory (required by multi-gpu) - shm_size: 24GB - # Disk size - disk: 500GB + - uv pip install -r fine-tuning/qlora/requirements.txt + - python fine-tuning/qlora/train.py + +retry: + on_events: [no-capacity, error, interruption] + # Retry for up to 1 hour + duration: 1h ```
-The `cpu` property lets you set the architecture (`x86` or `arm`) and core count — e.g., `x86:16` (16 x86 cores), `arm:8..` (at least 8 ARM cores). -If the architecture is not set, `dstack` allows any architecture supported by the `image`, or `x86` if no `image` is set. -Since the default `dstack` image only supports `x86`, requesting `arm` requires setting `image` and is not compatible with `docker: true`. +If one job of a multi-node task fails with retry enabled, +`dstack` will stop all the jobs and resubmit the run. -The `gpu` property lets you specify vendor, model, memory, and count — e.g., `nvidia` (one NVIDIA GPU), `A100` (one A100), `A10G,A100` (either), `A100:80GB` (one 80GB A100), `A100:2` (two A100), `24GB..40GB:2` (two GPUs with 24–40GB), `A100:40GB:2` (two 40GB A100s). +!!! info "Retry duration" + The duration period is calculated as a run age for `no-capacity` event and as a time passed since the last `interruption` and `error` for `interruption` and `error` events. -If vendor is omitted, `dstack` infers it from the model or defaults to `nvidia`. +#### Priority - - Currently, only 8 TPU cores can be specified, supporting single TPU device workloads. Multi-TPU support is coming soon. --> +
-??? info "Shared memory" - If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure - `shm_size`, e.g. set it to `24GB`. +```yaml +type: task +name: train -> If you’re unsure which offers (hardware configurations) are available from the configured backends, use the -> [`dstack offer`](../reference/cli/dstack/offer.md#list-gpu-offers) command to list them. +python: 3.12 +commands: + - uv pip install -r fine-tuning/qlora/requirements.txt + - python fine-tuning/qlora/train.py -### Docker +priority: 50 +``` -#### Default image +
-If you don't specify `image`, `dstack` uses its [base](https://github.com/dstackai/dstack/tree/master/docker/base) Docker image pre-configured with - `uv`, `python`, `pip`, essential CUDA drivers, `mpirun`, and NCCL tests (under `/opt/nccl-tests/build`). +`dstack` tries to provision runs with higher priority first. +Note that if a high priority run cannot be scheduled, +it does not block other runs with lower priority from scheduling. -Set the `python` property to pre-install a specific version of Python. +## Ports + +A task can configure ports. In this case, if the task is running an application on a port, `dstack apply` +will securely allow you to access this port from your local machine through port forwarding.
```yaml type: task -name: train +name: streamlit-hello + +python: 3.12 + +commands: + - uv pip install streamlit + - streamlit hello +ports: + - 8501 +``` + +
+ +When running it, `dstack apply` forwards `8501` port to `localhost:8501`, enabling secure access to the running +application. + +## Configuration options + +### Environment variables + +
+ +```yaml +type: task +name: trl-sft python: 3.12 env: + - HF_TOKEN + - HF_HUB_ENABLE_HF_TRANSFER=1 - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb @@ -391,23 +382,48 @@ commands: --num_processes $DSTACK_GPUS_PER_NODE resources: - gpu: H100:1..2 - shm_size: 24GB + gpu: H100:1 ```
-#### NVCC +If you don't assign a value to an environment variable (see `HF_TOKEN` above), +`dstack` will require the value to be passed via the CLI or set in the current process. + + +??? info "System environment variables" + The following environment variables are available in any run by default: + + | Name | Description | + |-------------------------|------------------------------------------------------------------| + | `DSTACK_RUN_NAME` | The name of the run | + | `DSTACK_REPO_ID` | The ID of the repo | + | `DSTACK_GPUS_NUM` | The total number of GPUs in the run | + | `DSTACK_NODES_NUM` | The number of nodes in the run | + | `DSTACK_GPUS_PER_NODE` | The number of GPUs per node | + | `DSTACK_NODE_RANK` | The rank of the node | + | `DSTACK_MASTER_NODE_IP` | The internal IP address of the master node | + | `DSTACK_NODES_IPS` | The list of internal IP addresses of all nodes delimited by "\n" | + | `DSTACK_MPI_HOSTFILE` | The path to a pre-populated MPI hostfile. The file lists GPU nodes as ` slots=` and CPU nodes as `` | + | `DSTACK_WORKING_DIR` | The working directory of the run | + | `DSTACK_REPO_DIR` | The directory where the repo is mounted (if any) | + +### Docker + +#### Default image -By default, the base Docker image doesn’t include `nvcc`, which is required for building custom CUDA kernels. -If you need `nvcc`, set the [`nvcc`](../reference/dstack.yml/dev-environment.md#nvcc) property to true. +If you don't specify `image`, `dstack` uses its [base](https://github.com/dstackai/dstack/tree/master/docker/base) Docker image pre-configured with + `uv`, `python`, `pip`, essential CUDA drivers, `mpirun`, and NCCL tests (under `/opt/nccl-tests/build`). + +Set the `python` property to pre-install a specific version of Python. + +
```yaml type: task name: train python: 3.12 -nvcc: true env: - MODEL=Qwen/Qwen2.5-0.5B @@ -415,17 +431,46 @@ env: commands: - uv pip install trl - - uv pip install flash_attn --no-build-isolation - - | + - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ - --attn_implementation=flash_attention_2 \ --num_processes $DSTACK_GPUS_PER_NODE resources: - gpu: H100:1 + gpu: H100:1..2 + shm_size: 24GB ``` +
+ +??? info "NVCC" + By default, the base Docker image doesn’t include `nvcc`, which is required for building custom CUDA kernels. + If you need `nvcc`, set the [`nvcc`](../reference/dstack.yml/dev-environment.md#nvcc) property to true. + + ```yaml + type: task + name: train + + python: 3.12 + nvcc: true + + env: + - MODEL=Qwen/Qwen2.5-0.5B + - DATASET=stanfordnlp/imdb + + commands: + - uv pip install trl + - uv pip install flash_attn --no-build-isolation + - | + trl sft \ + --model_name_or_path $MODEL --dataset_name $DATASET \ + --attn_implementation=flash_attention_2 \ + --num_processes $DSTACK_GPUS_PER_NODE + + resources: + gpu: H100:1 + ``` + #### Custom image If you want, you can specify your own Docker image via `image`. @@ -462,6 +507,9 @@ resources:
+!!! info "No commands" + If `commands` are not specified, `dstack` runs `image`’s entrypoint (or fails if none is set). + #### Docker in Docker Set `docker` to `true` to enable the `docker` CLI in your task, e.g., to run or build Docker images, or use Docker Compose. @@ -522,56 +570,6 @@ resources: shm_size: 24GB ``` -### Environment variables - -
- -```yaml -type: task -name: trl-sft - -python: 3.12 - -env: - - HF_TOKEN - - HF_HUB_ENABLE_HF_TRANSFER=1 - - MODEL=Qwen/Qwen2.5-0.5B - - DATASET=stanfordnlp/imdb - -commands: - - uv pip install trl - - | - trl sft \ - --model_name_or_path $MODEL --dataset_name $DATASET \ - --num_processes $DSTACK_GPUS_PER_NODE - -resources: - gpu: H100:1 -``` - -
- -If you don't assign a value to an environment variable (see `HF_TOKEN` above), -`dstack` will require the value to be passed via the CLI or set in the current process. - - -??? info "System environment variables" - The following environment variables are available in any run by default: - - | Name | Description | - |-------------------------|------------------------------------------------------------------| - | `DSTACK_RUN_NAME` | The name of the run | - | `DSTACK_REPO_ID` | The ID of the repo | - | `DSTACK_GPUS_NUM` | The total number of GPUs in the run | - | `DSTACK_NODES_NUM` | The number of nodes in the run | - | `DSTACK_GPUS_PER_NODE` | The number of GPUs per node | - | `DSTACK_NODE_RANK` | The rank of the node | - | `DSTACK_MASTER_NODE_IP` | The internal IP address of the master node | - | `DSTACK_NODES_IPS` | The list of internal IP addresses of all nodes delimited by "\n" | - | `DSTACK_MPI_HOSTFILE` | The path to a pre-populated MPI hostfile. The file lists GPU nodes as ` slots=` and CPU nodes as `` | - | `DSTACK_WORKING_DIR` | The working directory of the run | - | `DSTACK_REPO_DIR` | The directory where the repo is mounted (if any) | - ### Working directory If `working_dir` is not specified, it defaults to the working directory set in the Docker image. For example, the [default image](#default-image) uses `/dstack/run` as its working directory. @@ -821,70 +819,106 @@ The local path can be either relative to the configuration file or absolute. Currently, you can configure up to one repo per run configuration. -### Retry policy - -By default, if `dstack` can't find capacity, or the task exits with an error, or the instance is interrupted, -the run will fail. - -If you'd like `dstack` to automatically retry, configure the -[retry](../reference/dstack.yml/task.md#retry) property accordingly: +### Resources - +When you specify a resource value like `cpu` or `memory`, +you can either use an exact value (e.g. `24GB`) or a +range (e.g. `24GB..`, or `24GB..80GB`, or `..80GB`). -
+
```yaml type: task -name: train +name: trl-sft python: 3.12 -commands: - - uv pip install -r fine-tuning/qlora/requirements.txt - - python fine-tuning/qlora/train.py +env: + - MODEL=Qwen/Qwen2.5-0.5B + - DATASET=stanfordnlp/imdb -retry: - on_events: [no-capacity, error, interruption] - # Retry for up to 1 hour - duration: 1h +commands: + - uv pip install trl + - | + trl sft \ + --model_name_or_path $MODEL --dataset_name $DATASET \ + --num_processes $DSTACK_GPUS_PER_NODE + +resources: + # 16 or more x86_64 cores + cpu: 16.. + # 200GB or more RAM + memory: 200GB.. + # 4 GPUs from 40GB to 80GB + gpu: 40GB..80GB:4 + # Shared memory (required by multi-gpu) + shm_size: 24GB + # Disk size + disk: 500GB ```
-If one job of a multi-node task fails with retry enabled, -`dstack` will stop all the jobs and resubmit the run. +The `cpu` property lets you set the architecture (`x86` or `arm`) and core count — e.g., `x86:16` (16 x86 cores), `arm:8..` (at least 8 ARM cores). +If the architecture is not set, `dstack` allows any architecture supported by the `image`, or `x86` if no `image` is set. +Since the default `dstack` image only supports `x86`, requesting `arm` requires setting `image` and is not compatible with `docker: true`. -!!! info "Retry duration" - The duration period is calculated as a run age for `no-capacity` event and as a time passed since the last `interruption` and `error` for `interruption` and `error` events. +The `gpu` property lets you specify vendor, model, memory, and count — e.g., `nvidia` (one NVIDIA GPU), `A100` (one A100), `A10G,A100` (either), `A100:80GB` (one 80GB A100), `A100:2` (two A100), `24GB..40GB:2` (two GPUs with 24–40GB), `A100:40GB:2` (two 40GB A100s). -### Priority +If vendor is omitted, `dstack` infers it from the model or defaults to `nvidia`. -Be default, submitted runs are scheduled in the order they were submitted. -When compute resources are limited, you may want to prioritize some runs over others. -This can be done by specifying the [`priority`](../reference/dstack.yml/task.md) property in the run configuration: + + ```yaml + type: task + name: train + + python: 3.12 + + commands: + - pip install -r fine-tuning/qlora/requirements.txt + - python fine-tuning/qlora/train.py + + resources: + gpu: v2-8 + ``` + + Currently, only 8 TPU cores can be specified, supporting single TPU device workloads. Multi-TPU support is coming soon. --> + +??? info "Shared memory" + If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure + `shm_size`, e.g. set it to `24GB`. + +> If you’re unsure which offers (hardware configurations) are available from the configured backends, use the +> [`dstack offer`](../reference/cli/dstack/offer.md#list-gpu-offers) command to list them. + + +### Spot policy + +By default, `dstack` uses on-demand instances. However, you can change that +via the [`spot_policy`](../reference/dstack.yml/task.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`. + +### `dstack` inside `dstack` + +Set `dstack` to `true` when a task needs to use the dstack CLI. dstack configures the server and +current project automatically. To run authenticated commands, pass `DSTACK_TOKEN` explicitly.
```yaml type: task -name: train - -python: 3.12 - +image: dstackai/dstack +dstack: true +env: + - DSTACK_TOKEN commands: - - uv pip install -r fine-tuning/qlora/requirements.txt - - python fine-tuning/qlora/train.py - -priority: 50 + - dstack ps ```
-`dstack` tries to provision runs with higher priority first. -Note that if a high priority run cannot be scheduled, -it does not block other runs with lower priority from scheduling. +> Besides inspecting runs, you can submit new runs with `dstack apply` and attach to them with `dstack attach`. ### Utilization policy @@ -968,32 +1002,6 @@ schedule: | `-` | Range of values | `0 1-3 * * *` runs at 1:00, 2:00, and 3:00 every day. | | `/` | Step values | `*/10 8-10 * * *` runs every 10 minutes during the hours 8:00 to 10:59. | -### Spot policy - -By default, `dstack` uses on-demand instances. However, you can change that -via the [`spot_policy`](../reference/dstack.yml/task.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`. - -### `dstack` inside `dstack` - -Set `dstack` to `true` when a task needs to use the dstack CLI. dstack configures the server and -current project automatically. To run authenticated commands, pass `DSTACK_TOKEN` explicitly. - -
- -```yaml -type: task -image: dstackai/dstack -dstack: true -env: - - DSTACK_TOKEN -commands: - - dstack ps -``` - -
- -> Besides inspecting runs, you can submit new runs with `dstack apply` and attach to them with `dstack attach`. - --8<-- "docs/concepts/snippets/manage-fleets.ext" !!! info "Reference" diff --git a/mkdocs/docs/concepts/volumes.md b/mkdocs/docs/concepts/volumes.md index bb338fbee..da0c0254c 100644 --- a/mkdocs/docs/concepts/volumes.md +++ b/mkdocs/docs/concepts/volumes.md @@ -172,9 +172,9 @@ and its contents will persist across runs. `dstack` will attach one of the volumes based on the region and backend of the run. - -??? info "Distributed tasks" - When using single-attach volumes such as AWS EBS with distributed tasks, + +??? info "Multi-node tasks" + When using single-attach volumes such as AWS EBS with multi-node tasks, you can attach different volumes to different nodes using `dstack` variable interpolation:
diff --git a/mkdocs/docs/examples/accelerators/amd.md b/mkdocs/docs/examples/accelerators/amd.md index f94af3d02..30fad373c 100644 --- a/mkdocs/docs/examples/accelerators/amd.md +++ b/mkdocs/docs/examples/accelerators/amd.md @@ -44,7 +44,7 @@ Kubernetes clusters or vanilla bare-metal hosts. For Kubernetes and SSH fleets, the network must be properly configured. To test whether the cluster is properly configured, run the - [RCCL tests via a distributed task](../clusters/nccl-rccl-tests.md). + [RCCL tests via a multi-node task](../clusters/nccl-rccl-tests.md). Once a fleet is created, you can run dev environments, tasks, and services. @@ -191,9 +191,9 @@ resources:
-!!! info "Distributed tasks" +!!! info "Multi-node tasks" To run training across multiple nodes, use - [distributed tasks](../../concepts/tasks.md#distributed-tasks). Distributed + [multi-node tasks](../../concepts/tasks.md#nodes). Multi-node tasks may run on a cluster; in that case, the fleet must use [cluster placement](../../concepts/fleets.md#cluster-placement). diff --git a/mkdocs/docs/examples/clusters/aws.md b/mkdocs/docs/examples/clusters/aws.md index b35e59d6c..522c4ce08 100644 --- a/mkdocs/docs/examples/clusters/aws.md +++ b/mkdocs/docs/examples/clusters/aws.md @@ -191,6 +191,6 @@ Provisioning... Instead of setting `python`, you can specify your own Docker image using `image`. Make sure that the image is properly configured for EFA. !!! info "What's next" - 1. Learn more about [distributed tasks](../../concepts/tasks.md#distributed-tasks) and [cluster placement](../../concepts/fleets.md#cluster-placement) + 1. Learn more about [multi-node tasks](../../concepts/tasks.md#nodes) and [cluster placement](../../concepts/fleets.md#cluster-placement) 2. Check [dev environments](../../concepts/dev-environments.md), [services](../../concepts/services.md), and [fleets](../../concepts/fleets.md) diff --git a/mkdocs/docs/examples/clusters/crusoe.md b/mkdocs/docs/examples/clusters/crusoe.md index 28901a8e3..b393eda92 100644 --- a/mkdocs/docs/examples/clusters/crusoe.md +++ b/mkdocs/docs/examples/clusters/crusoe.md @@ -137,7 +137,7 @@ Once the fleet is created, you can run [dev environments](../../concepts/dev-env ## NCCL tests -Use a [distributed task](../../concepts/tasks.md#distributed-tasks) that runs NCCL tests to validate cluster network bandwidth. +Use a [multi-node task](../../concepts/tasks.md#nodes) that runs NCCL tests to validate cluster network bandwidth. === "VMs" diff --git a/mkdocs/docs/examples/clusters/gcp.md b/mkdocs/docs/examples/clusters/gcp.md index eb9ddef0c..e88e292a0 100644 --- a/mkdocs/docs/examples/clusters/gcp.md +++ b/mkdocs/docs/examples/clusters/gcp.md @@ -310,13 +310,13 @@ Once you've configured the `gcp` backend, create the fleet configuration:
-Once the fleet is created, you can run distributed tasks, in addition to dev environments, services, and regular tasks. +Once the fleet is created, you can run multi-node tasks, in addition to dev environments, services, and regular tasks. ## Run tasks ### NCCL tests -Use a distributed task that runs NCCL tests to validate cluster network bandwidth. +Use a multi-node task that runs NCCL tests to validate cluster network bandwidth. === "A4" Pass the configuration to `dstack apply`: @@ -518,10 +518,10 @@ Use a distributed task that runs NCCL tests to validate cluster network bandwidt ### Distributed training === "A4" - You can use the standard [distributed task](../../concepts/tasks.md#distributed-tasks) example to run distributed training on A4 instances. + You can use the standard [multi-node task](../../concepts/tasks.md#nodes) example to run distributed training on A4 instances. === "A3 Mega" - You can use the standard [distributed task](../../concepts/tasks.md#distributed-tasks) example to run distributed training on A3 Mega instances. To enable GPUDirect-TCPX, make sure the required [NCCL environment variables](https://cloud.google.com/kubernetes-engine/docs/how-to/gpu-bandwidth-gpudirect-tcpx-autopilot#environment-variables-nccl) are properly set, for example by adding the following commands at the beginning: + You can use the standard [multi-node task](../../concepts/tasks.md#nodes) example to run distributed training on A3 Mega instances. To enable GPUDirect-TCPX, make sure the required [NCCL environment variables](https://cloud.google.com/kubernetes-engine/docs/how-to/gpu-bandwidth-gpudirect-tcpx-autopilot#environment-variables-nccl) are properly set, for example by adding the following commands at the beginning: ```shell # ... @@ -540,7 +540,7 @@ Use a distributed task that runs NCCL tests to validate cluster network bandwidt ``` === "A3 High/Edge" - You can use the standard [distributed task](../../concepts/tasks.md#distributed-tasks) example to run distributed training on A3 High/Edge instances. To enable GPUDirect-TCPX0, make sure the required [NCCL environment variables](https://cloud.google.com/kubernetes-engine/docs/how-to/gpu-bandwidth-gpudirect-tcpx-autopilot#environment-variables-nccl) are properly set, for example by adding the following commands at the beginning: + You can use the standard [multi-node task](../../concepts/tasks.md#nodes) example to run distributed training on A3 High/Edge instances. To enable GPUDirect-TCPX0, make sure the required [NCCL environment variables](https://cloud.google.com/kubernetes-engine/docs/how-to/gpu-bandwidth-gpudirect-tcpx-autopilot#environment-variables-nccl) are properly set, for example by adding the following commands at the beginning: ```shell # ... diff --git a/mkdocs/docs/examples/clusters/lambda.md b/mkdocs/docs/examples/clusters/lambda.md index 1ebe35ce7..f89c8ecaa 100644 --- a/mkdocs/docs/examples/clusters/lambda.md +++ b/mkdocs/docs/examples/clusters/lambda.md @@ -120,7 +120,7 @@ Once the fleet is created, you can run [dev environments](../../concepts/dev-env ## Run tasks -To run tasks on a cluster, you must use [distributed tasks](../../concepts/tasks.md#distributed-task). +To run tasks on a cluster, you must use [multi-node tasks](../../concepts/tasks.md#nodes). ### Run NCCL tests diff --git a/mkdocs/docs/examples/clusters/nccl-rccl-tests.md b/mkdocs/docs/examples/clusters/nccl-rccl-tests.md index e15ddcf04..8d516591b 100644 --- a/mkdocs/docs/examples/clusters/nccl-rccl-tests.md +++ b/mkdocs/docs/examples/clusters/nccl-rccl-tests.md @@ -5,10 +5,10 @@ description: Running NCCL and RCCL tests to validate cluster network bandwidth # NCCL/RCCL tests -This example shows how to run [NCCL](https://github.com/NVIDIA/nccl-tests) or [RCCL](https://github.com/ROCm/rccl-tests) tests on a cluster using [distributed tasks](../../concepts/tasks.md#distributed-tasks). +This example shows how to run [NCCL](https://github.com/NVIDIA/nccl-tests) or [RCCL](https://github.com/ROCm/rccl-tests) tests on a cluster using [multi-node tasks](../../concepts/tasks.md#nodes). !!! info "Prerequisites" - Before running a distributed task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). + Before running a multi-node task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). ## Running as a task diff --git a/mkdocs/docs/examples/clusters/nebius.md b/mkdocs/docs/examples/clusters/nebius.md index 20b1a4755..80e5f3df3 100644 --- a/mkdocs/docs/examples/clusters/nebius.md +++ b/mkdocs/docs/examples/clusters/nebius.md @@ -158,7 +158,7 @@ Once the fleet is created, you can run [dev environments](../../concepts/dev-env ## NCCL tests -Use a [distributed task](../../concepts/tasks.md#distributed-tasks) to run NCCL tests and validate the cluster’s network bandwidth. +Use a [multi-node task](../../concepts/tasks.md#nodes) to run NCCL tests and validate the cluster’s network bandwidth.
diff --git a/mkdocs/docs/examples/training/axolotl.md b/mkdocs/docs/examples/training/axolotl.md index 5d59e5802..5dd2cae0f 100644 --- a/mkdocs/docs/examples/training/axolotl.md +++ b/mkdocs/docs/examples/training/axolotl.md @@ -84,13 +84,13 @@ Provisioning... ## Distributed training !!! info "Prerequisites" - Before running a distributed task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). + Before running a multi-node task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). This section walks through running distributed fine-tuning of `Llama-3.1-70B` with QLoRA and FSDP across multiple nodes. ### Define a configuration -Once the fleet is created, define a distributed task configuration. Here's an example of a distributed `QLoRA` task using `FSDP`. +Once the fleet is created, define a multi-node task configuration. Here's an example of a multi-node `QLoRA` task using `FSDP`.
diff --git a/mkdocs/docs/examples/training/miles.md b/mkdocs/docs/examples/training/miles.md index 7ff038e25..9ec3bbda0 100644 --- a/mkdocs/docs/examples/training/miles.md +++ b/mkdocs/docs/examples/training/miles.md @@ -313,7 +313,7 @@ Using Ray via `dstack` gives you access to the Ray ecosystem while benefiting from `dstack`'s provisioning capabilities. !!! info "What's next" - 1. Read about [distributed tasks](../../concepts/tasks.md#distributed-tasks) + 1. Read about [multi-node tasks](../../concepts/tasks.md#nodes) and [fleets](../../concepts/fleets.md) 2. See the [SGLang inference](../inference/sglang.md) example 3. Browse Miles' [examples](https://github.com/radixark/miles/tree/main/examples) diff --git a/mkdocs/docs/examples/training/ray-ragen.md b/mkdocs/docs/examples/training/ray-ragen.md index d1119b36f..9df779e1c 100644 --- a/mkdocs/docs/examples/training/ray-ragen.md +++ b/mkdocs/docs/examples/training/ray-ragen.md @@ -11,7 +11,7 @@ to fine-tune an agent on multiple nodes. Under the hood `RAGEN` uses [verl](https://github.com/volcengine/verl) for Reinforcement Learning and [Ray](https://docs.ray.io/en/latest/) for distributed training. !!! info "Prerequisites" - Before running a distributed task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). + Before running a multi-node task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). ## Run a Ray cluster @@ -143,5 +143,5 @@ $ ray job submit \ Using Ray via `dstack` is a powerful way to get access to the rich Ray ecosystem while benefiting from `dstack`'s provisioning capabilities. !!! info "What's next" - 1. Read about [distributed tasks](../../concepts/tasks.md#distributed-tasks), [fleets](../../concepts/fleets.md), and [cluster placement](../../concepts/fleets.md#cluster-placement) + 1. Read about [multi-node tasks](../../concepts/tasks.md#nodes), [fleets](../../concepts/fleets.md), and [cluster placement](../../concepts/fleets.md#cluster-placement) 2. Browse Ray's [docs](https://docs.ray.io/en/latest/train/examples.html) for other examples. diff --git a/mkdocs/docs/examples/training/trl.md b/mkdocs/docs/examples/training/trl.md index e75a4d89b..11a819a1a 100644 --- a/mkdocs/docs/examples/training/trl.md +++ b/mkdocs/docs/examples/training/trl.md @@ -110,11 +110,11 @@ Provisioning... ## Distributed training !!! info "Prerequisites" - Before running a distributed task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). + Before running a multi-node task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). ### Define a configuration -Once the fleet is created, define a distributed task configuration. Here's an example using either FSDP or DeepSpeed ZeRO-3. +Once the fleet is created, define a multi-node task configuration. Here's an example using either FSDP or DeepSpeed ZeRO-3. === "FSDP" diff --git a/mkdocs/docs/guides/migration/slurm.md b/mkdocs/docs/guides/migration/slurm.md index 9edf863e9..9a1177045 100644 --- a/mkdocs/docs/guides/migration/slurm.md +++ b/mkdocs/docs/guides/migration/slurm.md @@ -139,7 +139,7 @@ Launching `train-model`... | **Output** | `--output=train-%j.out` (writes files) | `dstack logs` or UI (streams via API) | | **Working directory** | `--chdir=/path/to/dir` or defaults to submission directory | `working_dir: /path/to/dir` (defaults to image's working directory, typically `/dstack/run`) | | **Environment variables** | `export VAR` or `--export=ALL,VAR=value` | `env: - VAR` or `--env VAR=value` | -| **Node exclusivity** | `--exclusive` (entire node) | Automatic if `blocks` is not used or job uses all blocks; required for distributed tasks (`nodes` > 1) | +| **Node exclusivity** | `--exclusive` (entire node) | Automatic if `blocks` is not used or job uses all blocks; required for multi-node tasks (`nodes` > 1) | > For multi-node examples, see [Distributed training](#distributed-training) below. @@ -412,7 +412,7 @@ Slurm explicitly controls both `nodes` and processes/tasks. | N/A | `DSTACK_MPI_HOSTFILE` | Pre-populated MPI hostfile | !!! info "Fleets" - Distributed tasks may run only on a fleet with `placement: cluster` configured. Refer to [Partitions and fleets](#partitions-and-fleets) for configuration details. + Multi-node tasks may run only on a fleet with `placement: cluster` configured. Refer to [Partitions and fleets](#partitions-and-fleets) for configuration details. ## Queueing and scheduling @@ -676,7 +676,7 @@ $ sbatch --partition=gpu train.sh resources: gpu: A100:80GB:8 - # Optional: Enables inter-node connectivity; required for distributed tasks + # Optional: Enables inter-node connectivity; required for multi-node tasks placement: cluster # Optional: Split GPUs into blocks for multi-tenant sharing @@ -699,7 +699,7 @@ $ sbatch --partition=gpu train.sh type: fleet name: on-prem-gpu-fleet - # Optional: Enables inter-node connectivity; required for distributed tasks + # Optional: Enables inter-node connectivity; required for multi-node tasks placement: cluster # Optional: Allows to share the instance across up to 8 workloads @@ -868,7 +868,7 @@ srun python train.py --input=/tmp/input.txt #### Instance volumes -Instance volumes mount host directories into containers. With distributed tasks, the host can use a shared filesystem (NFS, Lustre, GPFS) to share data across jobs within the same task: +Instance volumes mount host directories into containers. With multi-node tasks, the host can use a shared filesystem (NFS, Lustre, GPFS) to share data across jobs within the same task:
@@ -936,9 +936,9 @@ resources:
-Network volumes cannot be used with distributed tasks (no multi-attach support), except where multi-attach is supported (Runpod) or via volume interpolation. +Network volumes cannot be used with multi-node tasks (no multi-attach support), except where multi-attach is supported (Runpod) or via volume interpolation. -For distributed tasks, use interpolation to attach different volumes to each node. +For multi-node tasks, use interpolation to attach different volumes to each node.
@@ -1643,7 +1643,7 @@ retry:
-For distributed tasks, if any job fails and retry is enabled, all jobs are stopped and the run is resubmitted (all-or-nothing). +For multi-node tasks, if any job fails and retry is enabled, all jobs are stopped and the run is resubmitted (all-or-nothing). Unlike Slurm, `dstack` does not support graceful shutdown signals. Applications must implement proactive checkpointing (periodic saves) and check for existing checkpoints on startup to resume after retries. diff --git a/mkdocs/docs/guides/tenant-isolation.md b/mkdocs/docs/guides/tenant-isolation.md index 10d200eb5..a55fe1f2f 100644 --- a/mkdocs/docs/guides/tenant-isolation.md +++ b/mkdocs/docs/guides/tenant-isolation.md @@ -50,7 +50,7 @@ You can disallow instance volumes or restrict access to certain paths by writing ## Host network access -By default, most `dstack` jobs run in host networking mode. This allows them to listen on any host network interface and communicate with other jobs over the internal network, which facilitates workloads such as [distributed tasks](../concepts/tasks.md#distributed-tasks) or [services with routers](../concepts/services.md#pd-disaggregation). +By default, most `dstack` jobs run in host networking mode. This allows them to listen on any host network interface and communicate with other jobs over the internal network, which facilitates workloads such as [multi-node tasks](../concepts/tasks.md#nodes) or [services with routers](../concepts/services.md#router). However, exposing the host network to the job also exposes internal `dstack` APIs used to manage containers and SSH authorized keys on the host. If this is not acceptable, bridge networking should be used, which isolates the job from the host network. Bridge networking, however, breaks workloads that do need inter-job communication. @@ -58,13 +58,13 @@ The `DSTACK_SERVER_JOB_NETWORK_MODE` environment variable controls which jobs ge | Value | Name | Behavior | |---|---|---| -| `1` | `HOST_FOR_MULTINODE_ONLY` | Host for distributed tasks, bridge otherwise | +| `1` | `HOST_FOR_MULTINODE_ONLY` | Host for multi-node tasks, bridge otherwise | | `2` | `HOST_WHEN_POSSIBLE` | Host whenever the job occupies a full instance (default) | -| `3` | `FORCED_BRIDGE` | Always bridge, including distributed tasks | +| `3` | `FORCED_BRIDGE` | Always bridge, including multi-node tasks | -### No distributed tasks +### No multi-node tasks -If you don't need distributed tasks or other runs with inter-job communication, you can set `DSTACK_SERVER_JOB_NETWORK_MODE=3` when starting the server: +If you don't need multi-node tasks or other runs with inter-job communication, you can set `DSTACK_SERVER_JOB_NETWORK_MODE=3` when starting the server: ```shell DSTACK_SERVER_JOB_NETWORK_MODE=3 @@ -72,6 +72,6 @@ DSTACK_SERVER_JOB_NETWORK_MODE=3 This forces bridge networking for all jobs on the server without exception, preventing access to internal `dstack` APIs, as well as communication between jobs. -### Allow distributed tasks in selected projects +### Allow multi-node tasks in selected projects -If you want distributed tasks or other runs with inter-job communication to be available in some projects but not others, use `DSTACK_SERVER_JOB_NETWORK_MODE=1` instead. With this mode, single-node jobs get bridge networking, while distributed tasks still run with host networking. Distributed tasks can then be selectively blocked per project or user by writing a [REST plugin](../reference/plugins/rest/index.md) or a [Python plugin](../reference/plugins/python/index.md). +If you want multi-node tasks or other runs with inter-job communication to be available in some projects but not others, use `DSTACK_SERVER_JOB_NETWORK_MODE=1` instead. With this mode, single-node jobs get bridge networking, while multi-node tasks still run with host networking. Multi-node tasks can then be selectively blocked per project or user by writing a [REST plugin](../reference/plugins/rest/index.md) or a [Python plugin](../reference/plugins/python/index.md). diff --git a/mkdocs/docs/guides/troubleshooting.md b/mkdocs/docs/guides/troubleshooting.md index f74504b30..bcfeeeea0 100644 --- a/mkdocs/docs/guides/troubleshooting.md +++ b/mkdocs/docs/guides/troubleshooting.md @@ -133,7 +133,7 @@ one of these features, `dstack` will only select offers from the backends that s and [Privileged containers](../reference/dstack.yml/dev-environment.md#privileged) are supported by all backends except `runpod`, `vastai`, and `kubernetes`. - [Clusters](../concepts/fleets.md#cluster-placement) - and [distributed tasks](../concepts/tasks.md#distributed-tasks) + and [multi-node tasks](../concepts/tasks.md#nodes) are only supported by the `aws`, `azure`, `gcp`, `nebius`, `oci`, and `vultr` backends, as well as SSH fleets. - [Reservations](../reference/dstack.yml/fleet.md#reservation) diff --git a/mkdocs/docs/guides/upgrade.md b/mkdocs/docs/guides/upgrade.md index 4cced2d51..7ccd62343 100644 --- a/mkdocs/docs/guides/upgrade.md +++ b/mkdocs/docs/guides/upgrade.md @@ -26,7 +26,7 @@ If you call the `dstack` HTTP API directly, note that UTC datetimes are now seri ### Gateway routers -The top-level `router` property of gateway and run configurations, deprecated in `0.20.17` in favor of [replica-based routers](../concepts/services.md#pd-disaggregation), has been removed. Configurations that use it are no longer accepted, and the behavior of gateways and services created with it before the upgrade is undefined. +The top-level `router` property of gateway and run configurations, deprecated in `0.20.17` in favor of [replica-based routers](../concepts/services.md#router), has been removed. Configurations that use it are no longer accepted, and the behavior of gateways and services created with it before the upgrade is undefined. > Terminate services and gateways that use the top-level `router` property before upgrading, then recreate them using replica-based routers. diff --git a/mkdocs/docs/quickstart.md b/mkdocs/docs/quickstart.md index d52392ee4..9a9b4a611 100644 --- a/mkdocs/docs/quickstart.md +++ b/mkdocs/docs/quickstart.md @@ -152,7 +152,7 @@ description: Quick guide to creating fleets and submitting runs Open the link to access the dev environment using your desktop IDE. Alternatively, you can access it via `ssh `. === "Task" - A [task](concepts/tasks.md) allows you to schedule a job or run a web app. Tasks can be distributed and can forward ports. + A [task](concepts/tasks.md) allows you to schedule a job or run a web app. Tasks can run on multiple nodes and forward ports. Create the following run configuration: @@ -181,8 +181,8 @@ description: Quick guide to creating fleets and submitting runs
- By default, tasks run on a single instance. To run a distributed task, specify - [`nodes`](concepts/tasks.md#distributed-tasks), and `dstack` will run it on a cluster. + By default, tasks run on a single instance. To run a multi-node task, specify + [`nodes`](concepts/tasks.md#nodes) or define multiple [node groups](concepts/tasks.md#node-groups), and `dstack` will run it on a cluster. Run the configuration via `dstack apply`: @@ -243,6 +243,9 @@ description: Quick guide to creating fleets and submitting runs
+ By default, a service runs a single replica. To run multiple replicas, specify + [`replicas`](concepts/services.md#replicas) or define multiple [replica groups](concepts/services.md#replica-groups), and `dstack` will run them behind a single endpoint. + Run the configuration via `dstack apply`:
diff --git a/mkdocs/docs/reference/env.md b/mkdocs/docs/reference/env.md index 9b1c9f17d..829001c41 100644 --- a/mkdocs/docs/reference/env.md +++ b/mkdocs/docs/reference/env.md @@ -155,7 +155,7 @@ For more details on the options below, refer to the [server deployment](../guide - `DSTACK_SSHPROXY_API_TOKEN`{ #DSTACK_SSHPROXY_API_TOKEN } – Authentication token for the SSH proxy API. Required to enable SSH proxy integration; must match the token configured when deploying [`dstack-sshproxy`](https://github.com/dstackai/sshproxy). - `DSTACK_SERVER_SSHPROXY_ADDRESS`{ #DSTACK_SERVER_SSHPROXY_ADDRESS } – Address of the SSH proxy exposed to users, in `HOSTNAME[:PORT]` form. `PORT` defaults to `22` if omitted. Required together with `DSTACK_SSHPROXY_API_TOKEN` to enable SSH proxy integration. - `DSTACK_SERVER_SSHPROXY_ENFORCED`{ #DSTACK_SERVER_SSHPROXY_ENFORCED } – When set to any value, restricts all SSH connections to go through the SSH proxy. -- `DSTACK_SERVER_JOB_NETWORK_MODE`{ #DSTACK_SERVER_JOB_NETWORK_MODE } – Controls the network mode assigned to jobs. Accepts an integer value: `1` forces bridge networking for single-node jobs while distributed tasks still use host networking; `2` uses host networking whenever the job occupies a full instance (default); `3` forces bridge networking for all jobs including distributed tasks. +- `DSTACK_SERVER_JOB_NETWORK_MODE`{ #DSTACK_SERVER_JOB_NETWORK_MODE } – Controls the network mode assigned to jobs. Accepts an integer value: `1` forces bridge networking for single-node jobs while multi-node tasks still use host networking; `2` uses host networking whenever the job occupies a full instance (default); `3` forces bridge networking for all jobs including multi-node tasks. - `DSTACK_SERVER_SSH_CONNECT_TIMEOUT`{ #DSTACK_SERVER_SSH_CONNECT_TIMEOUT } – The SSH `ConnectTimeout` for server-instance connections, in seconds. Defaults to `3`. Increase if there are high-latency links between the server and instances. - `DSTACK_SERVER_SSH_POOL_DISABLED`{ #DSTACK_SERVER_SSH_POOL_DISABLED } – Disables the reuse of server SSH connections to instances. If set, significantly decreases server RAM usage, but slows down processing and may cause CPU spikes due to frequent SSH-connection establishment. diff --git a/skills/dstack/SKILL.md b/skills/dstack/SKILL.md index d9bf1e7b7..a16851fe7 100644 --- a/skills/dstack/SKILL.md +++ b/skills/dstack/SKILL.md @@ -148,11 +148,11 @@ If background attach fails in the sandbox (permissions writing `~/.dstack` or `~ **"Connect to" or "open" a dev environment:** If a dev environment is already running, use `dstack attach --logs` (agent runs it in the background by default) to surface the IDE URL (`cursor://`, `vscode://`, etc.) and SSH alias. If sandboxed attach fails, request escalation or ask the user to run attach locally and share the link. -### Distributed tasks and multi-replica services +### Multi-node tasks and multi-replica services -Unless you use **Distributed tasks** (see `### 2. Tasks`) or **Multi-replica services** (see `### 3. Services`), both tasks and services run on a single node. That's why `dstack logs `, `dstack attach `, and `ssh ` default to the first replica/job. +Unless you use **Multi-node tasks** (see `### 2. Tasks`) or **Multi-replica services** (see `### 3. Services`), both tasks and services run on a single node. That's why `dstack logs `, `dstack attach `, and `ssh ` default to the first replica/job. -- In a distributed task, each node runs its own job, numbered from 0 in order across node groups. Target a node via `dstack logs --job 1` or `dstack attach --job 1`. +- In a multi-node task, each node runs its own job, numbered from 0 in order across node groups. Target a node via `dstack logs --job 1` or `dstack attach --job 1`. - In a multi-replica service, replicas are numbered from 0 in order across replica groups. Target a replica via `dstack logs --replica 1` or `dstack attach --replica 1`. - Attaching with a non-zero `--job` or `--replica` creates the SSH alias `ssh --`. @@ -230,7 +230,7 @@ resources: **Port forwarding:** When you specify `ports`, `dstack apply` forwards them to `localhost` while attached. Use `dstack attach ` to reconnect and restore port forwarding. The run name becomes an SSH alias (e.g., `ssh `) for direct access. -**Distributed tasks:** Set `nodes` to run a task across multiple nodes, or use `groups` to define node groups, each with its own `nodes` count, `resources`, `commands`, and `ports` (`groups` and top-level `nodes` are mutually exclusive). Requires a fleet that supports inter-node communication (see `placement: cluster` in fleets). +**Multi-node tasks:** Set `nodes` to run a task across multiple nodes, or use `groups` to define node groups, each with its own `nodes` count, `resources`, `commands`, and `ports` (`groups` and top-level `nodes` are mutually exclusive). Requires a fleet that supports inter-node communication (see `placement: cluster` in fleets). [Concept documentation](https://dstack.ai/docs/concepts/tasks.md) | [Configuration reference](https://dstack.ai/docs/reference/dstack.yml/task.md)