Add orphan cleanup for cloud resources

AGENTS.md

@@ -2,8 +2,8 @@
 
 Before touching a subsystem, read the relevant notes in `contributing/`: `ARCHITECTURE.md`,
 `PIPELINES.md`, `LOCKING.md`, `DATABASE.md`, `MIGRATIONS.md`, `RUNS-AND-JOBS.md`, `AUTOSCALING.md`,
-`BACKENDS.md`, `GPUHUNT.md`, `PROXY.md`, `RUNNER-AND-SHIM.md`, `FRONTEND.md`, `DOCS.md`,
-`DEVELOPMENT.md`, `RELEASE.md`.
+`BACKENDS.md`, `ORPHAN-CLEANUP.md`, `GPUHUNT.md`, `PROXY.md`, `RUNNER-AND-SHIM.md`, `FRONTEND.md`,
+`DOCS.md`, `DEVELOPMENT.md`, `RELEASE.md`.
 
 ## Project Structure & Module Organization
 - Core Python package lives in `src/dstack`; internal modules (including server) sit under `_internal`, API surfaces under `api`, and plugin integrations under `plugins`.

contributing/BACKENDS.md

@@ -144,6 +144,7 @@ Refer to examples:
 
 Go to `compute.py` and implement `Compute` methods.
 Optionally, extend and implement `ComputeWith*` classes to support additional features such as fleets, volumes, gateways, placement groups, etc. For example, extend `ComputeWithCreateInstanceSupport` to support fleets.
+To support the cleanup of orphaned resources, extend `ComputeWithResourceListingSupport` as described in [Orphan cleanup support](#32-orphan-cleanup-support).
 
 Refer to examples:
 [verda](https://github.com/dstackai/dstack/blob/master/src/dstack/_internal/core/backends/verda/compute.py),
@@ -228,3 +229,36 @@ To support a container-based backend, `dstack` expects the following:
 - A way to override the container user to root (as in `docker run --user root ...`)
 
 Examples of container-based backends include: `kubernetes`, `vastai`, `runpod`.
+
+### 3.2. Orphan cleanup support
+
+`dstack` can find and delete cloud resources it created but no longer tracks, e.g. if the server
+crashed after the provider created an instance but before it was recorded, or if instance
+termination failed. The `OrphanCleanupPipeline` lists resources of backends that extend
+`ComputeWithResourceListingSupport` and matches them with instances by `provider_ref`.
+See [ORPHAN-CLEANUP.md](ORPHAN-CLEANUP.md) for the server side.
+
+To support it in a backend:
+
+1. Set `provider_ref` on every resource created for an instance or a job. It's
+   `InstanceConfiguration.provider_ref` in `create_instance()` and `get_job_provider_ref(job)` in
+   `run_job()`. Internal resources that outlive the instance if left behind (disks, keys, IPs)
+   need it too, while resources deleted together with the instance don't.
+    * If the provider supports tags, set the `PROVIDER_REF_TAG_KEY` tag and keep instance names as is.
+      If the provider does not allow this key, use another one.
+    * Otherwise, encode `provider_ref` in names with `generate_unique_instance_name(..., with_provider_ref=True)`
+      and parse it with `parse_provider_ref_from_name()`.
+    * Do not set it on resources of `run_jobs()` since orphan cleanup does not support compute groups yet.
+2. Implement `list_resource_groups()`. It returns a `ListedResourceGroup` per provisioning attempt: the instance
+   (`primary_id`) together with internal resources created for it. Skip resources without `provider_ref`.
+   List resources of each region all or nothing: if any listing in a region fails, return no groups
+   of the region and report it in `ResourceListing.failures`. Otherwise, a tracked instance's resources
+   may be listed without the instance and deleted. Skip instances whose deletion is in progress
+   (e.g. shutting down), but list their internal resources.
+3. Implement `delete_resource_group()`. Delete everything in the group in the order the provider requires.
+   The method must be idempotent and never delete resources that don't carry the group's `provider_ref`,
+   such as attached volumes. If the deletion takes several steps, delete the instance first and raise
+   `NotYetTerminated` instead of waiting for the next step, e.g. for the instance to be gone before its
+   disk can be deleted. Next cleanups list the remaining internal resources and call the method again.
+
+Refer to the [aws](https://github.com/dstackai/dstack/blob/master/src/dstack/_internal/core/backends/aws/compute.py) example.

contributing/MIGRATIONS.md

@@ -73,3 +73,24 @@ with op.get_context().autocommit_block():
         postgresql_concurrently=True,
     )
 ```
+
+Keep concurrent index creation in a separate migration. Do not combine `autocommit_block()`
+with other schema changes such as `add_column`/`batch_alter_table` in the same migration.
+`autocommit_block()` commits the migration's transaction before running the index statements,
+so a failure inside the block leaves the earlier changes committed while the migration is not
+marked as applied. A retry then re-runs those changes and fails. For example:
+
+```python
+def upgrade() -> None:
+    with op.batch_alter_table("table", schema=None) as batch_op:
+        batch_op.add_column(sa.Column("col", sa.String(length=12), nullable=True))
+    with op.get_context().autocommit_block():
+        # If this fails, "col" is already committed. A retry fails on add_column
+        # with "column already exists".
+        op.create_index("ix_table_col", "table", ["col"], postgresql_concurrently=True)
+```
+
+Instead, add the column in one migration and create the index in the next one.
+
+Indexes on a table created in the same migration do not need `CONCURRENTLY`, since the table is
+empty. Create them without `autocommit_block()` so that the whole migration stays transactional.

contributing/ORPHAN-CLEANUP.md

@@ -0,0 +1,57 @@
+# Orphan cleanup
+
+`dstack` can leave cloud resources it no longer tracks: the server crashes or loses its lock between a provider call that creates resources and the commit that records them, the provider returns an error after creating them, or termination gives up after retrying. Orphan cleanup finds such resources by listing them in the provider and matching them with the DB, then reports or deletes them.
+
+How a backend supports it is described in [BACKENDS.md](BACKENDS.md#32-orphan-cleanup-support). This doc covers the server side.
+
+## `provider_ref`
+
+Each record that provisions resources gets a `provider_ref` (12-char string). Backends set it on the resources they create (as a tag or in names) and report it back when listing.
+
+- Only instances own `provider_ref`s (`InstanceModel.provider_ref`). Fleet instances get it on creation, for `create_instance()`. Placeholder instances, created for job submissions before `run_job()`, get it on creation too, pass it to `run_job()` (`JobSubmission.provider_ref`), and keep it when promoted.
+- Every issued `provider_ref` is also recorded in `ProviderRefModel`, committed with its owner before any provider call. Records are not deleted with owners, so resources of a deleted owner are still recognized. This matters for placeholder instances, which are deleted when a `run_job()` fails, possibly after creating resources.
+- Compute groups (`run_jobs()`) are not supported: their instances get no `provider_ref`, and backends must not set it on compute group resources.
+
+## Cleanup unit and enrollment
+
+An `OrphanCleanupModel` row exists per project backend (`BackendModel`) that supports resource listing. A scheduled task periodically creates rows in batches for backends of projects that have issued `provider_ref`s. Enrolling periodically instead of in a migration covers backends created by any replica, including older replicas during rolling deployments.
+
+## `OrphanCleanupPipeline`
+
+The pipeline processes each row at most once per cleanup interval (`DSTACK_SERVER_ORPHAN_CLEANUP_INTERVAL_SECONDS`). Failed cleanups are retried with an increasing delay.
+
+Processing a row:
+
+- STEP 1: Check whether the backend may have new orphans: it was never cleaned up, the cleanup mode changed, or the project had instance or job activity since the last clean cleanup (`clean_at`). If not, the row is only marked processed. Checking activity here rather than in the fetch query keeps fetching cheap regardless of project sizes, at the cost of a small query and row update per enrolled backend per interval.
+- STEP 2: List the backend's resource groups (`list_resource_groups()`). A resource group is one provisioning attempt as found in the provider: the owner resource (`primary_id`, the instance) and internal resources created with it.
+- STEP 3: Load the project's `ProviderRefModel` records and their instances (including soft-deleted ones) by `provider_ref`, in one statement. An instance is committed with its record, so it can't be missed while the record is seen.
+- STEP 4: Classify each group:
+    - Instance found:
+        - `PENDING`, `TERMINATING`: skip, since the resources may be in flight. An error is logged if a pending instance's resources look stuck.
+        - `TERMINATED`: orphan.
+        - `PROVISIONING`, `IDLE`, `BUSY`: tracked if the base backend and `primary_id` match the instance's provisioning data, orphan otherwise.
+        - Without provisioning data: ignored with an error log.
+    - Record found without an instance: orphan. The owner was deleted, e.g. a placeholder instance after a failed `run_job()`, so nothing can be creating its resources anymore.
+    - No record: ignored. Resources of other servers and projects are never touched.
+- STEP 5: Owners that were provisioned or terminated around the listing are skipped instead of classified as orphans, since the listing may predate the change. The margin covers clock skew between replicas, the lag between setting timestamps and committing them, and providers listing new resources with a delay.
+- STEP 6: Handle orphans according to `DSTACK_SERVER_ORPHAN_CLEANUP_MODE`:
+    - `report` (default): log, send to Sentry, and emit an event targeting the owner instance. If the instance was deleted, the event still targets it by the id and name kept in `ProviderRefModel`, so it shows next to the instance's earlier events.
+    - `delete`: call `delete_resource_group()` and emit an event. If the backend raises `NotYetTerminated` for a group with an owner resource, the owner's deletion was requested and the event says the resources are being deleted. The remaining internal resources are listed and deleted by next cleanups.
+    - `off`: the pipeline and enrollment are disabled.
+- STEP 7: Set the outcome:
+    - Clean (everything listed, nothing skipped, all orphans handled): `clean_at` advances to the listing time minus the margin, and the retry delay is reset.
+    - Unsettled (a group skipped, a deletion in progress, or some regions in `ResourceListing.failures`): `clean_at` stays, the retry delay is reset, and the backend is retried after the interval. Listed groups are still handled.
+    - Failed (listing raised or a deletion failed): `clean_at` stays and the retry is backed off.
+
+Events are emitted in the lock-guarded apply, so a worker that lost the lock emits nothing.
+
+## Termination give-up
+
+When instance termination gives up after retrying, the log says that orphan cleanup will retry deleting the instance if the mode is `delete` and the backend supports listing. Otherwise, it asks to terminate the instance manually.
+
+## Limitations
+
+- Resources are not cleaned up if the backend credentials change or the backend is deleted.
+- Compute groups, placement groups, volumes and gateways are not covered. `OwnerType` only has `INSTANCE`.
+- `ProviderRefModel` records are never deleted yet. There is one per fleet instance and per placeholder instance, i.e. per job submission that provisions new capacity.
+- On dstack Sky, each project's `dstack` backend lists the shared base backend accounts, so listings repeat per project. Cleaning up each shared account once is a possible optimization.

mkdocs/docs/reference/env.md

@@ -151,6 +151,8 @@ For more details on the options below, refer to the [server deployment](../guide
 - `DSTACK_SERVER_GATEWAY_MAX_REPLICAS`{ #DSTACK_SERVER_GATEWAY_MAX_REPLICAS } - Maximum number of replicas allowed for a gateway. Defaults to `9`.
 - `DSTACK_SERVER_INSTANCE_HEALTH_MIN_COLLECT_INTERVAL_SECONDS`{ #DSTACK_SERVER_INSTANCE_HEALTH_MIN_COLLECT_INTERVAL_SECONDS } – Minimum time interval between consecutive health checks of the same instance.
 - `DSTACK_SERVER_EVENTS_TTL_SECONDS`{ #DSTACK_SERVER_EVENTS_TTL_SECONDS } - Maximum age of event records. Set to `0` to disable event storage. Defaults to 30 days.
+- `DSTACK_SERVER_ORPHAN_CLEANUP_MODE`{ #DSTACK_SERVER_ORPHAN_CLEANUP_MODE } – Controls the cleanup of cloud resources that `dstack` created but no longer tracks, e.g. after the server crashed during provisioning or failed to terminate an instance. `report` emits events about such resources, `delete` also deletes them, and `off` disables the cleanup. Defaults to `report`. Currently supported for the `aws` backend.
+- `DSTACK_SERVER_ORPHAN_CLEANUP_INTERVAL_SECONDS`{ #DSTACK_SERVER_ORPHAN_CLEANUP_INTERVAL_SECONDS } – Minimum interval between orphan cleanups of the same backend. A backend is cleaned up only after instances were provisioned or terminated in its project. Defaults to `3600`.
 - `DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY`{ #DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY } – A default Docker registry to use for job images that do not specify an explicit registry. E.g., if set to `registry.example`, then `image: ubuntu` becomes equivalent to `image: registry.example/ubuntu`. **Note**: This setting should only be used for configuring registries that act as a pull-through cache for Docker Hub. The default `dstack` images are also pulled from the configured registry.
 - `DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY_USERNAME`{ #DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY_USERNAME } – Username for authenticating with the default Docker registry. See `DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY_PASSWORD`.
 - `DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY_PASSWORD`{ #DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY_PASSWORD } – Password for authenticating with the default Docker registry. Applied only when the image has no explicit registry and the run configuration does not specify `registry_auth`. **Note**: The value may be visible to anyone who can SSH into instances managed by `dstack`, which usually includes all users of that `dstack` server.