Add draft endpoint prototyping loop

[peterschmidt85]

What This PR Does

Adds a draft endpoint configuration type for model-serving endpoints.

This is a draft PR for reviewing the endpoint UX and the efficiency of the prototyping loop. The code is intentionally still subject to review and rewrite; the goal now is to validate whether the workflow, CLI surface, agent behavior, logs, and preset reuse model feel right before polishing internals.

Main pieces:

• Adds type: endpoint with model, preset_policy, and normal profile constraints such as fleets, backends, spot_policy, and max_price.
• Adds endpoint CLI/API/server lifecycle: create, list, get, stop, logs, and endpoint presets.
• Adds project-local endpoint presets: model-level recipes that can be reused without invoking the agent.
• Adds a Claude Code based endpoint agent for preset_policy: create / reuse-or-create when no usable preset is available.
• Adds endpoint progress logs so users can follow the agent’s decisions while it prototypes and deploys.
• Adds tests for endpoint planning, lifecycle, CLI output, presets, agent report handling, and restart/session behavior.

Preset mechanics are implemented and validated on initial e2e scenarios, but recipe quality and selection still need broader testing.

How To Use

Start the server with the endpoint agent enabled:

export DSTACK_AGENT_ANTHROPIC_API_KEY=...
uv run dstack server --port 3000

For local development only, the server can use the existing Claude CLI login instead of DSTACK_AGENT_ANTHROPIC_API_KEY:

export DSTACK_AGENT_CLAUDE_USE_EXISTING_AUTH=1
uv run dstack server --port 3000

Production servers must use DSTACK_AGENT_ANTHROPIC_API_KEY. Do not set DSTACK_AGENT_CLAUDE_USE_EXISTING_AUTH together with DSTACK_AGENT_ANTHROPIC_API_KEY; the endpoint agent treats that as a configuration error.

Optionally pass Claude CLI effort for endpoint agent sessions:

export DSTACK_AGENT_CLAUDE_EFFORT=high

Supported values are low, medium, high, xhigh, and max. If unset, dstack does not pass --effort and Claude CLI uses its default.

Create or select a project with at least one active fleet. Endpoint creation does not create fleets.

Example endpoint config:

type: endpoint
name: qwen25-docs-gate-e2e

model: Qwen/Qwen2.5-0.5B-Instruct
preset_policy: create

fleets:
  - probe-quality-guard

spot_policy: auto
max_price: 1.5

Apply it:

uv run dstack apply --project <project> -f endpoint.dstack.yml

Watch endpoint progress:

uv run dstack endpoint logs --project <project> qwen25-docs-gate-e2e -w

Inspect the endpoint:

uv run dstack endpoint get --project <project> qwen25-docs-gate-e2e --json

List learned presets:

uv run dstack endpoint preset --project <project>
uv run dstack endpoint preset get --project <project> Qwen/Qwen2.5-0.5B-Instruct --json

Stop the endpoint:

uv run dstack endpoint stop --project <project> qwen25-docs-gate-e2e -y

Example

This is an excerpt from the latest successful e2e run.

Endpoint config:

type: endpoint
name: qwen25-docs-gate-e2e

model: Qwen/Qwen2.5-0.5B-Instruct
preset_policy: create

fleets:
  - probe-quality-guard

spot_policy: auto
max_price: 1.5

Endpoint progress:

[2026-07-08 14:50:02] Starting endpoint prototyping agent for Qwen/Qwen2.5-0.5B-Instruct. Allowed fleets: probe-quality-guard. The agent will inspect offers, choose a service recipe, deploy it, and verify the model API before the endpoint becomes active.
[2026-07-08 14:51:24] Chose backend jarvislabs on fleet probe-quality-guard, L4:24GB offer at $0.44 (available), resources gpu nvidia 16GB..1. jarvislabs is VM-based (supports idle instances + instance volumes); did not choose runpod which is container-based (no idle/instance volumes). Submitting prototyping task qwen25-docs-gate-e2e-1 with vllm image to test serving Qwen2.5-0.5B-Instruct before the service.
[2026-07-08 14:58:34] Task qwen25-docs-gate-e2e-1 verified: vllm/vllm-openai image, 'vllm serve Qwen/Qwen2.5-0.5B-Instruct --port 8000 --max-model-len 4096' on jarvislabs L4 returned correct chat output. Stopping task and submitting the same recipe as service qwen25-docs-gate-e2e-2.
[2026-07-08 14:59:13] Instance now idle in fleet probe-quality-guard; submitting service qwen25-docs-gate-e2e-2 to reuse it.
[2026-07-08 15:01:37] Service qwen25-docs-gate-e2e-2 verified: /v1/chat/completions for Qwen/Qwen2.5-0.5B-Instruct returned 'Tokyo' through the dstack service URL. Endpoint deployment successful.

Backlog for v0 (WIP)

• Broader e2e coverage across larger models, especially Qwen 27B.
• More testing for preset quality, recipe ordering, and reuse under capacity churn.
• Stronger crash/restart reconciliation tests around final report handoff, submitted runs, and multi-server locking.
• More validation of task-first prototyping across VM, SSH, Kubernetes, and container backends.
• Raw Claude traces in debug mode, without mixing them into normal endpoint progress logs.
• Docker/server packaging validation for the bundled agent runtime.
• Public docs and UX polish after the draft behavior stabilizes.

Out Of Scope For V1

• Endpoint-level resources overrides.
• Agent budget/cost governance.
• Recipe ranking or explicit recipe selection.
• Frontend UI.
• Autoscaling optimization, benchmarking, and metrics collection.
• Fixing drivers, generating custom kernels, or doing deep metrics-driven optimization.
• P/D disaggregation and distributed serving.
• Using host CUDA/driver versions for offer matching, because offers/fleets do not expose that today.
• Letting the agent target specific fleet instances for follow-up runs when it intentionally wants to reuse a warmed instance or instance volumes.

Notes On Implementation

• The endpoint agent currently invokes the Claude CLI instead of the Claude Python SDK because the SDK depends on Pydantic v2 while dstack still uses Pydantic v1.
• The production path uses DSTACK_AGENT_ANTHROPIC_API_KEY and keeps Claude in --bare mode.
• DSTACK_AGENT_CLAUDE_USE_EXISTING_AUTH=1 is a local-development-only fallback. It is mutually exclusive with DSTACK_AGENT_ANTHROPIC_API_KEY; setting both is an endpoint agent configuration error.
• DSTACK_AGENT_CLAUDE_EFFORT passes Claude CLI --effort for endpoint agent sessions. If unset, Claude CLI uses its default.
• When no API key is set and existing-auth mode is enabled, Claude runs without --bare so it can read the server user's existing Claude CLI auth and settings; the server logs a warning when this mode is used.
• In existing-auth mode, Claude runs with the server user's real HOME, and the agent installs workspace-local dstack and ssh wrappers so dstack CLI config and SSH aliases still use the endpoint workspace home.
• Agent process abort handling uses POSIX process groups when available and falls back to signaling the stored process id on platforms without os.killpg; this keeps the Windows pyright matrix clean without changing POSIX behavior.
• Both auth modes were e2e-tested on Qwen/Qwen2.5-0.5B-Instruct: existing Claude CLI auth and DSTACK_AGENT_ANTHROPIC_API_KEY.

AI Assistance

This PR was developed with AI assistance from Codex.