From a9be2055a7f0e07bbcdeea97dc6b2f12256e96ce Mon Sep 17 00:00:00 2001 From: Felipe Ocampo Osorio Date: Mon, 7 Sep 2026 11:54:28 -0400 Subject: [PATCH] docs: add Modal GPU serving guide How we used Modal as a drop-in replacement for MIT SLURM GPU partitions during the BODHI-Medcalc rerun campaign (Sept 2026), when MIT's GPU QOS was maxed out. Covers the one-class-per-model vLLM serving pattern and the seven gotchas that cost real time: silent GPU-quota exhaustion without max_containers, TP=2+AWQ hangs, incomplete client venvs, the checkpoint-hash relaunch trap, per-model SLURM time limits, HF Hub rate-limiting on concurrent dataset loads, and NFS cache races. Co-Authored-By: Claude Sonnet 5 --- docs/MODAL_GPU_SERVING.md | 225 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 docs/MODAL_GPU_SERVING.md diff --git a/docs/MODAL_GPU_SERVING.md b/docs/MODAL_GPU_SERVING.md new file mode 100644 index 0000000..86a5daa --- /dev/null +++ b/docs/MODAL_GPU_SERVING.md @@ -0,0 +1,225 @@ +# Serving vLLM on Modal instead of MIT SLURM GPUs + +Written after using [Modal](https://modal.com) as a drop-in replacement for MIT's SLURM GPU +partitions during the BODHI-Medcalc rerun campaign (Sept 2026), when the MIT GPU QOS was maxed +out (`QOSMaxGRESPerUser`). Modal has its own, separate GPU pool, so it let us keep running while +MIT's queue was full — and once it was set up, we could run all 4 model families **in parallel**, +which we couldn't do on the shared MIT partition. + +This is not a Modal tutorial — it's the specific pattern we used, plus every gotcha we hit, +so the next person doesn't have to rediscover them. + +## The core idea + +Split the job in two, and keep them talking over plain HTTP: + +- **The GPU server** (vLLM, OpenAI-compatible) runs on Modal. +- **The client** (your existing eval script) runs wherever it already runs (MIT SLURM, + a laptop, whatever) and just points `OPENAI_COMPATIBLE_BASE_URL` at the Modal endpoint. + +If your client code already talks to an OpenAI-compatible server (most of our eval harnesses do), +**zero client-side code changes are needed.** Only the serving side moves. + +## Setup (one time) + +```bash +pip install modal +modal setup # opens a browser, links your Modal account/workspace +``` + +You need a Modal workspace with GPU access (ask whoever owns the org's Modal account to add you). + +## The server: one `@app.cls` per model + +Don't try to build one generic parameterized class for every model — `@modal.web_server` needs a +fixed port and label at decoration time, so a parameterized class doesn't give you stable, +predictable URLs. Write one small class per model instead. It's more typing but it's boring and +it works. + +```python +import subprocess, time, urllib.request, urllib.error +import modal + +app = modal.App("my-vllm-servers") +VLLM_PORT = 8000 + +hf_cache_volume = modal.Volume.from_name("my-hf-cache", create_if_missing=True) + +vllm_image = ( + modal.Image.debian_slim(python_version="3.12") + .pip_install("vllm==0.28.0", "huggingface_hub[hf_transfer]") + .env({ + "HF_HOME": "/cache/huggingface", + "HF_HUB_ENABLE_HF_TRANSFER": "1", + }) +) + +def _wait_for_health(port, timeout_s=1800): + """Block until vLLM's own /v1/models responds. Do NOT rely on @modal.web_server's + default startup_timeout for this -- a cold weight load (especially a big/quantized + checkpoint) can take several minutes and the default is too short.""" + deadline = time.time() + timeout_s + url = f"http://127.0.0.1:{port}/v1/models" + while time.time() < deadline: + try: + with urllib.request.urlopen(url, timeout=5) as resp: + if resp.status == 200: + return + except (urllib.error.URLError, ConnectionError, TimeoutError): + pass + time.sleep(5) + raise RuntimeError(f"vLLM on port {port} never became healthy") + +@app.cls( + gpu="H100", + image=vllm_image, + volumes={"/cache/huggingface": hf_cache_volume}, + scaledown_window=600, # stay warm 10 min between requests + min_containers=0, # no idle GPU spend + max_containers=2, # SEE "GPU quota" GOTCHA BELOW -- do not skip this + timeout=3600, +) +class MyModelServer: + @modal.enter() + def start_vllm(self): + cmd = ["vllm", "serve", "org/my-model-id", + "--served-model-name", "org/my-model-id", + "--port", str(VLLM_PORT), "--dtype", "auto", + "--max-model-len", "8192", "--gpu-memory-utilization", "0.90", + "--tensor-parallel-size", "1"] + self._proc = subprocess.Popen(cmd) + _wait_for_health(VLLM_PORT) + + @modal.exit() + def stop_vllm(self): + if getattr(self, "_proc", None): + self._proc.terminate() + + @modal.web_server(VLLM_PORT, startup_timeout=1800, label="my-model") + def serve(self): + pass +``` + +Deploy it: + +```bash +modal deploy path/to/vllm_server_app.py +``` + +This prints a stable HTTPS URL per class, e.g. `https://--my-model.modal.run`. Smoke +test before trusting it with real work: + +```bash +curl https://--my-model.modal.run/v1/models +``` + +## The client side + +Point your existing OpenAI-compatible client at the Modal URL: + +```bash +export OPENAI_COMPATIBLE_BASE_URL="https://--my-model.modal.run/v1" +python your_existing_eval_script.py --provider openai_compatible --model org/my-model-id ... +``` + +If the client runs on SLURM: **don't request a GPU** for the client job (`--gres` etc.) — the GPU +work happens on Modal, the client is CPU/API-bound only. + +## Gotchas (the part worth actually reading) + +### 1. GPU quota is silent and per-workspace — always set `max_containers` + +If you deploy 3-4 model classes without an explicit `max_containers` cap, whichever ones get +traffic first will auto-scale and eat the whole workspace's GPU quota. A later class trying to +start its first container will just... never come up, with no obvious error — it looks like a +hang, not a quota error. We lost real time misdiagnosing this as a container crash loop before +realizing it was quota exhaustion. + +**Fix:** always set `max_containers=N` explicitly on every class, sized so the sum across all +your classes fits your workspace's actual GPU quota. Cap first, ask questions later. + +To check what's actually running and eating quota: + +```bash +modal container list --json +``` + +Don't guess which containers belong to which model from timing alone — verify: + +```bash +modal container exec -- python3 -c \ + "import urllib.request; print(urllib.request.urlopen('http://localhost:8000/v1/models').read())" +``` + +### 2. Tensor-parallel + quantized weights can silently never come up + +We tried `gpu="H100:2"` + `--tensor-parallel-size 2` for an AWQ-quantized 32B model to speed up +generation. The endpoint never became healthy — no crash, no error, just an eternal 000 from +curl. We didn't chase the root cause under time pressure; reverted to `gpu="H100"` + +`tensor-parallel-size 1`, which worked immediately. If you hit this, don't assume it's your code — +try TP=1 first as a sanity check before debugging further. + +### 3. Client venv must have every package the eval script imports — including transitive ones + +We had several venvs lying around from different experiments; more than one was missing a package +the harness needs (`sympy` for one math-tool eval, `openai`/`pandas`/`huggingface_hub` for others). +A relaunch using the wrong venv fails fast with `ModuleNotFoundError` — annoying but at least +loud. Keep one blessed venv per project with every dependency installed and reuse it everywhere, +rather than grabbing whichever `.venv312` happens to be nearby. + +### 4. If you use content-addressed checkpoints, changing any CLI flag on relaunch silently starts over + +If your harness resumes from a checkpoint keyed by a hash of its config (CLI args + script +source), relaunching with even one different flag (e.g. a different token budget passed via an +env var you forgot to re-set) produces a **different hash**, which the harness treats as a brand +new run — it won't error, it'll just quietly start from zero and abandon your real progress. This +bit us once: relaunching several near-finished tasks without re-exporting a token-budget env var +threw away 90%+-complete runs before we caught it (checkpoint file was created fresh instead of +being resumed). + +**Fix:** when relaunching/resuming anything, diff the full set of CLI args/env vars against the +original launch before hitting go, not after. + +### 5. Match your client-side wall-clock limit (SLURM `--time`, etc.) to the actual model, not a generic default + +Per-request latency varies a lot by model even at similar sizes — one model we ran took roughly +3-4x longer per request than another of similar parameter count, for reasons we didn't fully +pin down (not explained by output length or contention alone). A `--time` limit that was generous +for the fast models wasn't enough for the slow one, and several tasks hit SLURM's `TIMEOUT` state +89%+ through a run. Check actual observed per-request latency for each model early, and size the +time limit per model rather than copy-pasting one value everywhere. + +### 6. Stagger simultaneous dataset downloads + +Launching two or more client jobs back-to-back that all call `load_dataset(...)` on the same HF +dataset at nearly the same instant can trip Hugging Face Hub's rate limiter +(`HfHubHTTPError: 429`) on the revision-tree check. Launch such jobs one at a time with a real +delay between them, not in a tight loop. + +### 7. Isolate per-task caches on shared filesystems + +If many parallel array tasks call `load_dataset()` against the same on-disk HF cache directory +simultaneously, you can hit `OSError: [Errno 116] Stale file handle` from lock contention on NFS. +Give each task its own `HF_DATASETS_CACHE`/`HF_HOME` (e.g. under `/tmp`, keyed by job/task id) +instead of sharing one across a whole array. + +## Quick reference + +| Task | Command | +|---|---| +| Deploy/update servers | `modal deploy path/to/vllm_server_app.py` | +| Dev/test one class live | `modal serve path/to/vllm_server_app.py` | +| List running containers | `modal container list --json` | +| Tail a container's logs | `modal container logs ` | +| Run something inside a live container | `modal container exec -- ` | +| Stop a container | `modal container stop ` (Modal will restart it if there's still demand and you haven't lowered `max_containers`) | +| Stop the whole app | `modal app stop ` | + +## When to reach for this + +Good fit: you're blocked on shared cluster GPU quota, you already have an OpenAI-compatible +client, and you can tolerate a few minutes of cold-start latency the first time a model spins up. + +Not a fit: sub-second latency requirements, or workloads that need tight co-location with other +cluster resources (shared filesystem throughput, InfiniBand between GPUs, etc.) that Modal doesn't +give you control over.