feat(volumecache): adaptive size-bucketed transport

README.md

@@ -149,6 +149,19 @@ runpod.serverless.start({"handler": handler})
 
 See [Worker Fitness Checks](https://github.com/runpod/runpod-python/blob/main/docs/serverless/worker_fitness_checks.md) documentation for more examples and best practices.
 
+### Network-Volume Warm Cache
+
+When a network volume is attached, `VolumeCache` warms local directories (such as a model cache) across cold starts — hydrating them on startup and syncing new files back on exit — so a repeated multi-GB model download becomes a one-time cost per endpoint. It is stdlib-only and best-effort.
+
+```python
+from runpod.serverless import VolumeCache
+
+with VolumeCache(dirs=["/root/.cache/huggingface"]):
+    model = load_model()
+```
+
+See [Network-Volume Warm Cache](https://github.com/runpod/runpod-python/blob/main/docs/serverless/volume_cache.md) documentation for configuration and details.
+
 ## 📚 | API Language Library (GraphQL Wrapper)
 
 When interacting with the Runpod API you can use this library to make requests to the API.

docs/serverless/volume_cache.md

@@ -0,0 +1,110 @@
+# Network-Volume Warm Cache (VolumeCache)
+
+`VolumeCache` warms local directories across serverless workers using a mounted
+network volume. It keeps a browsable mirror of your cache directories on the
+volume and reconciles it against the container on each use: on cold start it
+restores previously-cached files into place, and after use it copies newly
+written files back to the volume. This turns a repeated multi-GB model
+download on every cold start into a one-time cost per endpoint.
+
+It is stdlib-only and best-effort: any failure degrades to a cold worker and
+never raises into your handler or worker loop.
+
+## Requirements
+
+- A **network volume** attached to the endpoint (mounted at `/runpod-volume`).
+- Set on serverless automatically: `RUNPOD_ENDPOINT_ID` (used to scope the
+  mirror per endpoint/namespace).
+
+If no volume is mounted, or the namespace is empty, every operation is a safe
+no-op.
+
+## Usage
+
+`VolumeCache` is a context-manager closure — using it around a model load
+hydrates the cache before the block runs and syncs any changes back after:
+
+```python
+from runpod.serverless import VolumeCache
+
+with VolumeCache(dirs=["/root/.cache/huggingface"]):
+    model = load_model()   # downloads land in the cached directory
+```
+
+- **On enter**, `hydrate()` copies files that are missing or newer on the
+  volume mirror into the container.
+- **On exit**, `sync()` copies files that are missing or newer in the
+  container onto the volume mirror. By default this runs on a background
+  daemon thread and returns immediately, so the `with` block doesn't block on
+  the sync; a process-exit hook joins any outstanding syncs so short-lived
+  processes (including local test runs) still complete the sync before
+  exiting.
+
+You can also call the phases directly when they happen at different points in
+your worker's lifecycle:
+
+```python
+vc = VolumeCache(dirs=["/data/models"], namespace="my-model-cache")
+
+vc.hydrate()                        # restore cached files (e.g. at startup)
+model = load_model()                # populate the cache
+vc.sync(background=False)           # persist new files back to the volume, inline
+```
+
+## Constructor
+
+| Argument | Default | Purpose |
+| --- | --- | --- |
+| `dirs` | required | Local directories to cache. |
+| `namespace` | `RUNPOD_ENDPOINT_ID` | Isolation key for the on-volume mirror. Must be a single safe path component. |
+| `volume_path` | `/runpod-volume` | Network-volume mount point. |
+| `best_effort` | `True` | Swallow and log errors instead of raising. Set `False` while debugging. |
+| `max_workers` | `min(32, (os.cpu_count() or 4) * 4)` | Thread count for parallel copy of large files (I/O-bound). |
+
+## How it works
+
+- **Size-bucketed mirror.** Cached files live at
+  `{volume_path}/.cache/{namespace}`, split by size: files below 256 KiB are
+  packed into a single `small.tar` archive (collapsing per-file metadata
+  round-trips on the network volume), and larger files are copied unpacked
+  into a `big/` subdirectory, preserving their original relative path. A
+  versioned `manifest.json` — written last — records file metadata (size,
+  mtime) for every cached file and is the commit marker for a complete
+  mirror; a mirror without a valid, current-version manifest is treated as
+  absent.
+- **Incremental large files, whole-archive small files.** `big/` transfers
+  are diffed per file by size/mtime against the manifest, so unchanged large
+  files are skipped. The `small.tar` archive is re-packed as a whole whenever
+  any small file has changed, since unpacking and re-diffing many tiny files
+  individually is slower than the network volume's per-file overhead.
+- **Parallel copy.** Large-file transfers run across a thread pool sized by
+  `max_workers` (default `min(32, (os.cpu_count() or 4) * 4)`), since the
+  work is I/O-bound.
+- **Packing/extraction.** Packing the small-file archive uses the `tar`
+  binary when available (falling back to the stdlib `tarfile` module
+  otherwise). Extraction always goes through `tarfile`, validating every
+  member's resolved path against the configured `dirs` before writing.
+- **Idempotent.** Re-running `hydrate`/`sync` after nothing has changed
+  copies zero files and does not repack `small.tar`. `manifest.json` is still
+  refreshed on every `sync` call — a small atomic write that also drops
+  entries for files deleted locally since the last sync.
+- **Last-writer-wins.** Under concurrent workers, the mirror simply reflects
+  whichever worker synced most recently — there's no locking or merge.
+- **Safety.** Symlinked sources are never followed/copied. Every archive
+  member and every big-file destination is checked to resolve inside one of
+  the configured `dirs` before any write, so a mirror entry can't be used to
+  write outside the cached directories.
+
+## Limitations
+
+- **Cold-scale write amplification.** If N workers cold-start at the same time,
+  each may miss the still-empty mirror, download the model, and sync a full
+  copy back. There's no coordination between concurrent syncs.
+- **Background sync on short-lived processes.** `sync()` schedules the copy on
+  a daemon thread; if the process exits without going through normal
+  interpreter shutdown (e.g. `os._exit`, `SIGKILL`), the atexit hook never
+  runs and the sync may not complete.
+- **Orphaned big files are never pruned.** If a large file is deleted or
+  renamed locally, its `big/<relpath>` copy stays on the volume — hydrate
+  is manifest-driven and simply ignores it, but volume space grows across
+  model-version swaps. Same behavior as the prior flat-mirror design.

examples/serverless/volume_cache_handler.py

@@ -0,0 +1,48 @@
+"""Warm-cache model weights across cold starts with VolumeCache.
+
+`VolumeCache` keeps a browsable mirror of local cache directories on a mounted
+network volume and reconciles the two on each use: it restores cached files on
+cold start (`hydrate`) and syncs new downloads back afterward (`sync`). Wrapping
+a model load in `with VolumeCache(...)` turns a repeated multi-GB download into a
+one-time cost per endpoint.
+
+Why not just point HF_HOME at the volume? Because then every read hits the
+network mount. Here the Hugging Face cache stays on fast local disk and
+VolumeCache mirrors it to the volume, so inference reads are local while cold
+starts stay warm.
+
+Requirements to run:
+- Attach a network volume to the endpoint (mounted at /runpod-volume).
+- RUNPOD_ENDPOINT_ID is set automatically on Runpod serverless (it scopes the
+  cache per endpoint). Without a mounted volume or endpoint id, VolumeCache is a
+  safe no-op and the handler still works.
+- pip install "transformers" "torch" (the model library used below).
+
+Local test:
+    python volume_cache_handler.py --rp_serve_api
+"""
+
+import os
+
+import runpod
+from runpod.serverless import VolumeCache
+
+# Hugging Face caches models here by default. Keep it on local disk and let
+# VolumeCache mirror it to the network volume.
+HF_CACHE = os.environ.get("HF_HOME", os.path.expanduser("~/.cache/huggingface"))
+
+# Hydrate the cache from the volume before loading, then sync new files back on
+# exit. On a warm endpoint the model is already local and nothing is downloaded.
+with VolumeCache(dirs=[HF_CACHE]):
+    from transformers import pipeline
+
+    classifier = pipeline("sentiment-analysis")
+
+
+def handler(job):
+    """Classify the sentiment of an input string."""
+    text = job["input"].get("text", "")
+    return {"input": text, "predictions": classifier(text)}
+
+
+runpod.serverless.start({"handler": handler})

runpod/serverless/__init__.py

@@ -17,12 +17,14 @@
 from .modules.rp_logger import RunPodLogger
 from .modules.rp_progress import progress_update
 from .modules.rp_fitness import register_fitness_check
+from .utils.rp_volume_cache import VolumeCache
 
 __all__ = [
     "start",
     "progress_update",
     "register_fitness_check",
-    "runpod_version"
+    "runpod_version",
+    "VolumeCache",
 ]
 
 log = RunPodLogger()

runpod/serverless/utils/__init__.py

@@ -2,9 +2,11 @@
 
 from .rp_download import download_files_from_urls
 from .rp_upload import upload_file_to_bucket, upload_in_memory_object
+from .rp_volume_cache import VolumeCache
 
 __all__ = [
     "download_files_from_urls",
     "upload_file_to_bucket",
-    "upload_in_memory_object"
+    "upload_in_memory_object",
+    "VolumeCache",
 ]