Honour CPU offload on H3, fix the training status, unblock large HF downloads

CLAUDE.md

@@ -159,6 +159,11 @@ linking. Generation is Core nodes, installed-extension nodes, and fal nodes on t
   `VolumeIcon`); reuse or add to those rather than dropping in an emoji.
 - **Tests (Vitest).** Cover the logic that matters: fal node input/request resolution, frame-input and
   hero-take resolution, DB migrations. UI is verified by running the app - don't chase view coverage.
+- **Arithmetic mirrored from Core lives in `src/shared/` and is pinned by a test.** `clipGrid.ts`
+  restates H3's frame grid so the Trainer can show what a setting resolves to; if the two drift the
+  UI promises a number the run will not honour.
+- **Surface what a setting resolves to, not just what was typed.** A control that silently snaps
+  (H3 clip length rounds down onto its frame grid) reads as broken.
 - **Commits.** Conventional Commits (`feat:`, `fix:`, `chore:`), small and scoped. `lint` +
   `typecheck` run on pre-commit (husky + lint-staged).
 - **Never commit automatically.** Claude (or any AI agent) must **not** run `git commit`/`git push`

core/CLAUDE.md

@@ -126,6 +126,8 @@ between nodes and are never takes.
   every stop/start**, taking the models and leaving dangling symlinks in `models/`. This has cost two
   full re-downloads of MiniMax H3 at ~130 GB each. Weights go on the persistent root volume, or on an
   attached volume that survives a restart. Scratch is fine for logs and temporary output only.
+- **`huggingface_hub>=0.32`** - below it `hf_xet` is only an extra, and files over ~50GB (every
+  H3 transformer) refuse to download at all.
 - **Models root** - `INLINE_MODELS_DIR`, else `./models`. **Bring your own weights; nothing is
   downloaded.** ComfyUI-style category subfolders (`diffusion_models/`, `vae/`, `text_encoders/`,
   `loras/`, `controlnet/`, `checkpoints/`, `clip_vision/`, `upscale_models/`, `embeddings/`). The
@@ -233,6 +235,20 @@ real codec that moves tensors lives with the model runner.
   vision-language encoder loaded in place of a text one, an unnormalized latent, and a control context
   cast to a quantized weight's `uint8` storage dtype. A unit test cannot see a plausible-but-wrong
   image. Render something and look at it.
+- **Launchers are twins.** `webui.sh` and `webui.bat` change together; only the `launcher` CI job
+  can prove the `.bat`, since it cannot run on a dev box.
+- **Match CUDA arches by within-major compatibility, never exactly.** An `sm_8x` cubin runs on any
+  `sm_8y` where `y >= x`, so `sm_86` covers Ada's `sm_89`.
+- **Pass the widest `uv` flag that works.** `--no-sources-package` needs uv 0.10+; `--no-sources`
+  works back to 0.4 and means the same while torch is the only `[tool.uv.sources]` entry.
+- **An explicit user setting beats a heuristic.** Test `on`/`off` before any auto rule, or the
+  control is silently dead for whichever arch the rule excludes.
+- **Report a phase before the slow work, not after.** Progress emitted only on completion makes a
+  slow step look like a hung previous phase.
+- **In a `.bat`, `call` anything that might be a `.bat`.** `nvidia-smi` is sometimes a shim, and
+  without `call` it takes over the script and never returns.
+- **Log allocated AND reserved VRAM.** `nvidia-smi` shows only reserved, so allocator cache and a
+  leaked reference look identical from outside.
 - **Tests (pytest).** Cover the logic that matters: graph validate/topo/executor/cache, the catalog
   scan, the run store + server contract, the device/memory policy, the parallel group + xfuser seam,
   and each model runner (import-guarded, no GPU needed). See `tests/`.

core/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 # PyPI name; the import package is `inline_core` (src/inline_core).
 name = "inline-core"
-version = "1.2.65"
+version = "1.2.66"
 description = "The generation engine behind Inline Studio."
 readme = "README.md"
 license = "GPL-3.0-or-later"
@@ -29,7 +29,10 @@ runtime = [
   # Backs the Beta sigma schedule (diffusers gates use_beta_sigmas on scipy).
   "scipy>=1.11",
   # We call snapshot_download directly for the model popup, so pin it rather than rely on transit.
-  "huggingface_hub>=0.23",
+  # 0.32 is where hf_xet became a hard dependency rather than an extra. Below it, Hugging Face
+  # refuses the largest files outright ("too large to be downloaded using the regular download
+  # method"), which is every H3 transformer at 66GB.
+  "huggingface_hub>=0.32",
   # ControlNet preprocessors (the Apply ControlNet node): OpenPose/DWPose, MiDaS/Zoe depth, canny,
   # HED, lineart, MLSD, scribble, normal. DWPose runs its detector on ONNX Runtime.
   "controlnet-aux>=0.0.7",
@@ -85,7 +88,10 @@ all = [
   # Clip decode for MiniMax H3 LoRA training, and H3's reference node.
   "av>=12",
   "scipy>=1.11",
-  "huggingface_hub>=0.23",
+  # 0.32 is where hf_xet became a hard dependency rather than an extra. Below it, Hugging Face
+  # refuses the largest files outright ("too large to be downloaded using the regular download
+  # method"), which is every H3 transformer at 66GB.
+  "huggingface_hub>=0.32",
   "controlnet-aux>=0.0.7",
   "onnxruntime>=1.17",
   # server

core/src/inline_core/training/models.py

@@ -325,20 +325,23 @@ def resolve_offload(
 ) -> bool:
     """Whether to stream saved activations to host RAM this run.
 
-    Only meaningful for a full-precision (bf16) base: the point is to keep the 26GB Krea 2 base
-    resident and fit the ~21GB of 1024 activations elsewhere, rather than dropping the frozen base
-    to NF4. A quantized base already fits, so offload just adds PCIe traffic for nothing. ``auto``
-    turns it on exactly when the bf16 plan would otherwise not fit VRAM; ``on``/``off`` force it."""
+    ``auto`` was written for a full-precision base: keep the 26GB Krea 2 base resident and put the
+    ~21GB of 1024 activations elsewhere, rather than dropping the frozen base to NF4. Under a
+    quantized base ``auto`` stays off, because there the base is the whole story and offload would
+    buy PCIe traffic for nothing.
+
+    ``on``/``off`` are tested before the quant rule, or the control is dead for MiniMax H3 (always
+    4-bit), whose base is small and whose clip activations are what overflow the card."""
     from ..device.policy import Quantization
 
     if pref == "off":
         return False
-    if quant is not Quantization.NONE:
-        return False  # a quantized base already fits; offload would only slow it down
     if pref == "on":
         return True
     if pref not in ("auto", ""):
         raise RuntimeError(f"Unknown offload preference {pref!r}.")
+    if quant is not Quantization.NONE:
+        return False  # auto only: a quantized base already fits, so do not pay for offload
 
     import torch
 

core/src/inline_core/training/trainer.py

@@ -119,6 +119,20 @@ def _peak_vram_gb() -> float | None:
     return round(torch.cuda.max_memory_allocated() / 1e9, 2)
 
 
+def _vram_note(label: str) -> str:
+    """Both numbers: nvidia-smi shows only reserved, so allocator cache and a leaked reference look
+    identical from outside."""
+    import torch
+
+    if not torch.cuda.is_available():
+        return label
+    gb = 1e9
+    return (
+        f"{label}: allocated {torch.cuda.memory_allocated() / gb:.1f}GB, "
+        f"reserved {torch.cuda.memory_reserved() / gb:.1f}GB"
+    )
+
+
 def _activation_offload(enabled: bool) -> Any:
     """A context that streams saved activations to host RAM (pinned) for the forward, pulling them
     back on backward. Keeps a full-precision base resident on a card that could not otherwise hold
@@ -205,9 +219,11 @@ def train(manifest: dict[str, Any]) -> str | None:
     )
     plan = quant.value + (" + cpu offload" if offload else "")
     protocol.progress(0, steps, status=f"loading model ({plan})")
+    print(_vram_note("VRAM after caching, before the base loads"), flush=True)
     transformer = models.load_transformer(
         manifest["modelsDir"], arch.key, manifest["baseMode"], str(device), dtype, quant
     )
+    print(_vram_note("VRAM after the base loaded"), flush=True)
     transformer.requires_grad_(False)
     # PEFT picks its bitsandbytes-aware LoRA layer off this one attribute. Without it, and because
     # bnb's Linear4bit subclasses nn.Linear, the generic dispatcher matches instead: grads still
@@ -237,6 +253,10 @@ def train(manifest: dict[str, Any]) -> str | None:
     signal.signal(signal.SIGTERM, stop)
 
     transformer.train()
+    # Before the first step, not after: emitting only on completion makes a slow step one look like
+    # the loader is still running.
+    print(_vram_note("VRAM entering the training loop"), flush=True)
+    protocol.progress(start, steps, status="training")
     for step in range(start, steps):
         if stop.flagged:
             break

core/tests/test_minimaxh3_training.py

@@ -475,3 +475,15 @@ def test_clip_window_defaults_to_the_start(tmp_path) -> None:
     assert h3._clip_frames(clip, clip_frames=24)[0].getpixel((0, 0)) == (
         h3._clip_frames(clip, clip_frames=24, window="start")[0].getpixel((0, 0))
     )
+
+
+def test_the_loop_reports_training_before_the_first_step(monkeypatch) -> None:
+    """Emitted only on completion, a slow step one reads as a hung loader."""
+    import inspect
+
+    from inline_core.training import trainer
+
+    source = inspect.getsource(trainer.train)
+    enters_loop = source.index("for step in range(start, steps):")
+    announces = source.index('status="training"')
+    assert announces < enters_loop, "the training status must be sent before the loop, not after"

core/tests/test_training_models.py

@@ -97,8 +97,8 @@ def test_auto_quantization_accounts_for_resolution(monkeypatch, tmp_path) -> Non
 
 def test_offload_fits_a_bf16_base_that_would_not_otherwise(monkeypatch, tmp_path) -> None:
     """bf16 1024 on a 45GB card: base (26GB) + activations (~21GB) overflow, so auto-offload turns
-    on to keep the base full precision rather than dropping it to NF4. A quantized base already
-    fits, so offload stays off there no matter the preference."""
+    on to keep the base full precision rather than dropping it to NF4. Under a quantized base AUTO
+    stays off, but an explicit on/off is the user's answer and wins."""
     root = tmp_path / "models"
     (root / "diffusion_models").mkdir(parents=True)
     (root / "diffusion_models" / "krea2_raw_bf16.safetensors").write_bytes(b"")
@@ -120,8 +120,10 @@ def test_offload_fits_a_bf16_base_that_would_not_otherwise(monkeypatch, tmp_path
     assert off("auto", Quantization.NONE, str(root), archs.KREA2, "raw", 512) is False
     assert off("on", Quantization.NONE, str(root), archs.KREA2, "raw", 512) is True
     assert off("off", Quantization.NONE, str(root), archs.KREA2, "raw", 1024) is False
-    # A quantized base already fits, so offload would only add PCIe traffic - never on.
-    assert off("on", Quantization.NF4, str(root), archs.KREA2, "raw", 1024) is False
+    # Auto stays off under a quantized base; an explicit choice wins.
+    assert off("auto", Quantization.NF4, str(root), archs.KREA2, "raw", 1024) is False
+    assert off("on", Quantization.NF4, str(root), archs.KREA2, "raw", 1024) is True
+    assert off("off", Quantization.NF4, str(root), archs.KREA2, "raw", 1024) is False
 
 
 def test_zimage_has_no_four_bit_path_and_says_so(tmp_path) -> None:
@@ -198,3 +200,26 @@ def test_fails_open_when_the_machine_cannot_be_read(monkeypatch) -> None:
     _fake_env(monkeypatch, mode=0, ram_gib=30, size_gib=62)
     monkeypatch.setattr(models, "_base_size", lambda *a: 0)
     models.check_base_mappable("/m", "minimax-h3", "raw")
+
+
+def test_an_explicit_offload_choice_beats_the_quant_heuristic(monkeypatch, tmp_path) -> None:
+    """Dead for H3, always 4-bit, whose clip activations are what overflow the card."""
+    from inline_core.device.policy import Quantization
+
+    root = tmp_path / "models"
+    (root / "diffusion_models").mkdir(parents=True)
+    monkeypatch.setattr(models, "_base_size", lambda *a: 12 * 1024**3)
+    off = models.resolve_offload
+
+    assert off("on", Quantization.NF4, str(root), archs.MINIMAX_H3, "raw", 512) is True
+    assert off("off", Quantization.NF4, str(root), archs.MINIMAX_H3, "raw", 512) is False
+    assert off("auto", Quantization.NF4, str(root), archs.MINIMAX_H3, "raw", 512) is False
+
+
+def test_an_unknown_offload_preference_still_raises(monkeypatch, tmp_path) -> None:
+    """The reorder must not let a typo fall through to the auto path and silently mean 'off'."""
+    from inline_core.device.policy import Quantization
+
+    monkeypatch.setattr(models, "_base_size", lambda *a: 12 * 1024**3)
+    with pytest.raises(RuntimeError):
+        models.resolve_offload("yes", Quantization.NF4, str(tmp_path), archs.MINIMAX_H3, "raw", 512)

core/webui.bat

@@ -162,14 +162,10 @@ if !CAP_MAJOR! GEQ 10 if !DRIVER_MAJOR! GTR 0 if !DRIVER_MAJOR! LSS 580 (
 if /i "!TORCH_CHOICE!"=="cpu" goto install_cpu_forced
 set "TORCH_URL=https://download.pytorch.org/whl/!TORCH_CHOICE!"
 if /i "!TORCH_CHOICE:~0,4!"=="http" set "TORCH_URL=!TORCH_CHOICE!"
-rem unsafe-best-match: torchao is on the CUDA index too, older there than our torchao>=0.14 pin on
-rem some indexes; without this uv's first-index rule stops at that older copy instead of finding a
-rem new enough one on PyPI. It also makes the +cuXXX local version outrank PyPI's plain one, which
-rem is what pulls the CUDA build in rather than the CPU-only wheel PyPI serves on Windows.
-rem no-sources: the pyproject [tool.uv.sources] pin names one fixed index, and the card decides
-rem here. Deliberately the broad flag, not --no-sources-package torch: that one is too new for the
-rem uv versions people actually have, and it hard-errored their install. torch is the only entry in
-rem that table, so the two mean the same thing today. Adding another entry would change that.
+rem unsafe-best-match: without it uv stops at the older torchao on the CUDA index, and PyPI's plain
+rem torch outranks the +cuXXX build on Windows.
+rem no-sources: the pyproject pin names one index, and the card decides here. The broad flag, not
+rem --no-sources-package, which needs uv 0.10+; torch is the only entry so they are equivalent.
 set "TORCH_ARGS=--extra-index-url !TORCH_URL! --index-strategy unsafe-best-match --no-sources"
 echo NVIDIA GPU detected - installing the CUDA build of PyTorch (!TORCH_CHOICE!).
 goto install_pkgs
@@ -216,10 +212,8 @@ if /i "!PROBE_STATUS!"=="uncovered" (
 :install_run
 echo + uv pip install --python "!TARGET_PY!" !TORCH_ARGS! -e ".[!EXTRAS!]"
 uv pip install --python "!TARGET_PY!" !TORCH_ARGS! -e ".[!EXTRAS!]" || goto fail
-rem Torch LAST, and through --index-url (exclusive), when the index was named or the installed wheel
-rem is wrong. It cannot ride on the project install: [tool.uv.sources] pins torch to the cu126 index
-rem on win32, and --extra-index-url with unsafe-best-match picks the highest version ACROSS indexes,
-rem which lands back on PyPI's CPU wheel whenever PyPI leads.
+rem Torch LAST and through --index-url (exclusive): [tool.uv.sources] pins it to cu126 on win32,
+rem and --extra-index-url picks the highest version ACROSS indexes, so PyPI's CPU wheel can win.
 if "!TORCH_FORCE!"=="1" if defined TORCH_CHOICE (
   set "TORCH_PINS=torch torchvision"
   rem cu128 is frozen, so the current pair does not exist there. Pin the last one it has.
@@ -243,18 +237,14 @@ echo          .\webui.bat --install --torch-index !TORCH_CHOICE! --recreate
 echo Installed extras: !EXTRAS!. Start with: .\webui.bat
 exit /b 0
 
-rem Reads compute_cap and driver_version in one query, keeping the highest capability seen.
-rem set /a rather than a findstr guard: an old driver answers an unknown query with an error string,
-rem and `if LSS` would STRING-compare it, so "Unknown" would rank above 10 and win. set /a reads a
-rem bare word as an undefined variable and yields 0, which is exactly what we want here, so do not
-rem "fix" it back into a guard later.
+rem One query for both, keeping the highest capability. set /a not a findstr guard: `if LSS` would
+rem string-compare an error string like "Unknown" above 10; set /a yields 0 for it, as intended.
 :read_gpu_probe
 set "CAP_MAJOR=0"
 set "DRIVER_MAJOR=0"
 set "GPU_PROBE_RAW="
-rem Split on the comma ONLY. Including "." here would cut "12.0, 610.88" into 12 / 0 / 610 / 88, so
-rem token 2 would be the capability's minor rather than the driver, and the R580 floor could never
-rem fire. The majors are taken off each field by the inner loops.
+rem Comma ONLY: adding "." would cut "12.0, 610.88" into 12/0/610/88, making token 2 the minor
+rem rather than the driver, so the R580 floor could never fire.
 for /f "usebackq tokens=1,2 delims=," %%c in (`nvidia-smi --query-gpu^=compute_cap^,driver_version --format^=csv^,noheader 2^>nul`) do (
   rem Reset every iteration: set /a errors on garbage and would otherwise leave the previous line's
   rem value in place, double-counting a good line followed by a bad one.