Inline raw image refs for multimodal rendering

[eligotts]

Summary

Intermediate multimodal storage mode — the inline bundle. Keeps this repo's raw multimodal contract (renderer computes token layout via pure geometry math, no image processor at render time; refs + descriptors instead of processed tensors) but images travel inline as data:image/...;base64 URLs instead of offloaded file:// run assets.

• mm_store: the mmraw ref envelope carries raw_image_data (the inline data URL) instead of raw_image_uri. offload_image_to_run_assets, run_image_dir, and VF_RENDERER_IMAGE_OFFLOAD_DIR are gone. New decode_data_image_url() is the single supported source decode.
• Qwen3-VL / Qwen3.5 / Kimi-K2.5 raw layout describes decode the inline source once for dimensions + sha256 hash; descriptors embed the source.
• client.generate() serializes the inline source into every image slot's ref — every request to /inference/v1/generate carries the full inline payload for every image in the prompt (current and prior turns), and the prime-rl serving layer dedups processor work by content hash.

Relationship to the offload bundle

This PR is cut from the offload branch (#89) and is the storage-layer intermediate: same processor-ownership architecture, no shared image directory. The offload bundle (renderers #89 / verifiers #1746 / prime-rl #2836) layers content-addressed file:// offload on top of this contract.

Companion inline PRs: verifiers feat/v1-inline-raw-images, prime-rl feat/v1-inline-mm.

Validation

• Full suite: 2153 passed, 121 skipped, 1 xfailed (processor-parity tests included, run against the real Qwen3-VL-4B and pinned Kimi-K2.5 processors).

Note

Inline raw image refs for multimodal rendering instead of processed tensors

• Adds a new default multimodal output mode ("raw") across Qwen3-VL, Qwen3.5, and Kimi K2.5 renderers where images are emitted as raw run-image reference envelopes rather than processor-computed pixel tensors; processed tensors remain available via multimodal_output="processed".
• Introduces renderers/mm_store.py with a stable encoding/decoding scheme (raw_mm_item, raw_mm_ref, split_raw_mm_ref) for raw multimodal items shared by renderers and client feature serialization.
• Adds describe_qwen_image_layout and describe_kimi_image_layout to compute image token counts and grid metadata deterministically without running an HF processor.
• Replaces renderer-specific _build_mm_features / _build_qwen_vl_features in renderers/client.py with a renderer-agnostic _build_vllm_mm_features that serializes raw reference strings per item across modalities.
• Risk: non-raw mm_items now raise NotImplementedError at feature-build time; image_cache_max is removed from all renderer configs; renderer constructors no longer accept an injected processor argument.

^{Macroscope summarized 50ce3c3.}

Update: unwrapped the ref payload (603118b)

raw_mm_ref serializes its payload as compact JSON instead of base64-wrapping it. The ref travels as a string inside a JSON request body, so the only encoding cost is escaping the payload's own quotes (~200 bytes) — while the base64 wrapper inflated the entire payload, image included, by a third.

Measured on a 75 KiB JPEG: 130.3 KiB → 97.9 KiB per image slot (the 97.7 KiB data URL plus ~200 bytes), a 25% cut on every image slot of every request. Refs parse with a single partition(":") now that the payload contains colons; the base64-alphabet guard retired with the wrapper.

---

Note

High Risk
Changes the default MultiModalData.mm_items shape and the vLLM generate request payload, affecting inference and any caller that assumed processed tensors; multimodal bridge and multi-turn image carry-forward behavior also shift.

Overview
Multimodal rendering now defaults to lightweight raw image descriptors instead of running HuggingFace image processors at render time. A new multimodal_output setting on renderer configs chooses raw (JSON-safe refs for inference) or processed (pixel_values for SFT/training). Auto/default renderer resolution forwards multimodal_output like thinking_retention.

New renderers/mm_store.py defines the wire contract: decode inline data:image/...;base64 URLs, build prime_raw_mm_item envelopes, and emit compact mmraw: refs (payload is JSON, not base64-wrapped). merge_multi_modal_data centralizes bridge turn sidecar merging.

Qwen3-VL, Qwen3.5, and Kimi K2.5 compute pad counts and grid metadata via ported layout math (describe_*_image_layout) without materializing pixels in raw mode; processors load lazily only when multimodal_output='processed'. Constructor processor= injection and per-renderer image_cache_max config are removed.

client.generate() no longer dispatches on renderer class or imports torch/vLLM to base64-encode HF batches. _build_vllm_mm_features maps raw envelopes to vLLM features (hashes, placeholders, one ref per slot with inline image data). Prebuilt prompt_ids can pull multi_modal_data from prompt_attribution when omitted.

Packaging/tests: optional vision extra (pillow/torch/torchvision); multimodal parity tests use inline data URLs; new tests cover raw ref serialization, processed-mode payloads, and raw-vs-processor layout math.

^{Reviewed by Cursor Bugbot for commit 50ce3c3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 50ce3c3. Configure here.}

[cursor]

Vision extra pulls unused torch

Medium Severity

The new vision optional extra installs torch and torchvision, but this package no longer imports either after moving client serialization to raw refs and using return_tensors="np" in processed mode. Default raw multimodal rendering only needs Pillow for dimension reads, so renderers[vision] currently forces a large unused dependency set.

 

^{Reviewed by Cursor Bugbot for commit 50ce3c3. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new multimodal output mode system and changes how images are serialized for inference, switching from processed tensors to raw image refs. These are substantial architectural changes affecting core multimodal rendering behavior across multiple files.

^{You can customize Macroscope's approvability policy. Learn more.}