[codex] Support raw image refs for multimodal rendering

[eligotts]

Design update — inline/offload image storage

This PR now supports both raw image transport modes used by prime-rl:

• offload: existing behavior, raw image bytes are written to run-scoped image assets and refs carry a file-backed image id.
• inline: data-image URIs remain inline and raw refs carry the inline source instead of requiring raw_image_id.

This repo adds inline-capable mmraw:v3 refs while preserving mmraw:v2 parsing, keeps Qwen image hashes aligned to the raw decoded bytes, and emits raw descriptor items with either raw_image_id or raw_uri.

Validation after latest push: uv run pytest tests/test_client.py -q passed (14 passed).

Design update — dropped the None/cache-only image path

This PR and its companions (prime-rl #2836 / verifiers #1746 / renderers #89) no longer use the "send None for already-cached images" mechanism. Every image carries its raw descriptor ref at every slot (current and prior turns); /inference/v1/generate rematerializes each ref from disk every request.

Why: the None path coupled correctness to deployment (LRU cache present, single replica / DP-affinity, no eviction) and surfaced a miss as a hard vLLM EngineDeadError (qwen3-vl mrope dereferences a None image_grid_thw) that the retry net couldn't catch across the engine→API IPC. Dropping it is deployment-agnostic (a miss is impossible) and non-hacky. vLLM's mm_hash encoder cache still skips the expensive GPU re-encode for free — we only forgo the cheap IPC/CPU-reprocess dedup.

Validated: color-codeword (Qwen3-VL-4B) under DP=2, no affinity / no cache reliance: 0 crashes, 0 data=None, multi-turn accumulation correct, reward ~0.84. Also confirmed under TP.

This repo: every image emits a raw descriptor ref at every slot. _descriptor_only_mm_data no longer strips the pointer (pixel_values were never present in v1, so the strip was both stale and the root cause of the descriptor-only/rebuild churn). Removed the materialize_all_image_refs flag and the now-orphaned materialize_image_refs / materialize_kimi_image_refs.

---

Original description

Summary

• adds generic mmraw:v2 raw multimodal refs in renderers.mm_store, parsed as RawMMRef objects with family, fingerprint, modality, hash, asset id, and adapter-owned payload
• emits strict prime_raw_mm_item envelopes instead of processed image payloads for Qwen-VL and Kimi K2.5 image rendering
• keeps adapter-specific layout details in renderer-owned payloads (image_grid_thw for Qwen, grid_thws/media token metadata for Kimi)
• supports materializing all raw image refs for retry paths after vLLM multimodal cache misses
• keeps run-scoped image asset refs file-backed so downstream Prime-RL trainer materializes images with its own processor

Companion PRs

• Prime-RL: Support v1 raw multimodal image offload prime-rl#2836
• Verifiers: [codex] Support raw image offload in v1 train client verifiers#1746

Notes

• Draft/WIP: stacked with the Verifiers and Prime-RL raw image offload PRs.
• Verifiers is expected to offload image content to file://.../assets/images/... refs before rendering.
• This intentionally treats raw image refs as the supported path, not processed multimodal feature sidecars.

Validation

• uvx ruff@0.15.18 check . passed.
• uvx ruff@0.15.18 format --check . passed.
• uvx 'ty<0.0.22' check . exited 0; remaining diagnostics are warning-level advisories under the repo config.
• PYTHONPATH=/home/ubuntu/renderers uv run --no-project --active pytest -q tests/test_client.py passed: 14 passed.
• End-to-end hosted-style smoke through Prime-RL with /home/ubuntu/renderers, /home/ubuntu/verifiers, and /home/ubuntu/prime-rl-v1-raw-mm-offload completed inference, env rollouts, train batch creation, trainer step 0, and decoded strict trainer-bound raw image refs.

[!NOTE]

Add raw image ref support for multimodal rendering across Qwen and Kimi renderers

• Renderers now emit raw image references (https://github.com/PrimeIntellect-ai/renderers/pull/89/files#diff-f32a3fee405568ec4d5d9d9089bf70bf2a9ff402b421111d9320371190d9f453) by default instead of processed pixel tensors; processed payloads are only produced when config.multimodal_output == 'processed'.
• Adds processor-free layout computation (describe_qwen_image_layout, describe_kimi_image_layout) that calculates token counts and grid shapes using smart_resize without loading the HF image processor.
• Introduces mm_store.py with utilities for offloading inline images to content-addressed disk files and constructing/parsing stable raw multimodal refs.
• Adds a multimodal_output: 'raw' | 'processed' field to BaseRendererConfig (default 'raw'), propagated through auto-resolution; removes image_cache_max from all renderer configs.
• Exposes a vision install extra in pyproject.toml pulling in Pillow, Torch, and Torchvision.
• Risk: raw mode requires images to be file-backed (file:// URIs); inline data URIs and HTTP sources are rejected at render time unless offloaded first.

^{Macroscope summarized 66505d2.}

Update: review hardening (e3c12e9)

• Fixed render_completion_update mutating the caller's previous_multi_modal_data in place (shallow dict copy + setdefault().extend()); the merge now lives in a shared merge_multi_modal_data helper in base.py that copies inner lists, and the bridge test asserts the previous sidecar is unmutated.
• Qwen resize math is imported from transformers (torch-free PIL-backend module, with a fallback for older layouts) instead of maintaining a port.
• Raw layout describes resolve and read each image asset once; mm_store uses full sha256 content-addressed filenames and raises on undecodable base64.
• Added test_raw_layout_math_matches_image_processor: parity against the real Qwen3-VL-4B and Kimi-K2.5 (pinned revision) processors at rounding-boundary dimensions. Full suite: 2182 passed.

Update: merged main (e64cc58)

Reconciled with renderers main (12 commits: Hy3/PrimeQwen3/LagunaXS21 configs, kept-tokens parsing, SFT stop-token opt-in, a qwen thinking-wrapper fix, and bdb96b0 — main's own independent fix for the same bridge-mutation bug this PR's merge_multi_modal_data() already covers).

• kimi_k25.py/qwen35.py/qwen3_vl.py: kept this branch's shared merge_multi_modal_data() helper over main's three duplicated inline fixes for the identical bug — same behavior, one implementation instead of three.
• test_multimodal.py: re-injected main's _skip_for_disabled_thinking_deviation skip guard into this branch's parallel renderer_cases/processor_cases structure at all 4 call sites; took main's tuple-based prior_mm/prior_counts assertion over this branch's redundant duplicate check.
• test_client.py, __init__.py, configs.py: non-overlapping additions on both sides, no semantic conflict.
• client.py: no textual conflict (main's fix landed near, not on, this branch's rewritten _build_vllm_mm_features) — manually verified the merged generate() is consistent (prompt_attr/mm_data naming intact).

Full suite: 2578 passed, 153 skipped, 1 xfailed.

Update: unwrapped the ref payload (7c58fd6)

raw_mm_ref serializes its payload as compact JSON instead of base64-wrapping it. The ref travels as a string inside a JSON request body, so the wrapper's 33% inflation bought nothing. The saving is small in offload mode (the payload is a short file:// URI), but it keeps the ref format identical to the inline bundle, where the payload carries the image source and the wrapper cost ~32 KiB per image slot. Refs parse with a single partition(":") now that the payload contains colons.

Update: synced with main (1af6589)

Merged renderers main (5 commits: vLLM token-serving import fix #115, GLM tool-name validation #114, setuptools bump, ruff-action pin).

One conflict, in client.py: main's #115 updated the vLLM mm_serde import path inside the processed-tensor encoder that this PR deletes by design. Kept this branch's raw-ref _build_vllm_mm_features; main's fix applies to code that no longer exists here.

Full suite: 2622 passed, 125 skipped, 1 xfailed.

Update: single raw-mode image route (66505d2)

Raw mode now accepts exactly one image source: an offloaded file:// URL. The bare-filesystem-path fallback was a second route the rest of the stack couldn't honor anyway — prime-rl's trainer-side file_uri_to_path already rejects non-file:// sources and verifiers ingress only ever emits as_uri(). Anything else raises the existing "requires offloaded file:// image assets" error.

---

Note

High Risk
Large multimodal contract change across inference client, bridge merging, and three VL renderers; constructor API removal and mandatory offloaded image assets can break downstream integrations.

Overview
Multimodal inference path switches from embedding processed pixel_values in the sidecar to JSON-safe raw image descriptors (prime_raw_mm_item) with layout metadata and file:// URIs. Qwen-VL, Qwen3.5, and Kimi K2.5 compute placeholder counts via baked layout math (or transformers smart_resize for Qwen) without running the image processor in the default mode.

Adds multimodal_output on renderer config (raw default, processed for SFT/training), renderers/mm_store (offload, fingerprints, mmraw refs), and merge_multi_modal_data so bridge turns concatenate prior media without mutating the caller’s sidecar. generate() now builds vLLM features from raw refs only—removing renderer-specific torch/vLLM encoding—and expects offloaded images at every slot.

Breaking / API: VL renderer constructors no longer take processor=; processors lazy-load only for multimodal_output="processed". image_cache_max is removed from VL configs. New optional vision extra for Pillow/torch when using processed mode. Tests require file:// image fixtures for raw render parity.

^{Reviewed by Cursor Bugbot for commit a49e0fc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new multimodal rendering mode ('raw' vs 'processed') that changes the default behavior from shipping processed tensors to shipping raw image references. The scope spans new infrastructure (mm_store.py), client serialization changes, and modifications to multiple renderers - warranting human review for this runtime behavior change.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e64cc58. Configure here.}