Repository navigation
Multimodal VLM training fetches http(s) URLs taken from dataset rows, outside the _http.py SSRF guard and despite offline mode (hand to S) (CR-A01-011) #462
Description
Wave 2 · theme D2-1: Security and integrity: medium/large effort · highest severity High · 1 finding(s) · effort medium
Summary
An operator fine-tunes a VLM on a third-party image-caption dataset. A poisoned row sets image to http://169.254.169.254/latest/meta-data/... or to an internal service URL. The training host makes the request from inside the VPC or air gap, even with offline: true. A row naming a local image path makes the host read that file into training.
Primary finding
CR-A01-011 — Multimodal VLM training fetches http(s) URLs taken from dataset rows, outside the _http.py SSRF guard and despite offline mode (hand to S)
severity High · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
What happens. With model.multimodal.enabled, data.py hands the dataset to TRL unchanged. TRL's DataCollatorForVisionLanguageModeling then passes the row's image value to processor(images=...). In transformers 5.13, the processor's image_processor.fetch_images -> image_utils.load_image runs httpx.get(url, follow_redirects=True) for any http:// or https:// string, and PIL.Image.open for any existing local path. The URL comes from an IN-04 dataset row (TC-U), so the request never reaches forgelm/_http.py: no scheme allowlist, no private/loopback/link-local refusal, redirects are followed and there is no timeout floor. model.offline: true does not stop it, because only HF env vars are set. The config field description, configuration.md:36 and config_template.yaml:146 all advertise the column as 'image paths or URLs'.
Where.
forgelm/config.py:134forgelm/data.py:648-666forgelm/data.py:714-716forgelm/model.py:381-388forgelm/trainer.py:1029-1034forgelm/config.py:268-271forgelm/model.py:579-583forgelm/cli/_training.py:141-148forgelm/cli/_pipeline.py:1564-1566docs/reference/configuration.md:36
Evidence.
- Built a tiny Llava VLM (tiny-qwen text tower + CLIPImageProcessorPil) and served PNGs from a loopback HTTP server (outputs/.../srv.py). cfgB.yaml sets offline: true and multimodal.enabled: true. Its rows are {image: 'http://127.0.0.1:PORT/B.png', messages: [...]}. The run ended 'ForgeLM Training Pipeline Completed Successfully!' with EXIT=0 (runB.log). srv_hits.log recorded 6 'GET /B*.png from 127.0.0.1' requests. Library path (venv transformers 5.13.0): processing_utils.py:729-730 calls fet…
Recommended fix. Handed to S. In _passthrough_multimodal, resolve URL-valued image cells through forgelm._http.safe_get, or reject them unless the operator opts in, and confine local paths to the dataset directory. Honour model.offline by refusing URL cells. Fix the 'paths or URLs' wording.
Acceptance criteria
- A regression test reproduces the failure scenario and asserts the corrected behaviour.
- Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
- Every finding in this issue is resolved or explicitly closed with a reason.
Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.
Activity