feat(florence2): add native Florence-2 image-to-text support

[ssss141414]

Summary

This Draft replaces #1098 without closing it and adds native Transformers Florence-2 image-to-text support through a split image encoder and cached text decoder. The shipped engineering effort is L2: class-wide export/runtime support plus generated CPU recipes and tests. Both required CPU tuples pass the committed L3 ceiling; requested fp16 is honestly reported as a mixed composite with an fp32 encoder and fp16 decoder.

Model metadata

What the model does

Florence-2-base is a prompt-conditioned image-and-text encoder-decoder model. A user supplies an image and a task prompt; the model autoregressively generates text tokens that Florence2Processor can return as captions or parse into OCR text, boxes, labels, phrase-grounding results, or polygons.

• Evidence/confidence: pinned microsoft/Florence-2-base model card and native Transformers 5.14.1 Florence2ForConditionalGeneration / Florence2Processor sources (verified).

Primary user stories

• Supply an image with <CAPTION>, <DETAILED_CAPTION>, or <MORE_DETAILED_CAPTION> to obtain descriptions at different levels of detail (verified).
• Supply an image with an OCR or detection prompt to obtain text, labeled bounding boxes, or OCR quadrilaterals (verified).
• Supply an image plus a phrase or region prompt to obtain grounded boxes, categories, descriptions, or segmentation polygons (verified).

Supported tasks

Task	Support surface	Evidence	Confidence
image-text-to-text	checkpoint, Transformers	Model pipeline task; native processor and conditional-generation class	verified
image-to-text	checkpoint semantics mapped to WinML	WinML target-task bridge maps to image-text-to-text	mapped

Model architecture

Florence2ForConditionalGeneration
|-- Florence2Model
|   |-- Vision backbone: 4 stages, depths [1, 1, 9, 1], widths [128, 256, 512, 1024]
|   |   `-- Each block: spatial window attention + channel attention + MLP
|   |-- Multimodal projector: learned 2D/temporal positions, 1024 -> 768
|   |-- Text encoder: BART-family encoder layers x 6, hidden 768, heads 12
|   `-- Text decoder: BART-family decoder layers x 6, hidden 768, heads 12
|       |-- Causal self-attention with KV cache
|       `-- Cross-attention over encoder hidden states
`-- LM head: 768 -> vocabulary 51289

• Evidence/confidence: pinned checkpoint configuration plus native Transformers 5.14.1 Florence-2 configuration/model sources (verified).

Validation and support evidence

Baseline

• Exact current-main baseline commit: b7a7b893d757bb8e52b5b5a594a9aa72f0c0fff4.
• Frozen current-main baseline used WinML 0.2.0.
• Recipe-free CPU build: FAIL, exit 2 before model loading and before output creation because no OnnxConfig was registered for model_type='florence2' with task image-text-to-text.
• Auto-config: no starting recipe was emitted because registration failed.
• Perf: not reachable because the baseline produced no ONNX encoder/decoder artifacts.
• Eval floor: winml eval --schema --task image-to-text exited 0 and exposed image/text columns plus composite encoder/decoder inputs, but main had no default image-to-text dataset, no prompt field, and invoked the evaluator with empty text. Task evaluation therefore remained unreachable from the failed baseline build.
• Optimum probe: no vendor, post-WinML, or WinML-added Florence ONNX registration was found (UNREGISTERED). The established split encoder/decoder path was therefore a class-wide code fix, not a recipe-only override.

Goal

• Effort: L2, a metadata-derived class-of-models implementation for native florence2 export/runtime support.
• Committed ceiling: L3.
• Committed outcome: L2 engineering scope.
• Success definition: L0 split composite build and structure; L1 assembled-composite CPU runtime; L2 PyTorch/ONNX component, recursive-cache, token, and caption parity; L3 pinned Flickr30k CIDEr/CER evaluation.
• No downstream role changed the frozen effort, goal, outcome, target EP, or required precision plan.

Outcome

• Overall: PASS. Shipped L2 code, four generated CPU component recipes, registration/docs/test coverage, and reached L3 PASS for both required tuples.
• fp32 means encoder fp32 plus decoder fp32.
• Requested fp16 is explicitly mixed: encoder fp32 plus decoder fp16. The filename records the requested tuple; each component config serializes its actual precision. This is not a full-fp16 claim.
• No required tuple is deferred. The fp16 encoder stays fp32 because full-fp16 encoder attention amplified small projector/scatter error to an unacceptable hidden-state outlier; tested fp16 op/node exclusion policies did not remove it. No threshold was weakened.
• Quality gates: models 1516 passed, 6 skipped, 2 xfailed; commands/config/build/compiler/session/eval 3386 passed, 9 skipped; analyze 1515 passed, 45 skipped; optim 565 passed, 16 skipped, 1 xfailed; remaining non-hardware 846 passed, 2 skipped, 1 deselected. Ruff, whole-package mypy across 434 source files, license validation, and git diff --check passed.

Per-tuple results

• Dataset provenance: nlphuji/flickr30k@cd91f9a00273ce2e1584511cba8c10b917c488a3, split test, streaming subset source indices 0..7 (n=8).
  All runtime rows use CPUExecutionProvider / cpu. L1 used one warmup and five measured composite generations with max_new_tokens=8; RAM is peak RSS delta and VRAM was not measured.

Tier	Requested precision	Actual composite precision	Verdict	Evidence
L0	fp32	encoder fp32 + decoder fp32	PASS	Loadable external-data graphs; encoder sequence 585, hidden width 768; decoder vocabulary 51289 and six cache layers
L0	fp16	encoder fp32 + decoder fp16	PASS	Same structural contract; generated configs and recipes serialize mixed component precision
L1	fp32	encoder fp32 + decoder fp32	PASS	mean 2210.7615 ms; p50 2238.5597 ms; p90 2272.8940 ms; 0.452333 images/s; 4.070995 generated tokens/s; RAM +2563022848 bytes
L1	fp16	encoder fp32 + decoder fp16	PASS	mean 2269.7578 ms; p50 2265.8664 ms; p90 2304.3110 ms; 0.440576 images/s; 3.965181 generated tokens/s; RAM +2591752192 bytes
L2	fp32	encoder fp32 + decoder fp32	PASS	encoder cosine 0.999999999854, max delta 0.01077271; prefill max delta 0.00001621; cached max delta 0.00001240; exact token IDs and caption
L2	fp16	encoder fp32 + decoder fp16	PASS	encoder cosine 0.999999999854, max delta 0.01077271; fp16 decoder prefill max delta 0.00095296; cached max delta 0.00651455; exact token IDs and caption
L3	fp32	encoder fp32 + decoder fp32	PASS	Flickr30k test, streaming indices 0..7, <CAPTION>, n=8, zero skips; CIDEr 0.7842, CER 0.8713
L3	fp16	encoder fp32 + decoder fp16	PASS	Flickr30k test, streaming indices 0..7, <CAPTION>, n=8, zero skips; CIDEr 0.7842, CER 0.8713

For both L2 tuples, candidate-produced cache was consumed recursively and all four reference/candidate encoder-cache crossings selected cached-step token 102. The exact decoded result was </s><s>a blue background with a white border</s>. The low L3 values are reported as measured and pass under the frozen charter; they are not characterized as high quality or compared against an invented threshold.

Delta

• Baseline config/recipe: none was emitted, so there are no baseline JSON-pointer values to overwrite or claim as changed.
• Added four generated component recipes under examples/recipes/microsoft_Florence-2-base/cpu/cpu/: fp32 encoder/decoder and requested-fp16 encoder/decoder.
• New recipe values: fp32 encoder and decoder /precision: "fp32"; requested-fp16 encoder /precision: "fp32" with /quant: null; requested-fp16 decoder /precision: "fp16" with fp16 quantization. The encoder recipes are intentionally identical across requested tuples.
• Added native florence2 split-export registration and class-wide behavior for semantic dummy inputs, component precision propagation, registry-driven composite resolution, decoder KV-cache position/advancement, and prompt-aware image-to-text evaluation. The implementation is keyed by model metadata/class, not a checkpoint ID.
• Preserved current-main request/runtime device separation, component EP/device forwarding, skip_optimize, export compatibility policy, internal quantization behavior, and analyze behavior while reconciling overlapping shared paths.
• The production recipe README remains untouched.

Analyze summary

Static winml analyze --ep all completed with seven EP rows per fresh artifact. Its partial/unsupported findings describe static rule coverage, not runtime execution; CPU runtime acceptance is the measured evidence above.

Component-level summary

Artifact	Architecture coverage	Mapping confidence	Actionable findings
fp32 encoder	vision tower; multimodal projector; 6-layer text encoder; encoder handoff	hierarchy-tag/source mapped	NvTensorRTRTX: NonZero unsupported; QNN: Transpose, NonZero, and Gather partial
requested-fp16 encoder	same fp32 encoder artifact and regions	byte-identical to fp32 encoder	same findings as fp32 encoder
fp32 decoder	6-layer causal self-attention/cache; cross-attention; decoder layers; LM head; handoff	hierarchy-tag/source mapped	no additional unsupported type reported; QNN has partial decoder op coverage
fp16 decoder	same decoder regions with fp16 compute and fp32 I/O	hierarchy-tag/source mapped	no additional unsupported type reported; QNN has partial decoder op coverage

The final moved-main handoff reports complete artifact-level operator counts but does not re-emit a component-node mapped/unmapped total; none is fabricated here.

Op-level summary

Artifact	Graph	Dominant operators	EP roll-up
fp32 encoder	1319 ops / 27 types	Reshape 399; Transpose 250; Gemm 132; Add 117; LayerNormalization 66; MatMul 61; Conv 52	OpenVINO rules support the substantive types; NvTensorRTRTX reports NonZero unsupported; QNN reports partial types above
requested-fp16 encoder	1319 ops / 27 types	byte-identical to fp32 encoder	same roll-up as fp32 encoder
fp32 decoder	354 ops / 19 types	Reshape 99; Gemm 60; Transpose 54; Add 34; MatMul 25; LayerNormalization 19	no unsupported type reported in the final decoder rows; QNN remains partial
fp16 decoder	380 ops / 19 types	Reshape 99; Gemm 60; Transpose 54; Add 34; Cast 27; MatMul 25	no unsupported type reported in the final decoder rows; QNN remains partial

CUDA, DirectML, MIGraphX, and TensorRT rule-less rows classify operator types as unknown rather than proving support or failure.

Reproduce commands

These are the portable public commands supplied by the tester for build and static analysis. L1 composite perf, L2 recursive parity, and L3 pinned-dataset metrics were produced by the authoritative tester harness; that handoff did not publish portable standalone harness commands, so none are invented here.

$OUTPUT_DIR = Join-Path $PWD 'florence2-validation'
winml build -m microsoft/Florence-2-base -o $OUTPUT_DIR/fp32 --ep cpu --device cpu --precision fp32 --no-optimize --no-compile --rebuild
winml build -m microsoft/Florence-2-base -o $OUTPUT_DIR/fp16 --ep cpu --device cpu --precision fp16 --no-optimize --no-compile --rebuild
winml analyze -m $OUTPUT_DIR/fp32/encoder_model.onnx --ep all --format json -o $OUTPUT_DIR/analyze-encoder.json --overwrite
winml analyze -m $OUTPUT_DIR/fp32/decoder_model.onnx --ep all --format json -o $OUTPUT_DIR/analyze-decoder.json --overwrite

[ssss141414]

Independent reviewer verdict: REQUEST_CHANGES

Reviewed exact code SHA b1dd17ead27d9054e6b464a6c0565fb9a4e1f1ac after the body-only repair.

Code findings are unchanged: the sealed 26-path candidate remains clean and hash-identical, git diff --check passes, the focused invalidated suite passes (1,251 passed, 6 skipped, 2 xfailed), mixed-fp16 L2 parity reruns, and all 9 required GitHub checks are successful. Shipment metadata is otherwise correct: Draft, model-scale-by-skill, and zero review threads or inline comments requiring action.

Blocking explainer-owned issue:

• The live PR body does not satisfy the adding-model-support PR contract. Its top-level order is Replacement, What changed, Precision contract, Moved-main reconciliation, Validation, Quality gates; the required order is Summary, Model metadata, Validation and support evidence.
• Model metadata must include, in order, What the model does, Primary user stories, Supported tasks, and a source-derived Model architecture tree with durable evidence/confidence and no scratch paths or hashes.
• Validation and support evidence must include all seven subsections: Baseline, Goal, Outcome, Per-EP/device/precision results (including honest fp32 and requested-fp16 mixed tuple perf/eval evidence), Delta, Analyze summary (component and op level), and Reproduce commands.
• Preserve the existing mixed-precision truth: requested fp16 is encoder fp32 plus decoder fp16. Include baseline commit/version and auto-config result, committed Goal/Outcome, exact shipped tuple matrix and concrete perf/eval values, recipe/code delta, compact per-artifact analyze evidence, and portable copy-pasteable commands. Do not expose local scratch paths or evidence hashes.

No code change is requested. Update the PR description only, then re-enter independent review against the same SHA.

[ssss141414]

Independent reviewer verdict: REQUEST_CHANGES

Reviewed exact head b1dd17ead27d9054e6b464a6c0565fb9a4e1f1ac (single parent b7a7b893d757bb8e52b5b5a594a9aa72f0c0fff4) from a fresh detached checkout against the latest adding-model-support reviewer/orchestrator contracts.

Blocking finding

The published PR body does not satisfy the required shipment hierarchy. agents/reviewer.md requires the top-level order Summary -> Model metadata -> Validation and support evidence; the validation section must contain Baseline, Goal, Outcome, per-EP/device/precision results, Delta, Analyze summary, and Reproduce commands. The current free-form Replacement / What changed / Precision contract / Moved-main reconciliation / Validation / Quality gates body omits the frozen four-part model metadata, the source-owned baseline/goal/outcome/delta fields, the complete tuple rows (including throughput, memory, dataset revision/subset), component-plus-op analyze summary, and reproduction commands.

Action (explainer-owned): replace the body with the contract structure using only the frozen handoffs. Include:

• Summary with the honest L3/L2 contribution outcome.
• Model metadata in order: what the model does, primary user stories, supported tasks, and the collapsed source-derived architecture tree, preserving evidence/confidence without scratch paths or hashes.
• Validation and support evidence with baseline commit b7a7b893d757bb8e52b5b5a594a9aa72f0c0fff4 and recipe-free failure command/result; Goal L3; Outcome L2; explicit CPU/cpu fp32 and requested-fp16-mixed rows; L1 mean/p50, throughput, and memory; L2 numeric/cache/end-to-end results; L3 nlphuji/flickr30k@cd91f9a00273ce2e1584511cba8c10b917c488a3, test indices 0..7, <CAPTION>, CIDEr/CER, and zero skips; producer Delta; compact component-level and op-level Analyze summary; and exact Reproduce commands.

Independently verified non-blocking state

• Metadata/scope: OPEN Draft, model-scale-by-skill, base main, one commit, 26 changed paths, production recipe README untouched, mergeable clean.
• Seven overlaps were reconciled without dropping current-main request/runtime device separation, component EP/device forwarding, skip_optimize, export compatibility policy, quantization behavior, or analyze behavior.
• Implementation is native Transformers 5 Florence-2 (Florence2Config, Florence2ForConditionalGeneration, Florence2Processor) with registry-driven split export, semantic dummy inputs, six-layer static cache, and no checkpoint-ID branch.
• Requested fp16 is reported and serialized as encoder fp32 plus decoder fp16; recipes use nested cpu/cpu paths. No obsolete flat recipes, --trust-remote-code route, or catalog edit is present.
• Sealed moved-main evidence covers L0-L3, recursive cache parity, artifact structure, ruff, mypy, license, diff-check, and all five non-hardware pytest partitions.
• Exact-SHA GitHub checks: 9/9 completed SUCCESS (Analyze (Python), CodeQL, license/cla, lint, test (analyze), test (commands), test (models), test (optim), test (remaining)).
• Reviews: 0; inline comments: 0; conversation comments: 0; review threads: 0 unresolved.

Keep the PR Draft and retain model-scale-by-skill. Re-request independent review after the body is corrected; code/artifact changes are not requested by this finding.

[ssss141414]

Independent final metadata reviewer verdict: REQUEST_CHANGES

Reviewed the fresh live PR metadata at exact head b1dd17ead27d9054e6b464a6c0565fb9a4e1f1ac. Formal review is not valid because the authenticated account ssss141414 owns the PR, so this is the structured terminal reviewer comment.

Blocking metadata findings:

• Required H3 Per-tuple results is absent; the live body uses Per-EP/device/precision results.
• Required H3 Analyze summary is absent; the live body uses Analyze summary - component level and op level.
• The reproduce block exposes the local scratch-style reference temp/florence2-validation; the final public body must contain no local scratch references or hashes.

All other requested gates pass: exact H2 order (Summary, Model metadata, Validation and support evidence); honest requested-fp16 coverage as encoder fp32 plus decoder fp16; sealed one-commit, 26-path code scope at the exact SHA; 9/9 exact-SHA checks completed successfully; OPEN Draft with model-scale-by-skill; zero formal reviews, zero inline review comments, and no unresolved review threads. The two existing actionable conversation comments predate this final-body inspection.

No code change is requested. Repair only the three metadata items above, then re-enter independent review against the same SHA.

[ssss141414]

Independent reviewer verdict: REQUEST_CHANGES

Reviewed the fresh live PR body and exact head b1dd17ead27d9054e6b464a6c0565fb9a4e1f1ac. Formal self-review is not valid because the authenticated account owns the PR, so this is the structured independent-reviewer issue comment.

Concrete live metadata defects

• Under exact heading ### Baseline, the body identifies WinML 0.2.0 and the baseline failure, but omits the required baseline commit b7a7b893d757bb8e52b5b5a594a9aa72f0c0fff4. This is durable repository provenance, not a forbidden scratch-artifact hash.
• Under exact heading ### Per-tuple results, both L3 rows say only Flickr30k test; they omit required dataset/revision provenance nlphuji/flickr30k@cd91f9a00273ce2e1584511cba8c10b917c488a3 (test subset, streaming indices 0..7, <CAPTION>, n=8, zero skips are otherwise present).

These omissions were explicitly requested in prior issue comment 5133753159, so that comment is not fully resolved by the current live body. Repair the PR description only; no code or artifact change is requested.

Independently verified passing gates

• Exact head: b1dd17ead27d9054e6b464a6c0565fb9a4e1f1ac; one commit, 26 changed paths, mergeable.
• Required hierarchy is exact: Summary -> Model metadata -> Validation and support evidence; all four metadata subsections and all seven validation subsections are present, including exact Per-tuple results and Analyze summary.
• No forbidden local scratch paths, workspace references, or uncommitted artifact hashes appear in the live body.
• Requested fp16 is honestly reported as mixed: encoder fp32 plus decoder fp16; L1/L2/L3 values are presented without inflating the low L3 metrics.
• Code scope matches the L2 native Florence-2 contribution: four nested cpu/cpu component recipes, generalized export/runtime/cache/eval plumbing, docs/e2e registration, and focused tests; examples/recipes/README.md is untouched.
• Shipment metadata: OPEN Draft with model-scale-by-skill.
• Exact-SHA checks: 9/9 COMPLETED/SUCCESS: Analyze (Python), lint, test (analyze), test (models), test (optim), test (commands), test (remaining), CodeQL, license/cla.
• Review surfaces: 0 formal reviews, 0 inline review comments, 0 total review threads, 0 unresolved threads. The three prior REQUEST_CHANGES records are issue comments; their heading/scratch-path defects are resolved, but the provenance omission above remains actionable.

[ssss141414]

Independent reviewer verdict: APPROVE

Fresh-read final review of the live PR body and exact head b1dd17ead27d9054e6b464a6c0565fb9a4e1f1ac. Formal self-review is not valid because the authenticated account owns the PR, so this structured issue comment records the independent reviewer verdict.

Exact provenance and contract evidence

• The required hierarchy is exact: Summary -> Model metadata -> Validation and support evidence; all four model-metadata subsections and all seven validation subsections are present in contract order.
• ### Baseline contains exact current-main baseline commit b7a7b893d757bb8e52b5b5a594a9aa72f0c0fff4, WinML 0.2.0, the recipe-free registration failure, no emitted auto-config recipe, and the resulting unreachable perf/eval baseline.
• ### Per-tuple results contains exact dataset provenance nlphuji/flickr30k@cd91f9a00273ce2e1584511cba8c10b917c488a3, split test, streaming source indices 0..7, and n=8; both L3 rows report <CAPTION>, zero skips, CIDEr 0.7842, and CER 0.8713.
• Requested fp16 remains honestly reported as encoder fp32 plus decoder fp16. The body preserves the concrete L0-L3 tuple evidence, numeric L1 perf/memory, recursive-cache L2 parity, recipe/code delta, component/op analyze summaries, and portable reproduce commands without inventing missing harness commands or thresholds.
• The live body contains no local workspace path, temp/ or scratch reference, or evidence hash. The 26 changed paths contain no scratch artifact; examples/recipes/README.md remains untouched.

Shipment gates

• PR is OPEN and Draft with label model-scale-by-skill.
• Exact head is b1dd17ead27d9054e6b464a6c0565fb9a4e1f1ac; it is one commit with sole parent and base b7a7b893d757bb8e52b5b5a594a9aa72f0c0fff4, 26 changed paths, and GitHub reports it mergeable.
• Exact-head checks are 9/9 COMPLETED/SUCCESS: Analyze (Python), CodeQL, license/cla, lint, test (analyze), test (commands), test (models), test (optim), and test (remaining).
• Review surfaces are clear: 0 formal reviews, 0 inline review comments, 0 total review threads, and 0 unresolved threads.
• The prior issue-comment blockers are resolved by the current live body: exact Per-tuple results and Analyze summary headings, no forbidden scratch reference, baseline commit in Baseline, and pinned Flickr30k revision/split/sample provenance in Per-tuple results.

No concrete reviewer-contract blocker remains. Keep the PR Draft and retain model-scale-by-skill for the intended workflow.