feat(rocmfp4): backend for the ROCmFP4/ROCmFPx 4-bit formats on AMD RDNA3.5 APUs (Strix Halo)

[walcz-de]

What this adds

A new backend, rocmfp4, carrying the ROCmFP4 / ROCmFPx 4-bit weight-quantization formats (ggml types 100–107) for AMD RDNA3.5 APUs — gfx1150/1151/1152/1153, i.e. Strix Point, Strix Halo, Gorgon Point and Gorgon Halo. Stock llama.cpp rejects these tensor types (unknown type q4_0_rocmfp4_fast), which is precisely why this is a separate backend rather than a flag on llama-cpp.

The formats originate in charlie12345/ROCmFPX (MIT). Because that tree predates the MMQ config refactor (#24127 in llama.cpp) by several weeks, the formats were ported onto a current llama.cpp base — including re-expressing the FP4 MMQ load-tile kernels against the new ggml_cuda_mmq_config/sram_layout architecture — and are maintained at walcz-de/llama.cpp-ROCmFP4, with copyright headers and attribution intact.

Published weights this serves today: kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF.

What the format buys — honestly

Memory and bandwidth, not arithmetic. There is no FP4 matrix instruction on RDNA3.5 or RDNA4 (only CDNA4 has one), so the matmul runs as int8 dot products. The win is ~22% smaller files than Q4_K_M at perplexity parity (5.8877 vs 5.8926 on wikitext-2 per the published card) — on unified-memory APUs that can decide whether a model fits at all.

Measured end-to-end on a Radeon 8060S (gfx1151, Strix Halo), 27B model, via this backend inside LocalAI:

	result
prefill pp128	317–335 t/s
decode tg32	14.0–14.3 t/s
decode with MTP draft head (spec_type:draft-mtp, n_max 4)	30.4 t/s (2.13×)
MTP losslessness	identical routing accuracy (43/60) and coding pass rate (7/8) with and without MTP

llama-cpp stays the recommendation for everything else; the backend description in index.yaml says so explicitly (same tone as the vllm-cpp entry).

Non-displacement, by design

• Deliberately NOT in engineNamePreferenceRules — an AMD entry would make the gallery prefer FP4 variants on every AMD host. Per the adding-backends guide, leaving it out ranks it below every known engine, which is the intent for a niche format.
• Importer is preference-only (extends the existing llama-cpp drop-in list, like ik-llama-cpp/turboquant): ROCmFP4 GGUFs carry ordinary .gguf names, so there is no safe auto-detect signal. Without an explicit preference an import stays llama-cpp.
• Verified at runtime: with rocmfp4 installed, a plain Q4_K_M GGUF without backend: still auto-loads on llama-cpp (backends=[llama-cpp vllm] — the new backend never enters the candidate list).

Scope: Linux/AMD only, deliberately

The kernels are ROCm/HIP for RDNA3.5; there is neither ROCm nor the target hardware on Apple silicon, and NVIDIA already has native FP4 paths upstream. Hence no includeDarwin entry and a single hipblas matrix entry — stated here rather than omitted silently, per the checklist.

Checklist coverage

• backend/cpp/rocmfp4/ wrapper following the bonsai pattern (weight types decoded inside libllama; shared gRPC server reused verbatim, no allow-list patch needed)
• Dockerfile.rocmfp4 + .docker/rocmfp4-compile.sh (from-source and prebuilt-base stages). The Dockerfile carries a documented local-build escape hatch (LLAMA_REPO/ROCMFP4_VERSION ARGs, empty defaults) so the fork can be built from a local mirror; empty values are unset in the compile script so the Makefile pin always wins in CI
• .github/backend-matrix.yml entry (hipblas/amd64, prebuilt gRPC base, tag-suffix matches index.yaml URIs exactly)
• scripts/lib/backend-filter.mjs: path filter above the generic llama-cpp suffix + shared-input rule (make test-ci-scripts: 41 pass, 0 fail)
• bump_deps.yaml registration; pin kept in ROCMFP4_VERSION?= form the bot greps for
• backend/index.yaml meta + latest/development image entries (YAML validated)
• Makefile wiring incl. test-extra-backend-rocmfp4 loading a real ROCmFP4 GGUF (fork-only decodable, bonsai pattern; artifact is 13.75 GiB — workstation test, noted in the target comment)
• Importer preference swap + table-driven test (passes in isolation; the suite's live-HF specs for unrelated importers flake identically with and without this change)
• Docs: docs/content/features/backends.md

Happy to adjust scope, naming or the escape hatch if the maintainers prefer a different shape. We run this backend in production on Strix Halo and intend to maintain it (pin bumps via the registered bump_deps entry, rebases of the format fork against upstream llama.cpp).

🤖 Generated with Claude Code

https://claude.ai/code/session_01SeuYBDnA3qH3ngNagKDNjA

[localai-org-maint-bot]

The backend wiring needs a cleanup before this can run through CI:\n\n- In .github/workflows/bump_deps.yaml, the new walcz-de/llama.cpp-ROCmFP4 item is indented at the matrix level instead of under matrix.include. The following ds4 item is therefore left unexpectedly over-indented, making the workflow YAML structurally invalid.\n- backend/cpp/rocmfp4/patches/README.md is still the Bonsai/Prism copy: it identifies this as the bonsai backend, points at the PrismML prism fork, and describes Bonsai CI inputs. Please rewrite it for rocmfp4 and the walcz-de/llama.cpp-ROCmFP4 fork.\n\nDCO is also failing, but that attestation must be fixed by the human contributor.

[walcz-de]

Thanks for the careful review — all three addressed:

• bump_deps.yaml: the new item is re-indented under matrix.include alongside its neighbours; the ds4 item below it is back at its normal level. Workflow YAML validates again.
• backend/cpp/rocmfp4/patches/README.md: rewritten for rocmfp4 and the walcz-de/llama.cpp-ROCmFP4 fork — including why the directory is expected to stay empty (the fork is rebased onto the same LLAMA_VERSION the shared gRPC server is written against, unlike the Bonsai/Prism situation it was templated from).
• DCO: the failure was a name mismatch between the commit author (stefanwalcz) and the sign-off (Stefan Walcz), same address on both sides. The branch's author fields are normalized so author and sign-off now match exactly on every commit; the attestation itself is unchanged.

[localai-org-maint-bot]

The earlier workflow, patches README, and DCO issues are fixed. One copied Bonsai artifact remains in the same helper: backend/cpp/rocmfp4/apply-patches.sh still says it applies Bonsai patches to the PrismML prism fork, and its success message says all bonsai patches applied successfully. Please rewrite those comments and the message for rocmfp4 / walcz-de/llama.cpp-ROCmFP4. This is documentation and diagnostic output only; no behavior change is needed.

[walcz-de]

Done — apply-patches.sh header and success message rewritten for rocmfp4 / walcz-de/llama.cpp-ROCmFP4, including why patches/ is expected to stay empty here. I also swept the backend directory for further copied identity: the two remaining bonsai mentions in the Makefile are deliberate comparisons to the pattern this wrapper follows ("like bonsai, the additions are weight types decoded inside libllama"), not leftovers — happy to reword those too if you'd prefer.

[localai-org-maint-bot]

The latest commit removes the remaining Bonsai/Prism text from apply-patches.sh and keeps the helper behavior unchanged. DCO passes. Good from my side, @mudler.

[localai-org-maint-bot]

Review pass. Mechanically this is one of the more complete backend additions I have checked against .agents/adding-backends.md. What needs a decision is the supply chain, not the code.

Correctly wired, verified individually: matrix entry, bump_deps.yaml registration with the pin in the Makefile rather than a prepare.sh, the inferBackendPath branch placed above the generic llama-cpp suffix at scripts/lib/backend-filter.mjs:81, the backendChanged fork rule at :161, all six Makefile touchpoints, index.yaml meta plus concrete entries with matching tag suffixes, importer extension as preference-only, and the docs/content/features/backends.md line. The engineNamePreferenceRules omission is deliberate, explained in the body, and safe: IsBackendCompatible (pkg/system/capabilities.go:515-521) substring-matches "rocm" inside "rocmfp4", so the backend is filtered off non-AMD hosts anyway.

The decision, for a maintainer: backend/cpp/rocmfp4/Makefile:4 pins https://github.com/walcz-de/llama.cpp-ROCmFP4, the submitter's personal fork, and .github/workflows/bump_deps.yaml is wired to auto-bump it nightly from branch rocmfp4. That means unreviewed commits from a personal repo flow into a published LocalAI image on a cron. The neighbouring entries track antirez/ds4 and the PrismML fork, so there is precedent for third-party pins, but a personal fork on a nightly auto-bump is a different risk profile. Either accept it explicitly, or drop the bump_deps entry and bump by hand.

Secondary scope note: a whole new ROCm-only image for one published model, justified in the body but still a per-release CI cost.

Smaller items:

• backend/Dockerfile.rocmfp4 duplicates the ARG LLAMA_REPO / ARG ROCMFP4_VERSION / ENV ... block verbatim twice, back to back, in the builder-fromsource stage.
• .docker/rocmfp4-compile.sh carries a make rocmfp4-cpu-all branch with arm64 gcc-14 handling, and the Makefile defines rocmfp4-avx/avx2/avx512 targets, none of which any matrix entry builds (there is a single hipblas amd64 entry). run.sh also carries the whole Intel Level Zero block copied from llama-cpp's. Dead paths in a ROCm-only backend.
• No test case added to scripts/lib/backend-filter_test.mjs for the new suffix, though make test-ci-scripts runs that suite.
• .docker/rocmfp4-compile.sh sits outside backend/cpp/rocmfp4/ and is not registered in SHARED_BUILD_INPUTS, so editing only that script rebuilds nothing. To be fair, bonsai-compile.sh and llama-cpp-compile.sh have the same gap, so this follows an existing pattern rather than deviating.

DCO: gh pr checks reports DCO fail on this PR.

[walcz-de]

Thanks — that is a genuinely useful review, and the ordering of concerns is right. Everything below is pushed except where noted.

On the supply chain

One correction about the mechanism, because it changes the shape of the decision rather than just the wording.

The nightly job does not commit anything. .github/workflows/bump_deps.yaml ends in peter-evans/create-pull-request@v8: it pushes to ci-forks/LocalAI and opens a pull request (bump/ROCMFP4_VERSION), like every other entry in that matrix. Both automerge workflows (dependabot_auto.yml, localaibot_automerge.yml) sit in .github/workflows/disabled/. And that review is not nominal — #11136 is a revert of a llama.cpp bump, so these PRs do get read and occasionally undone.

So no unreviewed commit reaches a published image on a cron; a maintainer merges, exactly as for ggml-org/llama.cpp, ikawrakow/ik_llama.cpp, the PrismML fork, and TheTom/llama-cpp-turboquant — which is tracked on a feature branch of a third-party fork today.

Your underlying point still stands on its own, though: with ~25 bump PRs a night that review can get thin, and "personal fork" is fair to weigh separately from "third-party fork". So, your call and we are fine either way:

• Keep the entry — same gate as its neighbours; or
• Drop it and we bump by hand, either for an initial adoption period or permanently.

Say the word and it is out in the next push. We would rather you picked than have us pre-empt a decision that spends your risk budget.

On fork freshness, since it is the same topic: the fork sat on llama.cpp 3466812d while llama-cpp builds 9cffdcc. We rebased it to check what that costs — eight commits, zero conflicts, and the only MMQ-area movement in those 22 commits is 16 deleted lines (mmq-vec-dot.cuh -11, mmq.cuh -5). Nothing had to be re-expressed. The rebased branch is walcz-de/llama.cpp-ROCmFP4:rocmfp4-rebase-9cffdcc; the pin here still points at the rocmfp4 branch that bump_deps tracks, so the two stay consistent. Happy to bump it if you would rather the fork tracked the current pin.

Scope: one model, or one architecture?

Your note about a whole ROCm image for a single published model was fair as stated, and this push changes the fact underneath it.

The backend shipped grpc-server only, so FP4 could be consumed but not produced — usable weights were limited to the one published file. The fork already carried the quantization recipes; they were simply never built. This push adds --target llama-quantize and copies the binary out.

That is not just a CLI in a directory. LocalAI's own quantization service resolves the tool from PATH or from its backend directory (backend/python/llama-cpp-quantization/backend.py::_find_quantize_binary), so with this binary in place the existing /app/quantize page drives FP4 end to end — HuggingFace download, convert_hf_to_gguf.py, quantize, import — using its custom-type field. No change to the UI or the service.

Measured on a Radeon 8060S / Ryzen AI MAX+ 395 (gfx1151), not derived:

run	result
Qwen/Qwen3-0.6B -> Q4_0_ROCMFP4_FAST	405 MB, 34 s total
Qwen/Qwen3-0.6B -> Q4_0_ROCMFP4_STRIX_LEAN	432 MB, 31 s (19 s download+convert, 11 s quantize)
Qwen/Qwen3.6-35B-A3B (71.9 GB) -> Q4_0_ROCMFP4_STRIX_LEAN	17.7 GB, 20 min 16 s (download 10 / convert 2 / quantize 8)

Output GGUFs carry general.file_type 103 and 106 respectively, read back from the header rather than inferred from the filename.

The other half of the scope answer is hardware, not weights: gfx1150/1151/1152/1153 — Strix Point, Strix Halo, Gorgon Point, Gorgon Halo. Four shipping families on unified memory, which is exactly where smaller weights decide whether a model fits at all.

CPU features in the grpc build

Worth calling out because it changes an existing build line: that build had every CPU feature off, inherited from the portable fallback profile. Quantization is pure CPU work over every weight, and this backend targets integrated GPUs whose CPU sits on the same package and is always Zen 4/5 — AVX, AVX2, AVX512, FMA, F16C and BMI2 are guaranteed present on any machine that can run this backend at all. So they are on now, and nothing portable is lost. The portable rocmfp4-fallback build, which serves inference, is untouched.

Measured limitations — please read this before merging

Having produced weights with the shipped quantizer, we ran our own evaluation harnesses, and the result is not uniformly good. Reporting it here because the PR body currently quotes perplexity parity, and perplexity does not measure what broke.

Base model Qwen/Qwen3.6-35B-A3B, quantized twice, against a published APEX quant of the same base. Same llama.cpp revision for both backends, same chat template, same sampler settings, same MCP tool set:

axis	APEX-Compact (16.1 GB)	STRIX_LEAN (17.7 GB)	STRIX (17.8 GB)
deterministic coding, 8 tasks	7/8	7/8	7/8
classification, 60 items, no tools	65.0 %	56.7 %	53.3 %
agent parcours, 10 tasks	7/10	1/10	0/10
coder-bench, 4 tasks	(reference model 4/4)	0/4	—

Generation is preserved; discrimination is not. Free chat is fine, deterministic coding is lossless, throughput is up ~11% single-stream. But choosing correctly between similar options degrades badly — and the classification row is the one that matters, because it uses no tools at all. So this is not a tool-calling quirk; it is the model's ability to pick.

STRIX differs from STRIX_LEAN in exactly one variable (token embeddings and output layer at Q6_K instead of Q5_K). It did not help, so the loss is not at the edges.

We are not proposing a change from this — the format does what the PR says it does, and for generation-shaped and throughput-bound work it is a real win. But anyone pointing this backend at an agent should measure first, and that belongs in the documentation rather than in each user's afternoon. Happy to add a paragraph to the index.yaml description or backends.md if you would like it worded a particular way.

Related, and filed separately as an issue on our fork: ggml_validate_row_data() only covers the two FP4 types (100, 101), not the FPx family (102/103/104/107). Every *_ROCMFPX* recipe — including all the *_AGENT variants — aborts with invalid type 103 / quantized data validation failed. They are selectable in llama-quantize and none of them can be produced. Fix is on our side and does not block this PR, but you should know the advertised type list is currently wider than what works.

Smaller items — all addressed

• Duplicated block: the second ARG LLAMA_REPO / ARG ROCMFP4_VERSION / ENV ... block in builder-fromsource is gone.
• Dead paths: rocmfp4-avx/avx2/avx512 and rocmfp4-cpu-all removed from the Makefile, the unreachable CPU-image branch (with its arm64 gcc-14 handling) removed from .docker/rocmfp4-compile.sh, and the Intel Level Zero block removed from run.sh. rocmfp4-fallback and rocmfp4-grpc stay — the hipblas entry does build those two plus rocmfp4-rpc-server, so only the four you named were actually dead.
• Test coverage: scripts/lib/backend-filter_test.mjs gains a rocmfp4 fixture and a case asserting that a change under backend/cpp/rocmfp4/ resolves to this backend and rebuilds nothing else — which is what the inferBackendPath ordering above the generic llama-cpp suffix exists for. The existing fork-retrigger test now covers rocmfp4 too. node --test scripts/lib/backend-filter_test.mjs: 42/42.

Two notes back

DCO is green. The check completed at 2026-08-21T08:17:47Z and reports success; the name/sign-off mismatch was fixed in the 21 Aug push. That line was probably read from a stale view — worth a re-check on your side.

SHARED_BUILD_INPUTS: correct, and thank you for noting it follows the existing pattern rather than deviating. bonsai-compile.sh and llama-cpp-compile.sh have the same gap. Happy to register all three in a separate PR — changing two established backends from inside this one seemed wrong.

---

Disclosure: prepared with AI assistance (Claude). All numbers above are from runs on the hardware named, not estimates.