Read DeepSeek-V3 checkpoints in the converter - #26
Conversation
fp8 checkpoints (DeepSeek V3/R1, Kimi-K2) store weights as F8_E4M3 with a per-tile `_scale_inv` companion. Both safetensors readers now apply it. The tile size comes from `quantization_config.weight_block_size` rather than being inferred from the two shapes: inferring looks possible and is wrong whenever a dimension is not a multiple of the tile, and a compatible-but-wrong size is undetectable by shape alone. MoE tensor names differ across the family. Mixtral and Kimi use `block_sparse_moe.experts.E.w1/w3/w2`, DeepSeek uses `mlp.experts.E.gate_proj/up_proj/down_proj`. The layout is detected from what is on disk and normalised to one spelling, so the engine sees one name. Without it a conversion finds no experts and reports `0 MB [missing]` for every layer, after the download has already run. `tests/test_convert_resume.py` stubs `mxfp4` with a `types.ModuleType`, which has no `__file__`, so the new import has to resolve there or every check in that file fails at module load.
The two families disagree on MoE config keys as well as on tensor names, and this half fails differently. The engine reads `num_experts`; a DeepSeek config only spells it `n_routed_experts`, so `cfg_from_json` yields 0 experts and the bank loop in `waste_model_load` refuses the container with no diagnostic — after the conversion has completed. `num_experts_per_tok` against `..._per_token` is one letter and leaves `top_k` at 0. The manifest is WASTE's format rather than HF's, so the keys are normalised where the tensor names already are, and written only when absent so a config that already uses the canonical spelling wins. `moe_renormalize` is keyed on the field being present rather than on its value, so a plain alias of DeepSeek's `norm_topk_prob` would turn renormalisation on for a checkpoint that sets it false. It is emitted only when true.
The suite stayed green through the whole window in which `src/` applied no rotary, and it would have stayed green after a fix that pairs the wrong dims. Both have the same cause: every container the suite can reach is a Kimi, every Kimi sets `mla_use_nope`, and so nothing in `tests/` ever entered `rope_init` or `rope_apply`. This is the missing half of the previous commit. `make_test_container.py --rope` writes a DeepSeek-V3 at the 1/18 scale the file already builds a Kimi-Linear at: no `mla_use_nope`, `rope_theta` and the YaRN block copied from Kimi-K2-Instruct's config, and no `linear_attn_config` at all, which is what makes every layer MLA. All-MLA is deliberate twice over — it exercises the rotation at depth rather than in the single full-attention layer the Kimi mix leaves, and it is the shape `deepseek_ref.py` can read, since not indexing `linear_attn_config` is exactly what separates it from `kimi_ref.py`. K2's rope block rather than V3's because `beta_fast == beta_slow == 1.0` collapses YaRN's correction range to a two-dim ramp, which is the more awkward of the two to get right. The checks build their own container instead of using `$MODEL`, so they run on every host and do not wait on weights nobody can convert yet — sqliteai#26 is what makes a real V3 container, and the shape is what the engine branches on. - rotated MLA against the PyTorch oracle - chunked prefill == token-at-a-time with rotation, which holds by construction today because `mla_layer` is per-token on both paths, and is exactly the "by construction" a later batched MLA would break quietly - a rope slice wider than `WASTE_MAX_ROPE_HALF` is refused at load The first takes the same two-source shape as the Kimi oracle above it: generate from `deepseek_ref.py` where `uv` exists, fall back to a fixture where it does not, so the Linux image without `uv` runs it rather than skipping it. Unlike that one the fixture ships, because this container is generated rather than converted and so is byte-reproducible at `--seed 0` — the sidecar carries a digest of the container it was made from, so a later change to the generator's weights reads as "regenerate me" and not as an engine bug. The fixture is the reference's logits, never the engine's. `deepseek_ref.py` grows the `--dump` that `kimi_ref.py` already had, so the diff is over whole logit vectors and not a printed top-k. Verified by reverting `src/model.c` and `src/model.h` to their pre-fix state with `tests/` and `tools/` left alone: the oracle check and the refusal check both fail, which is the property that makes them worth having. Both fallback paths were exercised directly — `uv` off `PATH` passes against the fixture, and a corrupted digest skips with the regenerate message instead of reporting a divergence. `set -o pipefail` sank the refusal check on the first run, because a refused load exits non-zero and that is the point; the output is read into a variable now, with a comment saying why. Suite on this commit: 46 passed, 0 failed, 2 skipped against Kimi-Linear and K3 (43/0/2 before), 39/0/9 on the synthetic path CI takes, and `make asan` 33/0/14. Fuzzer and the 168 serve checks unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged. Verified before landing: CI 9/9, clean merge onto The check that mattered for the models this repo already ships is that none of The reasoning I want on the record is the tile size. Reading it from One thing is missing. The PR body describes Landed before #27 so that the reproduction path for the rotary fix is available |
Add opt-in exclusive model-container ownership, and cut 0.6.8. The PR wrote its entry under 0.6.7, which shipped from main while it was in review. Retargeted here to 0.6.8, with the section rewritten to cover what the release actually contains — the DeepSeek converter (#26), the MLA rotary fix (#27) and this lock — and one ABI paragraph naming every struct that moved rather than only waste_cfg. WASTE_VERSION_PATCH goes with it, per CLAUDE.md's rule that the changelog moves in the same commit as the version. The added paragraph on the lock is the part of the review that was not written down anywhere in the tree: the mechanism is host policy, container identity is a poor proxy for memory oversubscription, and the budget question stayed open as #31. Verified on the merge, against real containers rather than the synthetic path: tests/run.sh with WASTE_REF_MODEL on a default VQ3R Kimi-Linear is 54 passed, 0 failed, 2 skipped, K3 checks included, with the ownership check among them; test_lock passes against a real container; the serve suite is 211/211.
|
Follow-up on the one outstanding item from the merge review: the Nothing new is being asked there beyond what you already described. It adds two suggestions — the both-axes-partial corner, where the two independent @fab2s no obligation — if you would rather not, say so on #39 and I will write it from your description. |
Two things on top of tomatotomata's checks, which are unchanged and were right: the mutants I planted in unblock_scale — returning q untouched, transposing the block dims, dropping the [:M, :N] crop, removing the shape guard — were all caught, and expected_dequant being an independent index-array implementation rather than a second call to the code under test is what gives it that. It never ran. run.sh invoked it as bare `python3`, and torch is not a system package on the machines that test this: CLAUDE.md says it is never a repo dependency and every other torch checker goes through uv. So the whole thing reported "torch not installed" and skipped everywhere, CI included — where the Linux job is the one that installs uv, which is to say the one place it does get to run. Through run_uv now, with the guard on uv rather than python3, since without uv run_uv exits 127 and the catch-all would report FAIL where a SKIP is meant. 56 passed / 0 failed / 2 skipped becomes 57 / 0 / 2. And the missing-companion check exercised ST only. Deleting ShardReader's own `raise` in convert.py, so it returns the tensor unscaled instead, kept the suite green — a silent wrong answer in the reader sqliteai#26's description called out as "a second reader and was easy to miss". Both readers now, and that mutant is killed. The same mutation against ST survives, and should: without its guard, raw() still raises KeyError naming weight_scale_inv, so the contract the test asserts — refuses, and says which tensor — holds either way. That guard buys a better message, not a different decision, and asserting its exact prose would test the wording rather than the behaviour. Also wrote down the case nothing here can catch, which is the reason the tile size is read from config rather than inferred: 300 rows against 3 scale rows admits both 128 and 100, both pass every shape check, and the wrong one applies each scale to the wrong rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 participants
Read DeepSeek-V3 checkpoints in the converter
Two commits, both needed before
convert.pycan produce a container from aDeepSeek-V3 family checkpoint (V3, R1, Kimi-K2). Independent of #27, which
fixes a separate defect in the engine's attention path — that one builds and tests
without this, but reproducing it needs a container, so this comes first.
fp8 block scales. These checkpoints store weights as
F8_E4M3with a per-tile_scale_invcompanion. Both safetensors readers now apply it —mxfp4.STandconvert.py's ownShardReader, which is a second reader and was easy to miss.The tile size is read from
quantization_config.weight_block_sizerather thaninferred from the weight and scale shapes: inferring looks possible and is wrong
whenever a dimension is not a multiple of the tile, and a compatible-but-wrong
size passes the shape check while placing every scale on the wrong rows.
MoE tensor names. Mixtral and Kimi use
block_sparse_moe.experts.E.w1/w3/w2; DeepSeek usesmlp.experts.E.gate_proj/up_proj/down_proj. The layout is detected from what ison disk and normalised to one spelling, so the engine still sees one name. Without
it the expert probe misses on every layer and a run reports
0 MB [missing]60times after the download has already completed.
MoE config keys. The same split one level up, and this half fails differently:
src/model.creadsnum_experts, a DeepSeek config only spells itn_routed_experts, socfg_from_jsonyields 0 experts and the bank loop inwaste_model_loadrefuses the finished container with no diagnostic. Alsonum_experts_per_tokagainst..._per_token, which is one letter and leavestop_kat 0. Normalised into the manifest where the tensor names already are, andwritten only when absent so a config using the canonical spelling wins.
moe_renormalizeis handled separately becausemodel.ckeys it on the fieldbeing present rather than on its value — a plain alias of
norm_topk_probwould enable renormalisation for a checkpoint that sets it false. Emitted only
when true.
Testing
make checkwithWASTE_REF_MODELpointed at a default VQ3R Kimi-Linearcontainer.
tools/../test_fp8_blocks.pycovers the block mapping on synthetictensors (aligned tiles, partial tiles on both axes, a missing companion, a gross
size mismatch) and cross-checks one real tensor bit-exactly against an
index-array dequant; it also documents the case that cannot be caught, a
compatible-but-wrong tile size, which is why the value is read from config.
The
tests/test_convert_resume.pychange is in the first commit rather than itsown: that file stubs
mxfp4with atypes.ModuleType, so the new import has toresolve there or every check in the file fails at module load, and the commit
would not be green on its own.
Verified end to end by converting
Kimi-K2-Instruct— 61 layers, 384 expertstop-8, VQ3R, 354 GB expert set, 6.9 GB trunk — and opening it with
waste info:1.03 T total, 31.69 B active per token.
Nothing outside
tools/and that one test file changes.docs/LEARNED.md,CHANGELOG.mdandWASTE_VERSION_*are deliberately untouched.