blog: Running Qwen3.8-Flash-Next on a DGX Spark and RTX PRO 6000 - #267
Merged
Conversation
Benchmarks the 176.94B MoE (6B active/token) on two machines. DGX Spark, llama.cpp built from the unmerged qwen4_exp PR #27742: 34.5 tok/s generation, 797/747/599 tok/s prefill at 2K/8K/32K, 72.5 of 121 GiB, wikitext-2 PPL 4.79. RTX PRO 6000 with vLLM FP8: TP1 OOMs, TP2 81.5 tok/s single stream, TP4 805 tok/s at 32 concurrent, TP4+MTP 87.9 tok/s single. TP2 beats TP4 by 26% single stream on a box with no NVLink. Documents that the official NVFP4 and FP8 builds only quantize routed experts, leaving the 51B N-gram table in BF16/FP8, and that the GGUF UD-IQ1_S is 3.28 bpw effective rather than 1.56.
Deploying website with
|
| Latest commit: |
eb13547
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://4b184ee4.website-dab.pages.dev |
| Branch Preview URL: | https://blog-qwen38-flash-next.website-dab.pages.dev |
Adds the llama-bench and llama-perplexity invocations, the vllm bench serve command, the TP4 variants, the NVFP4 single-card run, and the checkpoint download steps, so the post is reproducible without leaving the page. Also records the wikitext fetch gotcha (llama.cpp's get-wikitext-2.sh does not follow the S3 redirect) and the ModelScope fallback.
Graph capture completed at 06:34:14 and 'Application startup complete' had still not printed 20 minutes later. Records that the only log errors were benign transformers docstring warnings, so the hang is not masking a crash.
Perplexity was used without ever being explained, and the name now collides with a search company. Adds a one-breath definition and states plainly that it runs locally with no API. The TP3 claim was wrong. I said it failed because of the 2 KV heads; testing it shows 'AssertionError: 16 is not divisible by 3', where 16 is linear_num_key_heads on the GDN layers. Conclusion held, reasoning did not.
Daniel Han wrote all 33 commits of llama.cpp PR #27742 and published the dynamic quant. Both halves of the Spark result depend on his work, so it gets a named thanks rather than a passing link. Also credits the llama.cpp maintainers and RadixArk for the NVFP4 conversion.
The Ollama section warned about something never tested on either box, so it was a tangent. Removed, along with its line in 'What I did not measure'. 'What it costs you' keeps every measurement but loses the meta-narration about almost publishing a weaker claim. Network diagnostics in Part 2 cut to the actionable sentence. 3780 -> 3560 words.
Someone asked what the MTP acceptance rate was and it was not in the post. Measured it, and the answer changes the section. vLLM spec-decode counters, TP4 FP8, k=3: synthetic random 1024/512 71.3% acceptance, 3.14/4 synthetic random 512/256 84.4% acceptance, 3.53/4 real code+prose, greedy 55.3% acceptance, 2.66/4 Real prompts accept worse than random ones. Random input makes the model emit repetitive low-entropy text that a draft head predicts easily, so --dataset-name random flatters speculative decoding. Re-measured throughput on five genuine prompts, greedy, single stream: MTP off 49.90 tok/s -> MTP on 124.93 tok/s = 2.5x which is better than the 36% the synthetic benchmark implied. Also adds an output-identity section: MTP on and off produce byte-identical output over 5 runs (3575aff8aa9c1df5), so spec decode is lossless here. New cover image, and the offload KV figure is now the measured 48.74 GiB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmarks Qwen3.8-Flash-Next (176.94B params, 6B active per token) on a single DGX Spark and on RTX PRO 6000 Blackwell cards.
Why this post
The claim going round was that the model needs two DGX Sparks. That is true for the NVFP4 build, and the post confirms it. But one build does fit on a single Spark, and the reason turned out to be the interesting part: the official FP8 and NVFP4 releases only quantize the routed experts, leaving the 51B N-gram embedding table in BF16/FP8. The Unsloth GGUF is the only build that compresses that table too.
Headline results (all measured on my own hardware)
DGX Spark, llama.cpp built from the unmerged
qwen4_expPR ggml-org/llama.cpp#27742:RTX PRO 6000 with vLLM and the FP8 checkpoint:
Two findings worth the post
SYS. With only 6B active parameters there is almost no compute to split, so extra cards add all-reduce traffic rather than speed.On claims
Everything quoted is measured on my Spark and the RTX box, and the post carries a "What I did not measure" section listing the gaps: no Ollama numbers, no high-precision perplexity baseline (the 4.01 figure is the PR author's), no QSA ablation, no NVFP4 throughput, no Spark concurrency sweep, no BF16 run.
Files
content/blog/running-qwen3-8-flash-next-on-dgx-spark-and-rtx-pro-6000.mdpublic/img/blog/running-qwen3-8-flash-next-on-dgx-spark-and-rtx-pro-6000/cover.png(1200x630)public/img/blog/running-qwen3-8-flash-next-on-dgx-spark-and-rtx-pro-6000/benchmarks-both-machines.png