fix(serving): prefix-cache binding, hit stats, dtype (needs #197) - #190
Merged
drunkcoding merged 3 commits intoSep 4, 2026
Merged
Conversation
…0 eager - kv_cache.set_block_store: normalize cuda device index so 'cuda' vs 'cuda:0' geometry no longer rejects the store (prefix cache now binds/activates) - prefix_cache/engine: surface prefix_cache_hits_total + matched_tokens_total - attention_backend: pass q_data_type/kv_data_type to FlashInfer plans - big_modeling: use eager attention for qwen3 when flash-attn unusable (sm120) - prefix_cache_benchmark: one model per subprocess (avoids multi-model crash) - engine: include bind exception detail in disabled reason
This was referenced Aug 31, 2026
…ness - benchmarks/prefix_cache_benchmark: add --parity-report (per-mode tensor dumps + numerical analysis instead of digest-only failure), and --device-memory-ratio/--kv-cache-ratio so the suite fits shared GPUs. - tests/integration/test_flashinfer_kernel_parity: prove full-prefill and append kernel schedules are each deterministic, within ULP-scale error of an fp32 reference, and mask partial-page tails (3/3 pass on sm120). Cross-kernel bitwise equality is not an invariant. - tests/serving/test_prefix_cache_cuda: run warm-vs-cold parity through one-engine-per-subprocess (kTopologyHandle-safe) with correct invariants: disabled==cold bitwise control, geometry-proven reuse, tolerance + near-tie-aware token comparison.
Warm-vs-cold token equality is unattainable: the paths use different FlashInfer kernel schedules (ULP-level per-op deltas) and expert-output accumulation order is nondeterministic, so near-tie argmax swaps occur even between same-mode reruns. Keep disabled==cold bitwise as the strict store-path regression check; at a flip require the two runs to share the same top-2 candidate pair, which still catches real corruption.
drunkcoding
marked this pull request as ready for review
September 4, 2026 08:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Productizing prefix-kv-reuse. Depends on #197 (
fix/expert-dispatch-stream-ordering→dev): without it, prefix-enabled runs are corrupted by an expert-dispatch stream race.Fixes in this PR
cudavscuda:0geometry no longer rejects the store — the prefix cache now actually binds/activates.prefix_cache_hits_total/prefix_cache_matched_tokens_total.q_data_type/kv_data_typeto the FlashInfer plans.qwen3when flash-attn is unusable (sm120).--parity-report(tensor dumps + numerical analysis),--device-memory-ratio/--kv-cache-ratiofor shared GPUs.HOLD resolved — what "warm not bit-exact on sm120" actually was
dev, exposed by prefix-mode timing) — found via NaN-poisoning:enabled_coldproduced 100%-NaN logits fromlayers[0].mlpwith finite inputs. Fixed in fix(dispatcher): stream-order expert exec against input producer and consumers #197.disabled == enabled_coldbitwise (token digeste37a9d14, logits Δ = 0.0).q=8, ctx=64, kv=72, hit counted). Warm step-0 argmax swaps within the shared top-2 pair (cold:2157@22.25 / 85355@21.5; warm:85355@22.125 / 2157@21.5) — near-tie amplification of ULP-level kernel deltas through 48 MoE layers.final_hidden_states_.add_/index_add_) is completion-order nondeterministic (pre-existing, mode-independent). The historical warm digest3739c3b3reproduces exactly this way.Merge order
dev← #197 first, then #181 (rebased on dev), then this PR into #181.