serving: opt-in KV-quant + flash attention — the unique half of #2053, rescued onto canary - #2244
Merged
Merged
Conversation
This was referenced Aug 11, 2026
Merged
Merged
…he elastic window (#232) Adds SERVING_KV_CACHE_TYPE (config, default f16/off): when set to q8_0 (or q4_0) the llama-server lane runs --cache-type-k/v <type>, cutting resident KV ~in half at near-lossless quality. That frees memory the elastic window (#234) can spend on a bigger context or more warm lanes — faster for multiple personas AND more room for hard coding, the same "faster + best code" pair. OFF by default and safe-by-construction: absent / f16 → byte-identical f16 launch (no behavior change), so this can't destabilize a backend whose build lacks Metal KV-quant kernels — enabling it is an explicit operator opt-in, never a blind assumption ([[verify-real-device-numbers-not-a-clamp-premise]]). Follow-up (noted in code): to have the PLAN grow the window on the freed memory rather than leave it as extra headroom, footprint_for must scale kv_per_token by the quant factor. This slice is the safe enablement; that fit-math coupling is the next step, and wants a live burst on a KV-quant-capable backend to validate quality + the speedup. continuum-core lib compiles clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…d KV memory (#232) Completes the KV-quant feature. The launcher flag (prior commit) makes a lane run q8_0 KV; this makes the PLAN know it: footprint_for scales kv_per_token by the quant divisor, so the served window is sized against the KV the lane WILL actually hold, and the elastic window (#234) grows into the freed memory instead of leaving it idle. - kv_divisor_for (pure, env-free, unit-tested): f16/unset/unknown → 1 (no change), q8_0 → 2, q4_0/q4_1 → 3. CONSERVATIVE by design — under the ideal ~3.5x for q4 — so the plan can never over-grow the window past the real KV and OOM (over-reserve = smaller window = safe). - Applied only in the config-aware footprint_for; footprint_from_parts stays pure so its tests are env-independent. Same SERVING_KV_CACHE_TYPE key as the launcher — one config, two consumers (flag + fit rate), documented to stay in sync. Default (f16 / unset) → divisor 1 → byte-identical: this can't change serving on a box that doesn't opt in. Test pins the mapping + the safe-default. Wants a live burst on a KV-quant-capable backend to confirm quality + the actual window growth. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…memory (#232) The fused attention kernel is faster on BOTH prefill and decode and lowers peak memory — directly attacking the prefill-bound turn latency (#139) and freeing room the elastic window (#234) can spend. SERVING_FLASH_ATTN=1|on|true adds --flash-attn to the lane; absent → llama.cpp default (no flag), byte-identical. OFF by default: Metal/backend flash-attn support + quality vary by build, so it's an operator opt-in, never a blind assumption ([[verify-real-device-numbers-not-a-clamp-premise]]). Composes with the KV-quant flag: enable both for the field-proven GLM-style speedup, then validate on a live burst. continuum-core lib compiles clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
joelteply
force-pushed
the
feat/kv-quant-flash-attn
branch
from
August 15, 2026 03:24
8f7aadf to
eac5800
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Salvage of #2053, which had become unmergeable for a reason worth stating: half of it was already in canary.
What was superseded
Canary now threads a
ServingDemandstruct (window_tokens, 61 references inserving_daemon). #2053's elastic-window commits pass two scalars (demand_lanes,demand_ceil) and predate that struct — so every rebase attempt collided on an API canary had already replaced with a better one. The feature landed; the branch's version of it is dead weight.What was NOT
flash_attn/kv_quant/cache_typereturn 0 matches in canary. This is real, unlanded work:0ea9538a1opt-in KV cache quantization — q8_0 halves KV, feeds the elastic window175cd6d51KV-quant fit coupling — the window GROWS into the freed KV memory10fed2798opt-in flash attention — faster prefill+decode, lower memoryCherry-picked onto canary, all three applied clean (no conflicts), which is itself evidence they were only ever blocked by the superseded commits sitting in front of them.
Verification
cargo check -p continuum-core --lib --testsclean.Why not just rebase #2053
Tried. It conflicts commit-by-commit on the replaced demand API, and resolving each one means hand-reconstructing a feature that is already live. The remaining #2053 commits (grid-overflow effector, expert_observe harness) are separate units and get the same treatment — verify against canary first, rescue what is unique.