feat: opt-in INT8 KV-cache quantization

[drunkcoding]

Description

Implements the Momus-approved plan (docs/superpowers/plans/2026-08-21-kv-cache-quantization.md) for an opt-in, correctness-gated int8_sym paged KV-cache format behind a format abstraction, preserving existing attention execution dtypes and deterministic fallback to the native/SDPA paths.

What landed

• Format contract (moe_infinity/runtime/kv_cache_format.py): KVCacheFormat, resolve_kv_cache_format capability resolver, tokenwise symmetric INT8 quantizer with upward-rounded FP16 scales (no -128, exact zero, element error <= stored_scale/2 + 1e-6), LayeredPagedKVStore/LayeredKVPageChunk, and model_info_from_config MLA/GQA detection.
• Runtime integration (attention_backend.py, paged_attention_ops.py, qwen3_paged_attention.py): store-bound PagedAttentionBackend with bind_store/write_chunk, INT8 decode via native CUDA dispatch or FP32 dequantized SDPA fallback, probe_native_int8_binding capability check, layer_idx threading.
• Native INT8 CUDA kernel (extensions/kernel/paged_attention_int8.{cuh,cu}, paged_attention.cu, setup.py): reusable INT8+FP16-scale paged decode kernel with FP32 accumulation, registered in the single existing PYBIND11_MODULE, linked via _PAGED_ATTN_SOURCES, plus a dedicated tests/cuda target. Verified MHA/GQA parity vs an independent FP32 dequantized SDPA oracle (max abs diff ~1e-4, cosine 1.0).
• Serving lifecycle (kv_cache.py, model_runner.py, scheduler.py, kv_cache_offload_coordinator.py): single-owner LayeredPagedKVStore, RESIDENT/SWAPPED residency state machine (swap_out(release_gpu_blocks=True), transactional swap-in, page-chunk truncation that slices INT8 scales, free_gpu_blocks rejects SWAPPED), single-call scheduler preemption, ModelRunner store-identity validation, coordinator.set_kv_store. Additive design preserves the legacy constructor and every existing test.
• Config + memory + engine handoff (config.py, memory_manager.py, api_server_v2.py, engine.py): kv_cache_format/kv_cache_allow_fallback fields, descriptor-driven get_max_kv_blocks (native unchanged, int8 counts FP16 scales), --kv-cache-format/--no-kv-cache-format-fallback CLI, and end-to-end consumption in ContinuousBatchingEngine — it resolves the decision from the model config + capabilities, allocates the effective-format store, builds a store-backed PagedKVCache, and exposes requested/effective_kv_cache_format, kv_cache_execution_backend, and kv_cache_format_decision_reason via engine stats and /v1/config, so a native fallback (e.g. MLA) is never reported as quantized.
• Quality gates: deterministic tensor-level INT8-vs-native decode equivalence (fp16/bf16, MHA/GQA, page boundaries, zero and non-contiguous cases), a model-free 2048-step logit/token-agreement parity gate, exact INT8 storage byte/ratio assertions, and an env-gated WikiText-2 perplexity release gate.
• Benchmark + docs: benchmarks/serving/kv_cache_quantization.py A/B matrix with separate storage/transfer/execution precision reporting and a strict-fallback guard; docs for the precision contract, memory formula, MLA/FlashInfer fallback, one-setting rollback, and effective-format stats.

Precision & memory

int8_sym stores INT8 payload + one FP16 scale per (layer, page, KV head, token); for the canonical (block_size=16, kv_heads=8, head_dim=128) page this is 33,280 bytes vs 65,536 native FP16 (0.5078125). Storage/transfer/execution precisions are reported separately so a fallback is never mistaken for a quantized run.

Scope & fallback

MHA/GQA validated; MLA visibly falls back (mla_not_validated) or errors under strict mode. FlashInfer stays active for native stores; an int8_sym request bypasses FlashInfer (flashinfer_no_int8_sym_contract) into the built-in validated path. No universal low-bit or 2-bit claim is made; KIVI/KVQuant are motivation only.

Verification

• Full KV suite: 132 passed, 1 skipped (FlashInfer unavailable) with the in-tree native binding; ruff clean on all changed files.
• Native INT8 kernel built (arch sm_120) and MHA/GQA parity verified vs FP32 oracle.
• End-to-end: an engine built with --kv-cache-format int8_sym allocates an INT8 store and reports effective_kv_cache_format=int8_sym; an MLA model reports effective_kv_cache_format=native, reason=mla_not_validated.

Environment blockers (recorded, not worked around)

• compute-sanitizer is not installed in this environment, so the sanitizer memcheck steps could not be executed.
• tests/cuda CMake harness cannot configure here (its supported-Python gate is 3.8–3.11 vs installed 3.13, and cmake 3.28 + CUDA 13.1 fail Torch's enable_language(CUDA) toolkit-root detection). The kernel itself compiles via nvcc (JIT + setuptools); the added CMake target is syntactically valid.
• The pinned Qwen3-30B-A3B checkpoint and WikiText-2 snapshot are not present, so the perplexity/model-parity gates and the full long-context benchmark matrix skip; the model-free logit/token and tensor quality gates run and pass.

Type of Change

• Production implementation (opt-in, native default)
• Tests + docs

Checklist

• Native default and one-setting rollback retained
• Storage/transfer/execution precision separately visible
• Quality gates specified and (non-model) executed
• No universal low-bit/2-bit claim

[drunkcoding]

Momus plan review: [OKAY] on the first review round.

[drunkcoding]

Independent implementation verification:

• Focused KV format/capability/handler/storage suites: 49 passed.
• Python LSP scan: 0 errors across 50 files.
• Branch clean and synchronized.

PR remains draft for native CUDA and model-quality qualification.