feat: chunked prefill scheduling (bounded, decode-first, opt-in)

[drunkcoding]

Description

Implements bounded, opt-in chunked prefill scheduling for MoE-Infinity continuous serving. Active decode rows run every feasible scheduler step while long prefills make fair, bounded progress within the existing per-step token and batch limits. Disabled by default.

Motivation

Whole-prompt prefill admission can stall active decode rows and inflate tail latency. Bounded chunking lets decode keep priority while long prompts advance.

What landed (11 atomic commits)

• feat(serving): track partial prefill progress — SequenceData.advance_prefill, committed_kv_tokens, prefill-progress invariants.
• feat(serving): reserve paged KV incrementally — idempotent ensure_sequence_capacity, observable reserved-token count, rollback_sequence_reservation.
• fix(serving): reject chunk and prefix co-enablement — single validated startup guard (ValueError) across engine + _build_engine_config.
• fix(serving): unify paged KV storage ownership — canonical LayeredPagedKVStore (layer-aware standard + FlashInfer tensors) owning export/import/checkpoint/restore; PagedKVCache.set_block_store; swap routed through active storage.
• feat(serving): carry exact prefill chunk metadata — PrefillChunk, canonical PagedBatchLengths(query_lengths, query_offsets, context_lengths, kv_seq_lengths), exact chunk slicing, terminal flags.
• feat(serving): schedule bounded prefills behind decode — decode-first bounded scheduler, round-robin/age fairness, scheduler-owned in-flight leases, transactional commit/rollback.
• feat(serving): execute partial prefills without sampling — transaction-boundary step(), terminal-only sampling, progress-only steps, execution/sampling rollback, capability gate, DFlash mid-prompt guard, stats.
• fix(serving): preserve partial prefill lifecycle state — cancellation no-leak, partial-prefill swap recovery to PREFILL at the same offset (chunked mode only; legacy path unchanged).
• fix(serving): separate chunk query and KV metadata — FlashInfer qo_indptr = query_offsets, page metadata from kv_seq_lengths; real-Qwen3 detection; CUDA/FlashInfer canary tests.
• feat(serving): expose experimental chunked prefill — CLI/programmatic controls (--enable-chunked-prefill, --prefill-chunk-size, --prefill-starvation-threshold-steps), disabled-by-default, docs.
• bench(serving): measure chunked prefill TTFT and TPOT tails — paired disabled/enabled streaming benchmark + CPU aggregation tests + docs (neutral candidate-minus-baseline, no speedup claims).

Scope / limitations

• Standalone: rejects simultaneous prefix caching at startup; imports/duplicates no prefix implementation. PR feat: prefix KV reuse #181 is design input only.
• No P/D disaggregation; no changes to model partitioning, expert offload, or DFLASH deficit scheduling.
• Real-Qwen3 + real-FlashInfer CUDA gates require a FlashInfer-enabled GPU host; on CPU-only/FlashInfer-absent environments they SKIP (canary acceptance still requires them to PASS on the rollout host).

Verification (this environment: CUDA present, FlashInfer absent)

• Full CPU serving suite: 196 passed, 2 skipped (skips are CUDA/FlashInfer-only).
• ContextPilot scheduler/lifecycle/eviction + unit schedulers: passed.
• compileall exit 0; ruff check clean; LSP diagnostics 0 errors across moe_infinity/serving.
• Strict RED→GREEN per task; spec-then-quality discipline; every commit passes pre-commit hooks (ruff/ruff-format/codespell).

Type of Change

• Production implementation (opt-in, disabled by default)

Checklist

• Default-disabled compatibility covered
• Rollback and cancellation specified and tested
• Chunk/prefix co-enablement rejected at startup
• Query-vs-KV metadata separation

Post-review fixes

• fix(serving): make speculative and disabled paths transaction-safe — oracle spec review found two transaction_id-handling gaps: (1) unguarded rollback_prefill_step(None) masked real exceptions on the default/decode-only path; (2) DFlash delegation paths ignored the in-flight chunk transaction (stuck sequence on generate() failure). Both fixed with RED→GREEN tests; _completed_prefill_transactions bounded.

[drunkcoding]

Momus plan review: [OKAY] after 3 review rounds. The plan is now standalone and explicitly rejects prefix-cache co-enablement pending reconciliation; revisions are in commit a9b2392.

[drunkcoding]

Independent implementation verification:

• Focused tests in loader-isolated processes: 88 passed.
• Python LSP scan: 0 errors across 50 files.
• Branch clean and synchronized.
• A single combined pytest process exposes the repository test loaders replacing sys.modules with distinct SequenceStatus classes; each intended isolated test module passes independently.

PR remains draft for CUDA/FlashInfer review.

[drunkcoding]

Productization findings (full report: docs/superpowers/reports/2026-08-30-productization-verdicts.md). Infra fixes: #189.

The 'client hang' was three stacked defects. Two are general serving-infra bugs, fixed in #189:

1. SSE generator concurrency — wait_for(to_thread(next, stream), 0.1) abandoned the still-running thread on every disconnect-poll timeout, so the next iteration raised ValueError: generator already executing → every streamed request hung.
2. FlashInfer plan dtype — prefill/decode plans defaulted to float16 while Qwen3 runs bf16 (dtype of q bfloat16 does not match q_data_type float16), which silently killed the async engine loop (retained-global task; exception never surfaced).

After those, whole-prefill generates. But chunked-prefill itself is broken and appears never-run-end-to-end:
3. _schedule_chunked_prefill needs block_store bound for COW, but _bind_layered_paged_kv_store silently fails its capacity gate: the logical KV cache (10940 blocks) > physical paged store (9725) because the two are budgeted independently (memory_manager.get_max_kv_blocks vs memory_coordinator.compute_num_kv_blocks). Forcing the bind (resize logical→physical) then immediately hits a FlashInfer workspace overflow on chunk 2 (batch_prefill_tmp_s (1 MiB), 0 bytes available). Each fix uncovers the next.

Recommend: land #189 (infra) here; then complete chunked-prefill separately — align the two KV block budgets and size the FlashInfer workspace for chunked append-attention.