Skip to content

fix(serving): serialize SSE generator + FlashInfer plan dtype (bf16) - #189

Open
drunkcoding wants to merge 1 commit into
plan/chunked-prefill-schedulingfrom
fix/serving-streaming-and-flashinfer-dtype
Open

fix(serving): serialize SSE generator + FlashInfer plan dtype (bf16)#189
drunkcoding wants to merge 1 commit into
plan/chunked-prefill-schedulingfrom
fix/serving-streaming-and-flashinfer-dtype

Conversation

@drunkcoding

Copy link
Copy Markdown
Contributor

General serving-infra fixes surfaced while productizing chunked-prefill (see docs/superpowers/reports/2026-08-30-productization-verdicts.md). These are independent of the chunked feature itself.

  • api_server_v2: persist a single shielded next(stream) future across the 0.1s disconnect-poll so concurrent next() no longer raises ValueError: generator already executing (this hung all streamed requests). Also log + set UNHEALTHY when the engine loop dies instead of hanging silently.
  • attention_backend: pass q_data_type/kv_data_type (bf16) to the FlashInfer prefill+decode plans — the float16 default caused dtype of q bfloat16 does not match q_data_type float16, which silently killed the async engine loop on Qwen3.
  • kv_cache.set_block_store: normalize cuda device index in the geometry check.

Verified: whole-prefill now generates ('The capital of France is' -> ' Paris...'). NOTE: chunked-prefill itself remains blocked by a separate KV block-budget sizing issue (logical 10940 > physical 9725) documented in the report — not addressed here.

… loop crash

- api_server_v2: persist a single shielded next(stream) future across the 0.1s
  disconnect-poll so concurrent next() no longer raises 'generator already
  executing'; log + set UNHEALTHY when the engine loop dies instead of hanging
- attention_backend: pass q_data_type/kv_data_type (bf16) to FlashInfer prefill
  and decode plans (was float16-default -> dtype mismatch killed the loop)
- kv_cache.set_block_store: normalize cuda device index in geometry check
@drunkcoding

Copy link
Copy Markdown
Contributor Author

Recommended first merge. Of the productization PRs (#187, #188, #189, draft #190), this one is the lowest-risk and most broadly beneficial:

  • Both fixes are general serving-infra bugs, independent of chunked-prefill — they fix the OpenAI server's streamed-completion path and the FlashInfer plan dtype for any bf16 model on FlashInfer, not just this feature branch.
  • Concretely: (1) eliminates ValueError: generator already executing that hung every streamed request, and (2) fixes the float16-vs-bf16 plan mismatch that silently killed the async engine loop. Also surfaces engine-loop crashes as UNHEALTHY instead of hanging.
  • No API/behavior change beyond making streaming + bf16 FlashInfer actually work; verified whole-prefill generates end-to-end.

Suggest landing this first (or cherry-picking the two commits to main), then the feature-specific PRs on top.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant