fix(serving): serialize SSE generator + FlashInfer plan dtype (bf16) - #189
Open
drunkcoding wants to merge 1 commit into
Open
fix(serving): serialize SSE generator + FlashInfer plan dtype (bf16)#189drunkcoding wants to merge 1 commit into
drunkcoding wants to merge 1 commit into
Conversation
… loop crash - api_server_v2: persist a single shielded next(stream) future across the 0.1s disconnect-poll so concurrent next() no longer raises 'generator already executing'; log + set UNHEALTHY when the engine loop dies instead of hanging - attention_backend: pass q_data_type/kv_data_type (bf16) to FlashInfer prefill and decode plans (was float16-default -> dtype mismatch killed the loop) - kv_cache.set_block_store: normalize cuda device index in geometry check
5 tasks
Contributor
Author
|
Recommended first merge. Of the productization PRs (#187, #188, #189, draft #190), this one is the lowest-risk and most broadly beneficial:
Suggest landing this first (or cherry-picking the two commits to main), then the feature-specific PRs on top. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
General serving-infra fixes surfaced while productizing chunked-prefill (see docs/superpowers/reports/2026-08-30-productization-verdicts.md). These are independent of the chunked feature itself.
next(stream)future across the 0.1s disconnect-poll so concurrentnext()no longer raisesValueError: generator already executing(this hung all streamed requests). Also log + set UNHEALTHY when the engine loop dies instead of hanging silently.q_data_type/kv_data_type(bf16) to the FlashInfer prefill+decode plans — the float16 default causeddtype of q bfloat16 does not match q_data_type float16, which silently killed the async engine loop on Qwen3.Verified: whole-prefill now generates ('The capital of France is' -> ' Paris...'). NOTE: chunked-prefill itself remains blocked by a separate KV block-budget sizing issue (logical 10940 > physical 9725) documented in the report — not addressed here.