Add opt-in fused Qwen3.5 inference kernels

[BruceSheng1202]

Summary

Opt-in fast path for Qwen3.5-architecture backbones (model_type qwen3_5, including the Qwen3.8 bases): LoadOptions(fused_kernels=True), JEVANY_FUSED_KERNELS=1, or --fused-kernels on jevany serve, jevany decide and scripts/benchmark_latency.py. Off by default. fuse_qwen3_5 rewrites the merged, eval-mode backbone in place:

• each zero-centred RMSNorm runs as one FLA kernel;
• MLP gate/up run as one GEMM plus FLA's fused SwiGLU;
• each Gated DeltaNet layer runs its four input projections as one GEMM; FLA's chunked kernel applies the gate (A_log, dt_bias), beta sigmoid, q/k L2 norm and grouped value heads itself; FLA's fused gated RMSNorm follows;
• each attention layer runs q (with its gate), k and v as one GEMM;
• decoder weights are stored transposed, so cuBLAS reads both GEMM operands in their natural layout.

The original nn.Linear modules keep views of the fused matrices, so memory does not grow (peak 9.93 GiB with and without). The DeltaNet path runs only without a cache, mask or packed-sequence metadata, attention only without a cache; other calls and training use the original forwards. Loading requires CUDA, a merged LoRA (JEVANY_MERGE_BF16=1 for BF16 checkpoints), one device and flash-linear-attention >= 0.5, and rejects compile_mode, device_map and choice readout. describe() reports what was fused.

Measurements

A100-SXM4-40GB, one node: JevAny-Qwen3.5-4B-LoRA, BF16 merged LoRA, SDPA, FLA 0.5.2, causal-conv1d 1.7.0, torch 2.8.0, transformers 5.17.0. python -m scripts.benchmark_latency --records 0 --repeats 3 --serving-kernels --cuda-graphs [--fused-kernels], public JevBench (231 requests x 3), forward ms:

	median	mean	p90	p95	> 2,048 tokens
main	16.43	41.79	130.14	140.91	134.6 (eager)
this PR	11.25	34.10	112.30	123.94	117.8 (eager)
+ perf/cuda-graph-flash-attention, 4096 limit	11.23	32.69	102.37	125.45	102.7 (graph)

Quality with graphs and serving kernels (jevany.benchmark): Transfer-v9 dev 823 -> 824 / 1,046 correct, NLL 0.5869 -> 0.5875, ECE 0.035 -> 0.029; JevBench 180 -> 180 / 231; 5 of 1,264 Transfer and 0 of 231 JevBench argmax decisions changed; largest probability change 0.044. On a random 4-layer Qwen3.5 the fused backbone matches transformers to 1.2e-5 relative error in fp32 and 1.2e-2 in BF16. Not measured: H200 and 27B.

Tests

• CPU: options and validation; fused and transposed weights keep nn.Linear outputs and share storage; backbone, device, CLI and choice-readout checks (transformers 5.17 and 5.18).
• CUDA + FLA: fused vs transformers backbone (fp32, BF16); checkpoint load with graphs and fused kernels matches the unfused path.
• pytest tests -m "not server" on the A100 node: 31 failures vs 30 on main; the extra test_moe_public_trainer_and_checkpoint[bf16-glm] also fails on main when rerun alone there.

Not in this PR: other ideas from "Running VLAs at Real-time Speed" (arXiv 2510.26742)

With both branches, a 128-token replay is ~10.8 ms of GPU time: 199 GEMMs 7.3 ms (1.6x the 4.6 ms weight-read floor), ~420 elementwise/copy kernels 1.5 ms, FLA 1.3 ms, norms 0.4 ms.

Paper technique	Here	Expected value
Small-M Triton GEMMs, tuned tiles (4.1)	no	largest: est. 1-2 ms on short rows
Activation, residual, norm scale in GEMM epilogues (4.2, 4.4)	partly	est. ~1 ms; needs the same custom GEMMs
RoPE precomputed / fused (3.2c)	no	small; positions are fixed per bucket
RMSNorm affine folded into next linear (3.2a)	no	~0 alone; only with epilogue fusion
Skip last layer (4.1)	analogue: last layer only on readout rows	exact; ~3% on long rows
Partial split-K (4.3)	no	paper: < 0.1 ms
Overlapped streams (6)	n/a	analogue: batched graphs (throughput)
Megakernel / software barrier (5)	no	paper: net loss

Measured but left out: readout head and softmax in the graph with one pinned copy each way (-1.5%); residual add fused into the post-attention norm (-0.5%).

🤖 Generated with Claude Code

https://claude.ai/code/session_01EpJAgfodeE1m4Fk2tDiLfB

[github-actions]

Human Review

• Applicable CI failed or cannot be verified for this PR revision; see the CI evidence below.

Human Review: see the findings and review evidence.

AI semantic review has not completed.

• CI ci.yml: incomplete (run) — CI cannot be bound to the current PR/base. Update the branch from main and run CI again.
• CI pages.yml: incomplete (run) — CI cannot be bound to the current PR/base. Update the branch from main and run CI again.

Reviewed head 00e0135a2fd4b19f5b3fceb50df3898240af22b6; base 2b9ed4c20f0e3d86db28bf7ddc1c11ac80c6dfe0.
CPU CI does not verify production-model quality, CUDA or multi-GPU performance.

[github-actions]

Human-Review. See the maintained PR review summary for evidence.

[github-actions]

Human-Review. See the maintained PR review summary for evidence.