diff --git a/README.md b/README.md index 5208d010..9dc4ae5b 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,8 @@ per-request Context limit. | GLM-5.3 Flash NVFP4-Spark · native MTP3, switched | [Shared SparkRing source image](runtime/sparkring/source_image/README.md) | Operator-selected NCCL links | TP4/DCP1 | 1M | 16 | — | [Switched quickstart](docs/GLM53_SWITCHED_TP4_QUICKSTART.md) | | GLM-5.2 EXL3 3.5-bpw | [SparkRing vLLM/ExLlamaV3 build](runtime/exl3-r7/README.md) | [SIRCL + NCCL](docs/SIRCL.md) | TP4/DCP4 | 1M | 16 | ~1.2M | [Quickstart](docs/GLM52_35BPW_QUICKSTART.md) | | DeepSeek-V4-Flash-0731 | [SparkRing vLLM/B12X image](runtime/deepseek0731-gb10/README.md) | [Patched NCCL](spark_transport/nccl/README.md) | TP4/DCP1 | 1M | 32 | ~1M | [Quickstart](docs/DEEPSEEK_V4_FLASH_QUICKSTART.md) | -| DeepSeek-V4.1-Flash · Engram on NVMe · DSpark k=5 | [Self-built stock vLLM `dsv41-feat` image](runtime/deepseek-v41-gb10/README.md) | [Patched NCCL](spark_transport/nccl/README.md) | TP4/DCP1 | 430,080 | 8 | ~2.18M | [Quickstart](docs/DEEPSEEK_V41_FLASH_QUICKSTART.md) | +| DeepSeek-V4.1-Flash · SGLang decoder-tail replay | [Source-pinned SGLang image](runtime/deepseek-v41-sglang/README.md) | [Patched NCCL](spark_transport/nccl/README.md), in-place library mount | TP4/EP4 | 262,144 | 8 | ~1.50M | [Operator guide](runtime/deepseek-v41-sglang/README.md) | +| DeepSeek-V4.1-Flash · vLLM fallback · DSpark k=5 | [Self-built stock vLLM `dsv41-feat` image](runtime/deepseek-v41-gb10/README.md) | [Patched NCCL](spark_transport/nccl/README.md) | TP4/DCP1 | 430,080 | 8 | ~2.18M | [Quickstart](docs/DEEPSEEK_V41_FLASH_QUICKSTART.md) | | Qwen3.8-27B EXL3 K5/K6 | [SparkRing vLLM/ExLlamaV3 build](runtime/qwen38/README.md) | [Patched NCCL](spark_transport/nccl/README.md) | TP4/DCP1 | 1M | 64 | — | [Quickstart](docs/QWEN38_27B_EXL3_K5K6_QUICKSTART.md) | | DeepSeek-V4-Flash-Vision-Exp with DSpark (research-only) | [Anemll image / MiaAI-Lab recipe](runtime/deepseek-vision-exp/profile.json) | [SparkRing patched NCCL](spark_transport/nccl/README.md) | TP4 | 1M | 48 | — | [Quickstart](docs/DEEPSEEK_V4_FLASH_VISION_EXP_TP4_QUICKSTART.md) | diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index c9e25320..d2ca52f8 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -418,3 +418,15 @@ with its pinned CUTLASS, CCCL and spdlog submodules. No FlashInfer source is inc `deepseek-ai/DeepSeek-V4.1-Flash` is distributed by DeepSeek under the MIT License. No weights or model files are included; operators fetch the checkpoint themselves. + +### 16. Mia DeepSeek V4.1 SGLang adapter (external build input) + +`runtime/deepseek-v41-sglang/build-image.sh` clones +[MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks](https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks) +at commit `e59e6eb67479aa68f6fa700c600dc90a0729b5ec` (AGPL-3.0). The adapter, +`boot.py`, row store, and Engram packer remain in that external repository and +the operator-built image; none of those files are vendored here. The external +Dockerfile also carries SGLang's Apache-2.0 FlashMLA wrapper. SparkRing's auth +patch targets the Apache-2.0 SGLang auth module in the pinned base image and +preserves its surrounding source. Operators retain the external components' +licenses and notices with their builds. diff --git a/docs/DEEPSEEK_V41_FLASH_QUICKSTART.md b/docs/DEEPSEEK_V41_FLASH_QUICKSTART.md index 6eec8595..7e1d9c3d 100644 --- a/docs/DEEPSEEK_V41_FLASH_QUICKSTART.md +++ b/docs/DEEPSEEK_V41_FLASH_QUICKSTART.md @@ -1,5 +1,10 @@ # DeepSeek-V4.1-Flash four-Spark cycle quickstart +The [SGLang cycle runtime](../runtime/deepseek-v41-sglang/README.md) adds +decoder-tail prefill and file-backed multi-key authentication. Its broader +deployment qualification is still in progress. This page documents the vLLM +fallback and its existing measured settings. + Serve `deepseek-ai/DeepSeek-V4.1-Flash` (the stock checkpoint) as four tensor-parallel ranks on a directly cabled four-Spark cycle, with the model's two Engram lookup tables left on each rank's NVMe. diff --git a/docs/profiles/DEEPSEEK_V41_FLASH.md b/docs/profiles/DEEPSEEK_V41_FLASH.md index c4c3be04..37df1020 100644 --- a/docs/profiles/DEEPSEEK_V41_FLASH.md +++ b/docs/profiles/DEEPSEEK_V41_FLASH.md @@ -1,5 +1,9 @@ # DeepSeek-V4.1-Flash four-Spark cycle profile +See the [SGLang cycle runtime](../../runtime/deepseek-v41-sglang/README.md) and +[controlled decoder-replay comparison](../../performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911.md) +for the SGLang profile. The measurements below belong to the vLLM fallback. + ## Status **Implemented and live-benchmarked on one private four-Spark cycle; not qualified.** The image diff --git a/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911.md b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911.md new file mode 100644 index 00000000..ad75b71f --- /dev/null +++ b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911.md @@ -0,0 +1,89 @@ +# SGLang decoder-tail replay on a four-Spark cycle + +Status: **implemented**. This controlled comparison attributes the measured +prefill improvement to decoder-tail replay. It does not qualify a larger +context setting or unattended production deployment. + +## Conditions + +Four GB10 DGX Sparks, directly cabled as a four-node cycle, TP4/EP4, local NVMe +checkpoint `deepseek-ai/DeepSeek-V4.1-Flash` at +`dba1be0a40aa45a94ad051997016db3960a90277`, SparkRing patched NCCL 2.30.7 mounted +in place over SGLang's pip NCCL. Both arms used the same locally built image, +Mia commit `e59e6eb67479aa68f6fa700c600dc90a0729b5ec`, image ID +`sha256:4252984d1cd642a51bd0cde06665a4f6a73726141dec760392a73cf968bce744`. +The source/base identity is recorded in the [runtime pins](../../../runtime/deepseek-v41-sglang/pins.json). + +The four ranks were rebooted before each arm. Context 262144, prefill chunk +4096, maximum eight requests, memory fraction 0.90, requested total-token pool +1500000, DSpark block five without SPS/STS tables, b12x MXFP8 backend, MoE fused +finalize disabled, expandable segments disabled, Engram NVMe packed layout, +96 I/O threads and zero GiB row cache. Only the decoder replay flag differed. + +## Measurement + +Client wall-clock prompt throughput is server-reported prompt tokens divided +by request wall time for a one-token response. Target prompt sizes are labels; +actual token counts are recorded in the raw samples. Decode uses fixed-length +256-token generation at C1 and C4. The prompt distribution and decode timing +are separate from the multi-category vLLM benchmark; do not compare their +absolute rates as though the workloads were identical. + +There was one observation per size/concurrency in each arm. A warm-up request +preceded the ladder, but the first nominal 16K prefill still included additional +initialization and is retained without being represented as steady-state +performance. There is no repeated-run confidence interval. Quality capture +used 20 fixed prompts, including four roughly 12–15K prompts, first-token top-20 +log probabilities, and 48-token greedy continuations. Truncated-distribution +KL is a diagnostic, not a full-vocabulary divergence measure. + +Raw observations: [replay-off prefill](sglang-decoder-replay-20260911/prefill-sgl-replay0.json), +[replay-on prefill](sglang-decoder-replay-20260911/prefill-sgl-replay1.json), +[replay-off decode](sglang-decoder-replay-20260911/decode-sgl-replay0.json), +[replay-on decode](sglang-decoder-replay-20260911/decode-sgl-replay1.json), +[off quality](sglang-decoder-replay-20260911/sgl-replay0.json), +[on quality](sglang-decoder-replay-20260911/sgl-replay1.json), +[off needle](sglang-decoder-replay-20260911/needle-sgl-replay0.json), and +[on needle](sglang-decoder-replay-20260911/needle-sgl-replay1.json). +Endpoint addresses are sanitized; numerical observations are retained. + +## Result + +| Measurement | Replay off | Replay on | +|---|---:|---:| +| Nominal 64K prefill, tok/s | 2006 | 3144 | +| Roughly 131K needle prefill, tok/s | 1861 | 2934 | +| Needle TTFT, seconds | 70.0 | 44.4 | +| C1 decode aggregate, tok/s | 35.4 | 37.4 | +| C4 decode aggregate, tok/s | 80.5 | 76.3 | + +Both needle requests returned the exact passphrase. All 20 first-token choices +and all 20 greedy continuations matched. Top-20 diagnostic KL mean was 0.006, +maximum 0.12. These are the handoff's rounded summaries; raw quality distributions +are retained for independent recomputation. + +## Conclusion + +Enabling decoder-tail replay increased nominal 64K prefill throughput by about +57% in this comparison. With replay disabled, SGLang prefill was near the +separately measured vLLM implementation's approximately 2K tok/s. The matched +SGLang comparison isolates the replay flag; it does not attribute this gain to +two-batch overlap or generic engine scheduling. + +The optimization retains all rows through layer 20, then executes layers 21–39 +on the final 128 extend tokens per request. For a full 4096-token chunk, the +layer-token row count falls from 163840 to 88448, approximately 54% of the full +path. This is a work-count illustration, not a FLOP or measured communication +ratio. Smaller late-layer tensors reduce computation and collective payload. + +## Limitations + +The shortcut retains global KV/indexer memory but changes late-layer local SWA +visibility. A finite matching prompt set does not prove mathematical equivalence +to full prefill. Prompt-token log probabilities and arbitrary full-prompt hidden +state capture are not supported by the shortcut. The A/B does not cover mixed +batches, exact tail/chunk-boundary placement, sampled decoding, vision/tools, +400K retrieval, or long soaks. Those gates belong to the ongoing deployment +qualification and are not implied by this result. No communication profile was +captured in this experiment. Absolute results apply to this image, hardware, +transport, and workload only. diff --git a/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/decode-sgl-replay0.json b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/decode-sgl-replay0.json new file mode 100644 index 00000000..f0affe76 --- /dev/null +++ b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/decode-sgl-replay0.json @@ -0,0 +1,54 @@ +{ + "api": "http://rank0:8000", + "model": "deepseek-v4.1-flash", + "mode": "decode", + "label": "sgl-replay0", + "gen": 256, + "thinking": false, + "started": "2026-09-11T18:49:19Z", + "samples": [ + { + "c": 1, + "wall": 7.234400033950806, + "completion_tokens": 256, + "prompt_tokens": 248, + "agg_tok_s": 35.386486619291205, + "per_session_tok_s": 36.88715282912961, + "ttft_p50": 0.29383206367492676, + "ttft_max": 0.29383206367492676, + "steps_s": 0.0, + "acceptance": null, + "tok_per_step": null, + "preemptions": 0, + "errors": 0, + "finish": [ + "length" + ], + "err_samples": [], + "max_wall": 7.233917713165283, + "run": 0 + }, + { + "c": 4, + "wall": 12.71714186668396, + "completion_tokens": 1024, + "prompt_tokens": 1056, + "agg_tok_s": 80.5212374553003, + "per_session_tok_s": 21.858063466301363, + "ttft_p50": 0.6920806169509888, + "ttft_max": 0.6926991939544678, + "steps_s": 0.0, + "acceptance": null, + "tok_per_step": null, + "preemptions": 0, + "errors": 0, + "finish": [ + "length" + ], + "err_samples": [], + "max_wall": 12.71572470664978, + "run": 0 + } + ], + "finished": "2026-09-11T18:49:39Z" +} diff --git a/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/decode-sgl-replay1.json b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/decode-sgl-replay1.json new file mode 100644 index 00000000..ab85d0fc --- /dev/null +++ b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/decode-sgl-replay1.json @@ -0,0 +1,54 @@ +{ + "api": "http://rank0:8000", + "model": "deepseek-v4.1-flash", + "mode": "decode", + "label": "sgl-replay1", + "gen": 256, + "thinking": false, + "started": "2026-09-11T18:35:54Z", + "samples": [ + { + "c": 1, + "wall": 6.840591192245483, + "completion_tokens": 256, + "prompt_tokens": 263, + "agg_tok_s": 37.423665996910096, + "per_session_tok_s": 38.92745617985822, + "ttft_p50": 0.26373791694641113, + "ttft_max": 0.26373791694641113, + "steps_s": 0.0, + "acceptance": null, + "tok_per_step": null, + "preemptions": 0, + "errors": 0, + "finish": [ + "length" + ], + "err_samples": [], + "max_wall": 6.840073108673096, + "run": 0 + }, + { + "c": 4, + "wall": 13.420984506607056, + "completion_tokens": 1024, + "prompt_tokens": 1052, + "agg_tok_s": 76.29842650484338, + "per_session_tok_s": 21.759539600812793, + "ttft_p50": 1.1422911882400513, + "ttft_max": 1.1427288055419922, + "steps_s": 0.0, + "acceptance": null, + "tok_per_step": null, + "preemptions": 0, + "errors": 0, + "finish": [ + "length" + ], + "err_samples": [], + "max_wall": 13.420344352722168, + "run": 0 + } + ], + "finished": "2026-09-11T18:36:14Z" +} diff --git a/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/needle-sgl-replay0.json b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/needle-sgl-replay0.json new file mode 100644 index 00000000..0bf24a5a --- /dev/null +++ b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/needle-sgl-replay0.json @@ -0,0 +1,9 @@ +{ + "target": 131072, + "prompt_tokens": 130258, + "depth": 0.5, + "ttft_s": 70.0, + "prefill_tok_s": 1860.9, + "answer": "COPPER-LANTERN-8315", + "pass": true +} diff --git a/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/needle-sgl-replay1.json b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/needle-sgl-replay1.json new file mode 100644 index 00000000..e8d9689c --- /dev/null +++ b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/needle-sgl-replay1.json @@ -0,0 +1,9 @@ +{ + "target": 131072, + "prompt_tokens": 130258, + "depth": 0.5, + "ttft_s": 44.4, + "prefill_tok_s": 2933.5, + "answer": "COPPER-LANTERN-8315", + "pass": true +} diff --git a/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/prefill-sgl-replay0.json b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/prefill-sgl-replay0.json new file mode 100644 index 00000000..9ae5f705 --- /dev/null +++ b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/prefill-sgl-replay0.json @@ -0,0 +1,58 @@ +{ + "api": "http://rank0:8000", + "model": "deepseek-v4.1-flash", + "mode": "prefill", + "label": "sgl-replay0", + "gen": 256, + "thinking": false, + "started": "2026-09-11T18:48:37Z", + "samples": [ + { + "c": 1, + "wall": 12.965696096420288, + "completion_tokens": 1, + "prompt_tokens": 14763, + "agg_tok_s": 0.07712659563847798, + "per_session_tok_s": 1000000.0, + "ttft_p50": 12.964907169342041, + "ttft_max": 12.964907169342041, + "steps_s": 0.0, + "acceptance": null, + "tok_per_step": null, + "preemptions": 0, + "errors": 0, + "finish": [ + "length" + ], + "err_samples": [], + "max_wall": 12.964907169342041, + "run": 0, + "target": 16384, + "prefill_tok_s": 1138.6199314108505 + }, + { + "c": 1, + "wall": 29.41055202484131, + "completion_tokens": 1, + "prompt_tokens": 59007, + "agg_tok_s": 0.03400140191708611, + "per_session_tok_s": 1000000.0, + "ttft_p50": 29.40910029411316, + "ttft_max": 29.40910029411316, + "steps_s": 0.0, + "acceptance": null, + "tok_per_step": null, + "preemptions": 0, + "errors": 0, + "finish": [ + "length" + ], + "err_samples": [], + "max_wall": 29.40910029411316, + "run": 0, + "target": 65536, + "prefill_tok_s": 2006.3207229215 + } + ], + "finished": "2026-09-11T18:49:19Z" +} diff --git a/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/prefill-sgl-replay1.json b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/prefill-sgl-replay1.json new file mode 100644 index 00000000..95d34404 --- /dev/null +++ b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/prefill-sgl-replay1.json @@ -0,0 +1,58 @@ +{ + "api": "http://rank0:8000", + "model": "deepseek-v4.1-flash", + "mode": "prefill", + "label": "sgl-replay1", + "gen": 256, + "thinking": false, + "started": "2026-09-11T18:35:24Z", + "samples": [ + { + "c": 1, + "wall": 10.706095218658447, + "completion_tokens": 1, + "prompt_tokens": 14758, + "agg_tok_s": 0.09340473623447816, + "per_session_tok_s": 1000000.0, + "ttft_p50": 10.70548939704895, + "ttft_max": 10.70548939704895, + "steps_s": 0.0, + "acceptance": null, + "tok_per_step": null, + "preemptions": 0, + "errors": 0, + "finish": [ + "length" + ], + "err_samples": [], + "max_wall": 10.70548939704895, + "run": 0, + "target": 16384, + "prefill_tok_s": 1378.4670973484285 + }, + { + "c": 1, + "wall": 18.76260209083557, + "completion_tokens": 1, + "prompt_tokens": 58999, + "agg_tok_s": 0.053297511462359545, + "per_session_tok_s": 1000000.0, + "ttft_p50": 18.761067152023315, + "ttft_max": 18.761067152023315, + "steps_s": 0.0, + "acceptance": null, + "tok_per_step": null, + "preemptions": 0, + "errors": 0, + "finish": [ + "length" + ], + "err_samples": [], + "max_wall": 18.761067152023315, + "run": 0, + "target": 65536, + "prefill_tok_s": 3144.4998787677505 + } + ], + "finished": "2026-09-11T18:35:54Z" +} diff --git a/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/sgl-replay0.json b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/sgl-replay0.json new file mode 100644 index 00000000..2f1c83e5 --- /dev/null +++ b/performance/records/deepseek-v41-flash/sglang-decoder-replay-20260911/sgl-replay0.json @@ -0,0 +1,587 @@ +{ + "api": "http://rank0:8000", + "model": "deepseek-v4.1-flash", + "long": true, + "results": [ + { + "i": 0, + "prompt_tokens": 38, + "first": "```", + "top": { + "```": -0.03276176005601883, + "import": -3.532761812210083, + "from": -6.032761573791504, + "def": -7.407761573791504, + "REG": -14.782761573791504, + "Here": -15.032761573791504, + "\n": -15.782761573791504, + "#!/": -15.782761573791504, + "#": -15.907761573791504, + "MON": -16.53276252746582, + "SEC": -16.90776252746582, + ">>>": -17.15776252746582, + "\"\"\"": -17.40776252746582, + "ISO": -17.65776252746582, + "^": -18.03276252746582, + "I": -18.15776252746582, + "<<": -18.15776252746582, + "_": -18.40776252746582, + "|": -18.53276252746582, + "0": -18.65776252746582 + }, + "cont": "```python\nimport re\n\ndef parse_iso8601_duration(duration):\n pattern = re.compile(\n r'^P'\n r'(?:(\\d+(?:\\.\\d+)?)Y)?'\n r", + "ttft_s": 0.26 + }, + { + "i": 1, + "prompt_tokens": 21, + "first": "{\"", + "top": { + "{\"": -0.0016210046596825123, + "```": -7.001620769500732, + "{\n": -7.251620769500732, + "{": -15.62662124633789, + " {\"": -16.62662124633789, + "```\n": -20.50162124633789, + "Here": -20.62662124633789, + "({\"": -20.87662124633789, + " {\n": -20.87662124633789, + "json": -21.12662124633789, + "{{": -21.37662124633789, + " ```": -21.62662124633789, + "\ufeff": -21.62662124633789, + "\n": -21.62662124633789, + "import": -21.75162124633789, + "{'": -22.00162124633789, + "<\uff5cend\u2581of\u2581sentence\uff5c>": -22.00162124633789, + "The": -22.00162124633789, + "[": -22.12662124633789, + "{\\": -22.25162124633789 + }, + "cont": "{\"name\":\"Japan\",\"population\":123900000,\"currency\":\"Japanese yen\"}", + "ttft_s": 0.23 + }, + { + "i": 2, + "prompt_tokens": 14, + "first": "Sun", + "top": { + "Sun": -0.07924772053956985, + "The": -2.5792477130889893, + "Ray": -8.45424747467041, + "When": -9.32924747467041, + " Sun": -10.82924747467041, + "G": -11.70424747467041, + "Short": -11.70424747467041, + "**": -12.32924747467041, + "Blue": -12.57924747467041, + "Sh": -12.57924747467041, + "M": -12.95424747467041, + "Light": -13.45424747467041, + "Air": -14.45424747467041, + "Small": -14.57924747467041, + "As": -14.70424747467041, + " sunlight": -14.95424747467041, + "Sky": -15.07924747467041, + "This": -15.20424747467041, + " The": -15.32924747467041, + "Here": -15.45424747467041 + }, + "cont": "Sunlight contains all colors, but air molecules scatter short-wavelength blue light much more strongly than red light (Rayleigh scattering). When you look at the sky, you see this scattered blue light coming from all directions rather than the direct light", + "ttft_s": 0.23 + }, + { + "i": 3, + "prompt_tokens": 18, + "first": "391", + "top": { + "391": -0.00038675934774801135, + "17": -7.875386714935303, + "Let": -13.625387191772461, + "392": -13.875387191772461, + "731": -14.500387191772461, + "431": -14.625387191772461, + "371": -14.875387191772461, + "390": -15.000387191772461, + "381": -15.500387191772461, + "491": -15.875387191772461, + "396": -16.00038719177246, + "291": -16.12538719177246, + "361": -16.12538719177246, + "399": -16.37538719177246, + "39": -16.37538719177246, + "791": -16.50038719177246, + "394": -16.50038719177246, + "397": -16.62538719177246, + "991": -16.75038719177246, + "388": -16.87538719177246 + }, + "cont": "391", + "ttft_s": 0.22 + }, + { + "i": 4, + "prompt_tokens": 18, + "first": "Le", + "top": { + "Le": -0.00013457823661156, + "'": -9.000134468078613, + "\u00ab": -11.500134468078613, + "\"": -14.750134468078613, + "La": -15.125134468078613, + " Le": -15.875134468078613, + "'The": -16.75013542175293, + "The": -17.25013542175293, + "\u00bb": -18.00013542175293, + "**": -18.25013542175293, + "Vo": -18.37513542175293, + "*": -18.62513542175293, + "'S": -19.12513542175293, + "Les": -19.12513542175293, + "L": -19.37513542175293, + "-Le": -19.75013542175293, + "`": -20.00013542175293, + "le": -20.00013542175293, + "Un": -20.12513542175293, + "Gard": -20.37513542175293 + }, + "cont": "Le gardien du phare regardait la temp\u00eate arriver.", + "ttft_s": 0.23 + }, + { + "i": 5, + "prompt_tokens": 14, + "first": "2", + "top": { + "2": -0.38751697540283203, + "The": -1.137516975402832, + "Here": -8.387516975402832, + "Let": -8.637516975402832, + "1": -9.387516975402832, + "Alright": -9.512516975402832, + "Okay": -11.262516975402832, + "First": -11.387516975402832, + "I": -11.512516975402832, + "Of": -11.637516975402832, + "7": -12.387516975402832, + "Hello": -12.512516975402832, + "We": -12.637516975402832, + "To": -12.637516975402832, + "3": -13.262516975402832, + "A": -13.387516975402832, + "Prime": -13.512516975402832, + "You": -13.637516975402832, + "Just": -13.637516975402832, + "Sure": -13.637516975402832 + }, + "cont": "2, 3, 5, 7, 11, 13, 17, 19, 23, 29", + "ttft_s": 0.22 + }, + { + "i": 6, + "prompt_tokens": 13, + "first": "A", + "top": { + "A": -0.6519076824188232, + "Green": -1.1519076824188232, + "Co": -1.9019076824188232, + "Old": -4.776907920837402, + "Fire": -6.651907920837402, + "W": -7.276907920837402, + "The": -7.651907920837402, + "Soft": -7.776907920837402, + "Pat": -7.776907920837402, + "Cold": -8.276907920837402, + "Ham": -8.276907920837402, + "From": -8.526907920837402, + "Ox": -8.651907920837402, + "Forg": -8.651907920837402, + " Copper": -9.151907920837402, + "We": -9.526907920837402, + "Burn": -9.651907920837402, + "Age": -9.776907920837402, + "Black": -10.151907920837402, + "D": -10.276907920837402 + }, + "cont": "Aged copper lantern,\nHolds a soft and steady glow,\nWarm against the night.", + "ttft_s": 0.21 + }, + { + "i": 7, + "prompt_tokens": 35, + "first": "C", + "top": { + "C": -7.772143726469949e-05, + "c": -9.750077247619629, + "W": -11.000077247619629, + "w": -14.125077247619629, + "**": -14.750077247619629, + "The": -14.875077247619629, + "Os": -15.250077247619629, + " Cairo": -15.375077247619629, + "L": -15.625077247619629, + "\u0421": -16.500078201293945, + "CA": -16.875078201293945, + "Since": -17.750078201293945, + "M": -17.750078201293945, + "City": -17.750078201293945, + "Yes": -17.875078201293945, + "<\uff5cend\u2581of\u2581sentence\uff5c>": -18.000078201293945, + "31": -18.000078201293945, + "Paris": -18.125078201293945, + "T": -18.250078201293945, + "A": -18.250078201293945 + }, + "cont": "Cairo", + "ttft_s": 0.26 + }, + { + "i": 8, + "prompt_tokens": 16, + "first": "#", + "top": { + "#": -0.12488572299480438, + "The": -2.4998857975006104, + "Here": -3.3748857975006104, + "```": -6.999885559082031, + "There": -10.249885559082031, + "Fixed": -10.749885559082031, + "I": -10.749885559082031, + " #": -11.499885559082031, + "This": -11.874885559082031, + "`": -11.874885559082031, + "**": -11.874885559082031, + "Looking": -11.999885559082031, + "##": -11.999885559082031, + "That": -12.624885559082031, + "<": -12.749885559082031, + "def": -12.874885559082031, + "Found": -12.874885559082031, + "You": -12.999885559082031, + "Let": -13.249885559082031, + "It": -13.249885559082031 + }, + "cont": "# Bug Fix\n\nThe function is supposed to **add** two numbers, but it's using the subtraction operator (`-`) instead of the addition operator (`+`).\n\n## Fixed Code\n\n```python\ndef add(a, b):\n ", + "ttft_s": 0.24 + }, + { + "i": 9, + "prompt_tokens": 17, + "first": "A", + "top": { + "A": -0.0016189813613891602, + "I": -6.5016188621521, + "The": -9.376619338989258, + "An": -11.626619338989258, + "\"I": -12.501619338989258, + "H": -12.626619338989258, + "In": -13.001619338989258, + "*": -13.251619338989258, + "#": -13.376619338989258, + "Determ": -13.501619338989258, + "Ich": -13.626619338989258, + "As": -14.376619338989258, + "Captain": -14.626619338989258, + "<\uff5cend\u2581of\u2581sentence\uff5c>": -14.626619338989258, + "**": -14.626619338989258, + " A": -14.626619338989258, + "Despite": -14.626619338989258, + "a": -14.751619338989258, + "With": -14.751619338989258, + "*A": -14.751619338989258 + }, + "cont": "A young sailor named Ishmael joins the whaling ship Pequod, whose captain, Ahab, is consumed by a obsessive quest for revenge against the white whale Moby Dick that took his leg, leading to the destruction of the", + "ttft_s": 0.26 + }, + { + "i": 10, + "prompt_tokens": 33, + "first": "#", + "top": { + "#": -0.22785161435604095, + "```": -1.9778516292572021, + "Here": -2.727851629257202, + "##": -10.977851867675781, + "``": -12.477851867675781, + " Here": -13.727851867675781, + " #": -13.852851867675781, + "###": -13.977851867675781, + "I": -14.102851867675781, + "SELECT": -14.102851867675781, + "This": -14.227851867675781, + "SQL": -14.602851867675781, + "**": -14.602851867675781, + "```\n": -14.727851867675781, + "To": -14.727851867675781, + " ```": -14.977851867675781, + "There": -14.977851867675781, + "`": -15.227851867675781, + "Let": -15.227851867675781, + "here": -15.352851867675781 + }, + "cont": "# Top 5 Customers by Total Order Value\n\n```sql\nSELECT\n c.id,\n c.name,\n SUM(o.amount) AS total_order_value\nFROM customers c\nJOIN orders o ON o.customer_id =", + "ttft_s": 0.25 + }, + { + "i": 11, + "prompt_tokens": 30, + "first": "Let", + "top": { + "Let": -0.11893303692340851, + "Alright": -2.3689329624176025, + "The": -4.493933200836182, + "We": -5.368933200836182, + "This": -6.118933200836182, + "Okay": -8.493932723999023, + "First": -9.243932723999023, + "These": -9.868932723999023, + "Looking": -10.243932723999023, + "Here": -10.743932723999023, + "To": -11.118932723999023, + "Sure": -11.368932723999023, + "You": -11.493932723999023, + "H": -11.868932723999023, + "From": -12.243932723999023, + "Step": -12.368932723999023, + "Ok": -12.493932723999023, + "So": -12.743932723999023, + "1": -12.743932723999023, + "**": -12.993932723999023 + }, + "cont": "Let's look at the differences between the terms:\n\n- 6 \u2212 2 = 4 \n- 12 \u2212 6 = 6 \n- 20 \u2212 12 = 8 \n- 30 \u2212 20 = ", + "ttft_s": 0.23 + }, + { + "i": 12, + "prompt_tokens": 18, + "first": "`", + "top": { + "`": -0.4805302619934082, + "```\n": -1.2305302619934082, + "^(": -2.730530261993408, + "^": -4.230530261993408, + "```": -5.105530261993408, + "Here": -6.605530261993408, + "/": -7.230530261993408, + "\\": -7.480530261993408, + "((": -7.730530261993408, + "**": -7.980530261993408, + "#": -9.85552978515625, + "/\\": -10.73052978515625, + "/(": -11.48052978515625, + "``": -11.73052978515625, + "(?": -11.73052978515625, + " `": -11.85552978515625, + "\\\n": -11.85552978515625, + "`\n": -11.85552978515625, + "**(": -12.35552978515625, + "You": -12.48052978515625 + }, + "cont": "`^(?:(?:25[0-5]|2[0-4]\\d|1\\d{2}|[1-9]?\\d)\\.){3}(?:25[0-5]|2[0", + "ttft_s": 0.23 + }, + { + "i": 13, + "prompt_tokens": 21, + "first": "A", + "top": { + "A": -0.6178566813468933, + "Unlike": -1.367856740951538, + "Full": -1.617856740951538, + "Whereas": -5.367856502532959, + "In": -6.117856502532959, + "Sl": -6.867856502532959, + "While": -11.617856979370117, + "#": -12.742856979370117, + "full": -12.992856979370117, + "**": -13.492856979370117, + " Unlike": -14.992856979370117, + "The": -14.992856979370117, + "Standard": -15.242856979370117, + "\n\n": -15.367856979370117, + "Rather": -15.992856979370117, + " Full": -16.117856979370117, + "Instead": -16.242856979370117, + "\u6ed1\u52a8": -16.367856979370117, + "To": -16.617856979370117, + "\n": -16.742856979370117 + }, + "cont": "A sliding-window attention layer differs from full attention in that each token attends only to a fixed-size local neighborhood of nearby tokens (e.g., the previous *w* and next *w* tokens) rather than to every token in the", + "ttft_s": 0.24 + }, + { + "i": 14, + "prompt_tokens": 16, + "first": "45", + "top": { + "45": -0.00016020445036701858, + "0": -10.375160217285156, + "2": -10.750160217285156, + "4": -11.625160217285156, + "453": -11.875160217285156, + "1": -11.875160217285156, + "29": -12.000160217285156, + "32": -12.125160217285156, + "101": -12.375160217285156, + "452": -12.375160217285156, + "Let": -12.375160217285156, + "To": -12.500160217285156, + "46": -12.625160217285156, + "5": -12.875160217285156, + "454": -13.000160217285156, + "The": -13.125160217285156, + "37": -13.125160217285156, + "455": -13.375160217285156, + "64": -13.625160217285156, + "44": -13.625160217285156 + }, + "cont": "45", + "ttft_s": 0.23 + }, + { + "i": 15, + "prompt_tokens": 23, + "first": "```", + "top": { + "```": -0.47528523206710815, + "#": -1.225285291671753, + "Here": -2.600285291671753, + "`": -4.975285053253174, + "**": -6.225285053253174, + "awk": -7.725285053253174, + "```\n": -8.475285530090332, + "##": -8.975285530090332, + "Use": -8.975285530090332, + " ": -9.350285530090332, + "You": -9.475285530090332, + "``": -9.600285530090332, + "Assuming": -9.975285530090332, + "This": -10.600285530090332, + "Sure": -10.850285530090332, + "Using": -10.975285530090332, + "To": -11.100285530090332, + "