Skip to content

dasLLAMA llama leg: the Vulkan window chain holds segments, a short continuation peels through the token command, batched steps past eight rows group, and both homes' decode rope-and-store ride one template - #4293

Merged
borisbat merged 30 commits into
masterfrom
bbatkin/llama-leg
Oct 12, 2026
Merged

dasLLAMA llama leg: the Vulkan window chain holds segments, a short continuation peels through the token command, batched steps past eight rows group, and both homes' decode rope-and-store ride one template#4293
borisbat merged 30 commits into
masterfrom
bbatkin/llama-leg

Conversation

@borisbat

@borisbat borisbat commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator

Behavior change: the Vulkan resident driver's per-row planes hold eight rows whatever the region count - a one-region plan charges about 5 MB more scratch on Llama-3.2-3B.

Why. The llama family served less on Vulkan than on Metal: a prompt of several sessions ran one session at a time, a short continuation paid the window chain (about two decode steps whatever its row count), more than eight streams stepped a row at a time, and the one-row decode's attention split filled a quarter of the card. The decode rope-and-store kernels also carried one body a home.

What changes.

  • The Vulkan window chain holds several sessions' chunks as segments, each on its own mirror region, with one classifier slot a region.
  • A continuation of at most eight rows peels through the token command with every row in the session's region (set_vulkan_prefill_peel is the lever).
  • A batched step past eight rows runs in balanced groups of the N-row command.
  • The one-row decode's attention split follows the command's rows, not the per-row planes' height.
  • Both homes' decode rope-and-store stamps ride one template, GkRopeKvStore; the Metal window form too.
  • The Metal prefill's last-layer narrowing takes a K-quant FFN's one-row GEMV form.
  • Serving capability added on one home lands on the other in the same change (REVIEW_GPU.md, new rule).

Observable behavior.

  • Two host-cached sessions' prompts on Vulkan: two window chains -> one chain, vk_rdec_prefill_segment_windows up by one.
  • A 1-8 row continuation on Vulkan: the window chain (13.5 / 11.7 / 13.4 / 13.5 ms at 1 / 2 / 4 / 8 rows on Llama-3.2-3B Q4_K_M, the race bed) -> the token command's peel (6.5 / 6.9 / 4.5 / 7.4 ms), one decode step (debug-jit, direction-grade).
  • Twelve device-home streams on Vulkan: each step twelve one-row commands -> two commands of six (rdec_nrow_steps).
  • The house replay's 22-turn tool call on the pod: the decode attention 2220 -> 1274 us a token at about 5k keys (the step 6232 -> 5295; external, direction-grade).
  • A routed model with more than eight picks a row: the N-row command declined whole -> steps MAX_ROUTED_SLOTS over its picks rows at once (six on a top-10 model).
  • A Vulkan one-region plan: per-row planes at one row -> eight (resident_plan_pick charges RDEC_NROW_MAX rows).

Where to look. dasllama_gpu_resident.das (rdec_prefill_gate, rdec_prefill_peel, rdec_batch_rows_at_once), dasllama_vulkan_prefill.das (pf_attn_block under row0), rd_layer_pieces in dasllama_vulkan_decode.das, and the GkRopeKvStore template in dasllama_gpu_kernels_common.das with its four Metal and two Vulkan stamps.

Validation, claims, ledger

Validation

  • Boxes: RunPod RTX PRO 4500 Blackwell 32 GB (Vulkan, Linux, no Metal module) and Apple M5 Max (Metal).
  • Pod, tip e38a26b: --changed --base origin/master 179 files, 399 cells skipped, 3 red - test_gpu_resident_regions_hybrid, test_gpu_resident_segments, test_vulkan_kernels, each a cell's own expectation (the handoff cell counting a recurrent model's whole prompt as two windows, the tails cell expecting the segmented-window counter to count windows rather than windows holding both segments and its busy arm leaving device sessions unparked, the h128 row-base arm asking a descriptor set of a class it never ensured), fixed in faf9ef9; twelve files run whole green in the same run (test_gpu_tier, test_gpu_serving_declines, test_vulkan_tier, test_vulkan_dec_tail, test_gpu_resident_llama, the plain, llama_k, e2b and hybrid regions files, test_gpu_resident_gemma4_e2b, test_gpu_resident_hc and test_gpu_resident_moe under DASLLAMA_PARITY_FULL=1). At faf9ef9 the three fixed files green, with test_gpu_resident_regions, _regions_e2b, _regions_q8kv, _regions_tq4kv, _regions_qwen3, _regions_llama_k, _regions_gemma4 (PARITY_FULL), _regions_llama3b_ctx32k, _llama and test_scheduler. At a9fa027 (the segments file's busy arm alone changed since; the commits after it touch PERF_LEDGER.md alone) test_gpu_resident_segments green on the f16 mirror and under DASLLAMA_VK_KV32=1, its readings in the ledger - the Q4_K_M short tails' session B reads 0.47 of its 0.50 split bar on the f16 mirror (0.30 on the f32 one), the file's nearest reading.
  • M5, tip e38a26b: --changed --base origin/master 179 files, 5 red - test_gpu_slot_swap, test_vulkan_dec_tail, test_vulkan_kernels, test_vulkan_tier, test_vulkan_tower_kernels, the box's MoltenVK class (ERROR_INITIALIZATION_FAILED at device creation, or its SPIR-V to MSL conversion refusing a kernel), the same files red on master; --suite kernels --arm kernels 7 files green.
  • The f32 mirror stamp (RopeKvStore) serves a model only under DASLLAMA_VK_KV32=1: test_gpu_resident_llama and test_gpu_resident_regions_llama_k green at e38a26b, test_gpu_resident_segments green at a9fa027 (its busy arm had homed its device sessions on the f16 default, which the f32 mirror refuses by panic; it now asks the armed codec). The race bed ran on the default f16 mirror; its log reads calls the driver passed to the CPU rails: none on both carriers at e38a26b (the 3B Q4_K_M and the 1B Q8_0, five reps a form at 1 / 2 / 4 / 8 rows).
  • Every figure is debug-jit and direction-grade (no Vulkan rig leg exists); the ledger section "From the llama leg's parity gaps" in PERF_LEDGER.md carries the command lines, commits and spreads. The Metal narrowing claims no speed.
  • Metal fold identity: the emitted MSL of MetalRopeStoreF16 reads boff = (0u), tcos[((0u) + cb) + j] and vb = ... + (0u) - the template's bases fold to the one-row form's constants.
  • Per-stamp kinds: Metal MetalRopeStoreF16/F32 (BOUND_VIEWS, one row, rope_kv_one), MetalRopeStoreBF16/BF32 (SPLIT_TABLES, the window's row table, rope_kv_tabled); Vulkan RopeKvStore/F16 (the token command's tok binding). Cells: rope_kv_arm (test_vulkan_kernels.das), rpst_flat_gate / rpst_b_gate (test_metal_rope_kernels.das), rope_store_pocket_gate (test_metal_prefill_kernels.das).
  • Tier probe registries the model-free tier cells register stay registered for the process (the cells unset their seats; the registry rows are the probes').
  • Master merged at ce0282b (the facade split into dasllama_core; the leg's one facade wrapper moved there). The docs lane's digest check reds on master since that split - the digest's dasllama page walked the facade module alone, which now holds only requires - and is green here: the page folds the core module (doc/reflections/das2rst.das), the regenerated digest differing from the tracked one by the wrapper's line alone. Preflight after the merge: the fast tier green, then the docs, tests-cpp, tests-interp, tests-jit and tests-aot lanes each green alone.
  • [arch] citations: every citing function of the sections the leg adds or changes read against its section at ecc4340 (logs/audits/bbatkin-llama-leg/ecc4340e9a-arch-cite-verdicts.md): five drifts, all in the documents - the split cap's row dependence (ATTN, NROW), a group picking its own form, what rdec_nrow_steps counts, the segmented window's final FFN - restated in the sections on 43c6086 and after; every other citation holds.

Claims - stated, not tested

  • The Vulkan tower's four fa_stamp_enc sites pass row0 = 0, so the flash tiles' tower rows sit where they did: the tower twins in --changed on the pod hold it (test_vulkan_tower_kernels 34 of 34 at e38a26b, test_vulkan_tts_* and the audio and vision twins green in the same run).

Not done

  • Folds assessed and declined, different computations: add+rms (Metal's one 1024-thread loop with its f16 twin against Vulkan's residual family with sandwich, requant and combine arms), swiglu (Vulkan fuses the Q8 requant), argmax (one threadgroup against a two-pass reduce), fwht (hs threads against 256 x 2). RopeKvStoreBT (the Vulkan window's rope-and-store) stays its own body: its k/v planes sit at voff, the q head is gated and the pass-through write is skipped - axes the shared template does not carry.
  • Tool-call streaming is its own PR.
  • The Vulkan rope_kv_arm kernel cell runs one NEOX row with the table, row and mirror bases at zero; the token command's NORM pairing over several rows with its bases is held at the model level (the llama resident file's forced-feed bar and the regions files' batched rows against the row alone), not by a kernel-unit arm.
  • A gated model's chunked attention at a row base past zero has no kernel cell: the gated carriers are recurrent, which the segmented window declines, so serving never passes a nonzero row0 to the gated stamps.
  • The one-row and N-row commands sum a layer's keys alike only while the N-row command's rows fit one SM wave and under the wide position; past that the regions rig holds the split bar (vk_rdec_rows_split_alike) - a card with few SMs reads the demotion in its log.
  • Device-home streams on a model the segmented window declines (recurrent, per-layer-embedding, NextN, routed, hyper-connection, a block-codec mirror) keep the one-stream tick (gpu_device_prefill_segments), as before the leg.
  • A q8_0 or tq4 K/V session on Metal keeps the panel-fed prefill window, its cached rows gathered on the host each continuation, where an f16 session takes the mirror-fed form; the Vulkan driver's block-codec mirror has no Metal twin in this leg (followup_metal.md row 36, a non-default knob).
  • Checklist self-review findings not folded (prose structure, no obligation changed): REVIEW_IMAGE.md's split of hidden duties (:42-45, :54-56, :111-113, :119-120), the glossary paragraph, the layout-stamp closure's "unless"; REVIEW_TOWER.md's run lists as a test tag and the opening paragraph; REVIEW_KQ_FORMATS.md's definition paragraph; REVIEW_KERNEL_CELLS.md's read-side routing trigger (owned by REVIEW_GPU.md); REVIEW_GPU_PARITY.md's glossary and knob mapping; REVIEW_PINNED_GATES.md's entry forms.

borisbat and others added 30 commits October 11, 2026 19:24
@borisbat @claude
The Vulkan window chain holds several sessions' chunks as segments
22e3b6b 
The resident driver registers a segments override beside its whole-prefill one: several
host-cached or device-home sessions' chunks prefill as one window chain, each segment in its
own mirror region. The GEMMs, feeds, activation and residual steps run once over the window;
the rope-and-store writes each segment's rows at its positions of its region, the attention
binds each segment's query and output rows at its row base and its keys at its region, and
every segment's last row takes the classifier into its own logits slot. A window past
PF_WINDOW rows loops as before, the segment parts inside each window the window's own. The
one-session call is one segment at row 0, so both forms are one code path.

The form declines what the driver serves one session a call - a recurrent, per-layer-embedding,
NextN-headed, routed or hyper-connection model, a block-codec mirror, a span, a session the
one-session form would pass, and two segments bound to one region - and the engine then runs the
segments one at a time through the one-session override.

The scheduler's prefill tick hands a device-home scheduler its segments too, but for a model
whose device prefill cannot continue past zero; a device-home stream's chunk takes the idle
quantum while no stream decodes, as a host-cached one does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The Metal prefill's last-layer tail narrows a K-quant FFN too
8624bce 
The last dense layer's FFN runs the window's last row alone where every FFN plane has a one-row
form: q8's GEMV, as before, or a superblock format's row-batched decode form at one row, whose
K must be whole superblocks. A Q4_K_M carrier's window no longer runs its last layer's three
FFN GEMMs over every row.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The decode rope-and-store rides one template on both homes
dc9f940 
GkRopeKvStore holds the fused rope and K/V store of one token row: a q pair ropes in place, a
k pair ropes into the mirror K row and copies its v pair into the V row, the bias rows added
first, the f16 stamps clamped into the half range, the pairs past the rotated width copied
through. Metal's MetalRopeStoreF16/F32 and Vulkan's RopeKvStore/F16 stamp it; each stamp's
kernel resolves the row's mirror bases its own way - Metal binds the row at an offset, Vulkan
reads its token block - and hands them to the shared pair body. The two per-home argument
structs give way to the template's, built by rope_store_args and rope_kv_args.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The Vulkan batched step past eight rows runs in groups, never a row a…
16880fe 
…t a time

A batched decode step of more rows than the N-row token command steps at once used to fall
to the one-row command for every row, so a ninth device-home stream cost the step a weight
pass a row. The resident driver now binds every row's region first and runs the rows in the
fewest balanced groups of at most the command's rows, a single-row remainder through the
one-row command. The llama K-quant regions file gains the twelve-row cell, and the GPU
checklist the rule that a serving capability one home gains lands on the other home in the
same change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The Vulkan resident driver peels a short window through the token com…
f993e3c 
…mands

A prefill of at most the rows the N-row token command steps at once ran the whole window chain,
which costs a short continuation about two decode steps whatever its row count. The resident
driver now runs such a window through the token command over the session's region, every row at
its own position with its own key count, the last row landing the logits. The llama resident file
holds the peeled tail to the one-row steps on the Q8_0 and Q4_K_M 1B.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The Vulkan driver's per-row planes hold eight rows whatever the regio…
5c5d49e 
…n count

The N-row token command's planes were sized to the region count, so a one-region driver had no
N-row form and the short window's peel could only take one row. The planes now hold RDEC_NROW_MAX
rows always, the peel cells take the prefill call a scheduler's chunk takes, and the performance
ledger gains the leg's measured rows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The decode attention's split follows the command's rows, not the plan…
a343012 
…es' height

The key-split piece count divided the SM wave by the per-row planes' height, so the one-row command
split its keys as if it stepped eight rows: the house replay on the pod read the decode attention
at 3.8 ms a token against 2.2 at a 5k-token context. The wave now takes the rows the command steps.
The diff's working comments are harvested: two facts land in the architecture docs, three names say
what their values are, and the serving doc's tick sentence matches the code.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The segments form declines what the one-session call declines
17f4579 
The engine offered the active override's segments form every window, where the one-session call
first declines a session roping from per-row tables or adding deepstack slices unless the override
registers that arm. The segments entry now asks the same of every segment, and a model-free cell
holds the predicate under a probe override with neither arm.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The rope-store twins hold one body a home, and the review round's fin…
e0b74f2 
…dings land

The Metal and Vulkan rope-store stamps each held the guard and index arithmetic twice; now a Vulkan
template and a shared one-row method carry them, the element bases a stamp binds at their offsets
fold to zero at compile time, the table rows are reads, and every division in the shared body
clamps its divisor. The peel's lever and the race harness answer the timing claim, the segmented
and peel declines log their concrete reason once per armed model, the admitted-format arm ANDs the
served-format predicate, the resident rigs share one split bar and decode both sides' argmax rows,
every served-route counter has its control, the in-place rope gates assert their teeth, and the
checklists the round audited carry their fixes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The window kernels' row base has kernel-unit cells past zero
c31c31a 
The flash tile, the batched decode attention and the batched rope-store gained a row base for the
segmented window, and their gates dispatched it at zero alone, where every read and write sits
where it did. Each now has an arm behind lead rows of poison: the lead rows stay untouched and the
rows read the oracle's. The kernel-cells checklist carries the duty.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The overrides and memory audits' findings land
f6f08f1 
The segmented staging arrays are exact-size, the pinned-segment decline and the peel's announce
name the setter a user would call, the mm-tail lever's off announce says the narrowed row keeps its
one-row form, the unset segments seat names the module to require, and the two checklists carry
their self-review fixes; the leg's lasting allocations are in the ledger in bytes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The hot-path audit's findings land
2cf6266 
The segments prefill entry carries the one-session entry's contract and reads the active override's
segments form picked at select, the said-once declines pay the lookup alone on the step and format
in a cold function, and the test and harness roots that reach a region entry carry the cold mark.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The dupe and TDD audits' findings land
0f1e793 
The Metal window rope-store stamps the shared template through a row-table method, both homes
build their rope args through one builder, the peel stages its rows through the embed and rope
helpers, the segments epilogue is the landing, the said-once helper lives in the tier for every
driver file, the one-session call and the segments form share one admission gate, the mirror base
takes a region, the narrowed row rides the site GEMV, the races share a fixture and the resident
rigs one stepper and one N-row body. The branches the audit found untested gain their cells: the
segments form's codec, busy and deepstack declines, the E-series groups past eight rows, the
short window on the models the peel declines, the say-once helper, the one-row GEMV predicate,
and the plan's eight-row charge.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The lane-pin, measurement, parity, kernel-cell and docs audits' findi…
a70c5a2 
…ngs land

The served cells pin the route and the peel's lever and put them back, the race bed prints its
slowest rep and the calls passed to the CPU rails, the ledger's leg section carries its tags,
command lines and commits, the rope benchmark binds the shared template's slots, the row-base
cell covers the three batched attention stamps, the in-place rope gates assert past the bar, and
the two over-cap architecture docs route their new sections under the line cap.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The peel's lever loses its facade underscore, and the format and visi…
3349994 
…on checklists' self-review findings land

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The image, tower and pinned-gates checklists' self-review findings land
2983b36 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The kernel races keep their fixture; the window races read their own
ecc4340 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The segmented admission decides before any bind, and the split cap's …
43c6086 
…drift is stated

The segments form's codec and region checks count no pass - the one-session fallback counts the
pass it takes - and the owners bind their regions before the segments with none; the busy leg
pins its device sessions. The attention split's wave cap reads each command's own rows, so the
one-row and N-row commands sum the keys alike only while the rows fit one wave:
vk_rdec_rows_split_alike says so and the regions rig holds the split bar past it. The N-row
counter counts commands, a group or a peel each; a segmented window runs the last FFN whole.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The Vulkan architecture doc sits under the line cap
67563e8 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
A routed model's N-row command steps the rows its slot planes hold, a…
2243ccc 
…nd the say-once tables clear on every upload

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The review round's findings land
660fc2b 
The one-row and N-row commands' split agree only while the rows fit one wave and under the wide
position: vk_rdec_rows_split_alike reads the one-row form at a position and the regions rig holds the
split bar past it, the long-context streams included. A device-home tick hands its streams to the
segmented window only where the driver's window serves the model (gpu_device_prefill_segments);
the device-mode cell witnesses the window. The codec gate sits where it sat, right after the pin;
the driver's segments entry asserts its own exclusions; an override with no segments form leaves
none cached. A two-window segmented call has its cell; the row-base kernel cells hold the rows past
the segment untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The dragons' first pass over the GPU, parity, kernel-cell, docs and m…
6cf64d7 
…easurement checklists lands

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The dragons' cold re-read of the GPU, parity, kernel-cell, docs and m…
28de4e2 
…easurement checklists lands

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The dragons' first pass over the image, format, vision, tower and tes…
b859310 
…ts checklists lands

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The TDD audit's untested branches get their cells, and the dragons' c…
e38a26b 
…old re-read lands

The peel's shape arms are one pure function held on dense shells; the say-once helper answers
whether it said it; the segments admission's dry run has its owner-first, outnumbered and gap
cells; the device-home tick's handoff has its cell on every regions rig; the hyper-connection
trunk holds the N-row command at six rows; one row splits a layer's keys as one row does. The
tower, image, format and tests checklists take their cold re-read's findings.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The pod's three reds at the frozen tip get their cells right
faf9ef9 
The h128 row-base arm of the batched decode attention cell asked a descriptor set
of a class it never ensured; it ensures the class the way the h128 cell does. The
300-row tails cell expected the segmented-window counter up by the call's windows,
but the counter counts the windows that held both segments - one, the window the
boundary falls in; the second window holds B alone. The busy arm of the decline
cells deleted its device-home sessions without parking them, so the regions stayed
pinned and the deepstack arm's other session passed as busy, which the arm's
reason-count compare could not see: the sessions are parked, and the arm holds the
total pass count. The handoff cell's decline arm counted the second stream's prompt
as two windows; a model whose prefill cannot continue takes it whole, two calls.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The segments file's busy arm homes its device sessions on the armed m…
a9fa027 
…irror's codec

Under DASLLAMA_VK_KV32=1 the mirror is f32 and a device-home session made on the f16 default is handed to the CPU rails, which the driver refuses by panic; the arm now asks the mirror's codec as the regions rig does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat @claude
The ledger carries the leg's race rerun, footprint pair and segmented…
5784db9 
…-window readings

The race bed's second reading at e38a26b beside the first, the one-region bench's max RSS at the tip against the base with the JIT cache warm, and the segments file's widest misses against the split bar on both mirrors.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat
Merge remote-tracking branch 'origin/master' into bbatkin/llama-leg
56f38be 
# Conflicts:
#	modules/dasLLAMA/dasllama/dasllama.das
@borisbat @claude
The dasllama digest page reads the core module the facade re-exports
c4c32c5 
The facade's documented functions moved to dasllama_core, which the digest's dasllama page did not walk, so the regenerated digest emptied the section; the page now folds both modules, and the device-home segments wrapper joins its serving group.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 12, 2026 17:27
Copilot AI reviewed Oct 12, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It is a very large, multi-backend GPU-kernel refactor (shared rope-and-store template, Vulkan segmented prefill/peel/batched steps, scheduler changes) whose correctness rests on hardware-validated numeric behavior that cannot be fully verified by static review.

0 open findings

What changed in this PR

This PR is a large "llama leg" for the dasLLAMA GPU inference backend. Its central refactor unifies the Metal and Vulkan decode "rope-and-store" kernels onto one shared GkRopeKvStore class template (with a gk_rope_kv_args argument builder), and layers several Vulkan serving improvements on top: a segmented prefill window that holds several sessions' chunks as mirror-region segments, a short-continuation "peel" through the token command (≤8 rows), batched decode steps grouped past eight rows, a one-row decode attention split that follows the command's rows, and a Metal prefill last-layer K-quant GEMV narrowing. A new REVIEW_GPU.md rule requires serving capability added on one backend to land on the other in the same change. The change is accompanied by extensive ledger/architecture/review-doc updates and new/updated test cells and race-fixture harnesses.

Changes:

  • Unify Metal + Vulkan rope-and-store kernels onto the shared GkRopeKvStore template and route all call sites through gk_rope_kv_args.
  • Add Vulkan segmented prefill (gpu_device_prefill_segments, segs_served_by_override), the ≤8-row peel, grouped N-row batched steps, and a row-following one-row attention split; add the Metal K-quant one-row GEMV narrowing (kq_fmt_one_row_gemv).
  • Update tests, harnesses, docs (everything.md, REVIEW_*.md, ARCHITECTURE_*.md, PERF_LEDGER.md) and the das2rst digest to fold dasllama_core.
File Description
skills/​daslang/​references/​everything.md Adds the gpu_device_prefill_segments SPI entry (alphabetical, matches the new core function).
modules/​REVIEW_SHADER_EMITTERS.md Clarifies that a claim about emitted text content (vs. naming a das construct) is an emitted-shape claim.
modules/​dasLLAMA/​tests/​test_metal_rope_kernels.das Converts RopeStoreArgs/RopeStoreBArgs to gk_rope_kv_args, reorders template binds, adds "has teeth" in-place-compare assertions.
modules/​dasLLAMA/​tests/​test_metal_prefill_kernels.das Same template-arg/bind conversion for the f32 rope-store pocket gate plus a teeth assertion.
modules/​dasLLAMA/​tests/​test_gpu_tier.das Adds test_segments_served_by_override and test_gpu_said_once cells; requires dasllama_common.
modules/​dasLLAMA/​tests/​test_gpu_serving_declines.das Adds test_kq_fmt_one_row_gemv and test_peel_shape_arms coverage for narrowing/peel predicates.
modules/​dasLLAMA/​tests/​test_gpu_resident_regions*.das Wire the new cell_scheduler_device_handoff cell into the regions scheduler tests.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@borisbat
borisbat merged commit fa7ce42 into master Oct 12, 2026
39 checks passed
@borisbat
borisbat deleted the bbatkin/llama-leg branch October 12, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@borisbat