Repository navigation
Conversation
Signed-off-by: owpeter <pt909795@163.com>
Signed-off-by: owpeter <pt909795@163.com>
Declare EmbodiInfer's core Triton operators (pi0.5, ActiveVLN vision, StreamVLN decode) once in scripts/kernel_tuning/operators.py and render complete task packages from them. The baseline is the production kernel, vendored verbatim with its repository imports. Each contract follows the production kernel: bit-exact where it already is, otherwise a stated BF16 rounding-step tolerance. Add `generate` and `tune-all`: one batch generates every task, preflights all production baselines with the evaluator, runs each operator serially in its own run/resume subprocess, exports promoted kernels and writes a summary. Task validation now rejects variable axes that cannot be inferred from inputs. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
FlashInfer's do_bench synchronizes before each sample, so the start event precedes the host launch and every sample includes launch latency and jitter. On an RTX 5090 this put microsecond kernels at about 10 us and let identical solutions differ by up to 53%, far above the 3% promotion threshold. Queue the cold-L2 flush and input reset ahead of the start event instead, so the events bracket only device execution. Generated tasks now default to 1000 iterations, and preflight reports each baseline latency so operators already at the device's launch floor can be skipped. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
`capture` runs an unmodified model script with every catalog kernel wrapped and records each call as task axes, scalars and fixed inputs with call counts; `generate --captured` and `tune-all --captured` turn the most frequent cases into weighted workloads. A pi0.5 LIBERO-10 capture is included. It shows a 560/576-token prefix rather than the estimated 968, a prefix-only split-KV attention call, and BF16 AdaRMS modulation, which the catalog now declares. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Count kernels captured into CUDA Graphs on every replay, so production graph configurations can be captured, and add `skeleton`, which builds a random-weight checkpoint from safetensors headers: shapes do not depend on weight values. Committed captures now replace the catalog estimates by default (`--estimated` restores them). New captures for the Qwen R2R low-level and panoramic policies and StreamVLN correct the vision shapes (1,980/7,704 patches in ragged windows) and show decode-only StreamVLN calls. The rotary tables are BF16, which makes rotate_half_rope bit-exact, and the vision kernels belong to the Qwen R2R policies: ActiveVLN does not call them. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Capture ATen overloads and backend calls during model preparation, exclude quantized scopes, and require complete task coverage and baseline preflight before starting a model tuning batch. Preserve tensor layouts, aliases, mutations, default-generator RNG transitions and float64 values during replay. Handle zero-dropout SDPA and retain the legacy catalog workflow behind explicit flags. Validation: 104 local tooling tests and 35 server discovery/replay tests passed. Pi0.5 random-weight validation on RTX 5090 generated 94 tasks and passed all 94 baselines without starting search. The broader CPU suite retains one existing Transformers import-order test conflict. Signed-off-by: owpeter <pt909795@163.com>
Treat aten.item as host-side scalar work like _local_scalar_dense. Torch 2.12 delivers Tensor.item() to the observer as aten.item, which was recorded as a blocked non-tensor call and marked the Qwen R2R low-level capture incomplete. Snapshot inputs when a host-input call is promoted to ready because its output lands on the target device. Host-to-device copies such as integer index tensors moved to CUDA previously had no fixture and failed preflight with "Non-floating inputs require a valid captured fixture". Promoted calls that mutate their inputs are blocked. Observe decoder.integrate. The pi0.5 benchmark times the flow loop through integrate, whose native path reaches _denoise_step_impl without passing any observed method, so the whole denoising loop was missing while coverage still reported complete (58 instead of 96 pi0.5 tasks). Validation: three regression tests fail before and pass after the change; tests/kernel_tuning passes on CPU (97 passed, 19 skipped). On RTX 5090 the fixed recorder captured pi0.5 (96 ready) and Qwen R2R low-level (124 ready) with no blocked calls, and every baseline passed preflight. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Signed-off-by: owpeter <pt909795@163.com>
Operator discovery used to observe only a fixed list of policy and decoder method names. Recurrent decoding (decoder.decode), RL sampling, serving pre/post-processing, engine output packing and any work on executor threads ran outside that scope and were silently missing from complete captures. Discovery now observes everything below the model-agnostic inference entries (EngineCore, GenerationBackend, policy collate and serving adapters), with per-thread scope so async and batched serving workers are covered. Model weights read outside every entry, or a run without inference, are capture errors instead of a smaller operator set. - Add discovery.drive, which builds the policy and engine with the serving arguments and replays recorded requests or observations, one session per recurrent episode, plus benchmark-backed input exporters for pi05, the Qwen navigation profiles and StreamVLN. - Treat zero-dropout SDPA backend overloads like the composite call and skip their uninitialized philox outputs during replay. - Classify host-side preprocessing with unrepresentable arguments as host_only. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
With native_inference and cuda_graph enabled, the prefix and denoise-loop graphs are captured by the PI0.5 native runtime, not by EngineCore, so the warmup check reported "warmup captured no graph" and stopped. Count both. The native runtime also captures one graph per language bucket and dense/masked variant. Ten warmup calls did not cover every layout of the 1,600 LIBERO-10 frames, so a capture happened during measurement and the run was rejected after it finished. Warm each layout once before timing, as compare.py already does, and record the warmed layouts in the report. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
PI0.5 chose between Torch and Triton norm, residual, activation and RoPE by importing embodiinfer.backend.triton directly behind per-call flags, so a tuned kernel had no place to plug in. Add embodiinfer.layers.ops: operators are named by their mathematics (gated_residual, ada_rms_norm, gated_gelu, rotate_qk), each with a Torch reference in embodiinfer.backend.torch.ops and the existing Triton kernel. KernelSelection resolves an operator for the caller's preferred implementation under a kernel_profile: - none (default): the preferred implementation, as before - torch: every registered operator uses its Torch reference - auto / <name>: verified kernels from embodiinfer/backend/triton/tuned/<name>, matched on GPU, compute capability and Torch/Triton versions, with a per-kernel applies() fallback and a source digest check PI0.5 routes its four switch points through the registry and accepts kernel_profile; the benchmark passes it through. prepare() resolves every operator before torch.compile regions so graph tracing only reads a cache. Torch references stay uncompiled. The former _inference_helper wrappers for these helpers were first created while Dynamo traced the prefix encoder, which stores the raw function, so they never ran compiled outside that graph. Compiling them now would change the Torch path's numerics. Verified on Jetson Thor (torch 2.13.0+cu132, triton 3.7.1, BF16, B=1, pi05_libero_finetuned_v044, LIBERO-10): kernel_profile=none is bit-exact against the previous code for 1,600 native-path frames and for 20 frames of the Inductor + CUDA Graph path with TORCHINDUCTOR_DETERMINISTIC=1 and fresh compilation; native-path mean latency 99.35 ms vs 98.47 ms before. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Kernel tuning so far optimized isolated operators whose results never reach the model. Propose a model-level loop instead: profile the production configuration, select the regions (including new fusion boundaries) worth tuning, tune Triton against the Torch implementation as baseline, install verified kernels per hardware profile behind kernel_profile, and accept them only under a frozen model-level numerical gate and end-to-end latency. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Add `python -m scripts.kernel_tuning profile --output DIR -- DRIVER`, run in the model runtime around an unmodified inference driver such as discovery.drive. Each top-level EngineCore.execute, GenerationBackend rollout or serving call is one step; after --warmup steps, --steps steps are profiled with torch.profiler. Each GPU kernel is attributed through its CUDA correlation id to the host launch and then to the ranges enclosing that launch: policy and decoder protocol stages, operator-registry calls, attention-backend calls and nn.Module calls with layer indices folded. Ranges are skipped while torch.compile traces, and kernels replayed from CUDA graphs keep only the ranges active at replay, so graphs and compiled regions report per stage. The report (hotspots.json, hotspots.md) ranks regions by GPU time per step with kernel counts, mean kernel time and the share of small kernels. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Kernels replayed from a CUDA graph were attributed only to the stage that replayed the graph, which hid operator-level cost in the production configuration where most work runs inside graphs. Profile each graph capture outside the measured window and record the ranges of every node launch in capture order. Measured replays are tagged with their graph, and a replay's kernels take the capture labels in device start order when the counts match, prefixed by the replay-time stages. Regions without an operator, attention or module range (compiled code, unlabelled replays) are split by kernel category from the kernel name. On Jetson Thor, pi0.5 with native graphs: 95% of graph-replayed kernel time is now labelled; with the Inductor full-loop graph: 100%. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
The denoise attention is the largest single hotspot of PI0.5's native path, but it was called on the Triton attention backend directly, so a tuned kernel could not replace it. Register split_kv_attention with the existing Triton kernel and a Torch reference (concatenated K/V, scaled_dot_product_attention with the padding mask as a boolean keep-mask, grouped-query aware). PI0.5 routes both native calls through the registry: the denoise call over cached and current K/V, and the native prefix call, which is the same kernel with an empty cache. Without a kernel profile both still call the same Triton function with the same arguments; _prepare_native_attention keeps its capability check. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Model discovery tunes isolated ATen calls against themselves. The new goal is a Triton kernel that beats the model's Torch implementation of an operator-registry entry, evaluated on the model's real inputs. Add `capture-ops` (scripts/kernel_tuning/regions.py): it runs an inference driver in eager mode and records every registry call of the measured steps, keeping the first real inputs of each layout as fixtures and counting the calls. The output is a discovery-format model capture whose operators have identity kind "registry", so calibrate, generate and tune-all apply as is: - the baseline and its FP64 oracle are the Torch reference in embodiinfer.backend.torch.ops, evaluated in FP64 for calibration; - output layouts are those of the Torch reference; - the implementation the model ran is vendored as the agent's starting point and digest-checked, like the baseline sources; - tasks time with CUDA graphs, as the operators run in production. On Jetson Thor (pi05, LIBERO-10, eager native path, 4 steps) capture-ops, calibrate and generate produced 7 tasks for split_kv_attention, ada_rms_norm, gated_residual, gated_gelu and rotate_qk. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Add `install EXPORT --profile NAME`, run in the model runtime on the target machine. It copies the winning sources of a registry task into embodiinfer/backend/triton/tuned/NAME/<op>/ as a package and records the operator in profile.json, matched to the current GPU, compute capability and Torch/Triton versions, with the run, task and capture identities as evidence. The generated package exposes applies(), true only for argument layouts the evaluator verified (shapes, strides, dtypes and constants of every recorded workload); other calls fall back to the preferred implementation. The profile loader now accepts package directories and checks a digest over all of their files. Reinstalling needs --replace, and a profile verified for another environment is refused. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Add `verify --profile NAME --output REPORT -- DRIVER_ARGS`, run in the model runtime with discovery.drive inputs. Every input runs three times with the same seed: an FP32 reference and a baseline in the requested dtype, both with kernel_profile=torch, and the candidate profile. For each input, continuous outputs must stay within twice the baseline's error against the reference (max-abs floor 1e-4, RMSE floor 1e-5), with thresholds fixed from the baseline alone; discrete outputs must equal the baseline. A baseline far from the reference means the runs did not share inputs or noise and is rejected rather than used as a threshold. The command exits nonzero when the gate fails. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
A per-input bound of twice the Torch baseline's error is noise-dominated: on Jetson Thor, two Torch runs of the same 40 pi0.5 inputs differed per input, and the existing Triton path failed the bound on two inputs where Torch happened to land within 1e-3 of the FP32 reference, although its mean errors equalled Torch's. Judge each continuous output over the whole evaluation set instead: the candidate's mean max-abs and mean RMSE may exceed the baseline's by at most 25%, and its worst max-abs may be at most twice the baseline's worst (floors 1e-4 and 1e-5). Per-input errors stay in the report; discrete outputs must still equal the baseline exactly. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
An installed kernel's `applies` compared strides exactly, so a size-one dimension recorded as (L*D, L*D, D, 1) in the eager capture never matched (L*D, D, D, 1) under Inductor, although such strides address no other element. Compare strides of size-one dimensions and of empty tensors as zero, in both the recorded signatures and the generated predicate. The evaluator calls a solution with the captured tensor arguments x0..xN only; constants such as the attention scale are part of the task. The generated package forwarded the policy's full argument list, so a matching call with a constant failed. Bind the tensors through the captured argument recipe and call the kernel as the evaluator did; `applies` already pins the constants to the verified values. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
The prefix pass cached K and V as views of the fused projection in eager mode, but Inductor returns them contiguous. Registry calls are captured eagerly while production compiles the prefix, so the denoise attention of the two runs saw different prefix V layouts and a layout-gated tuned kernel was never selected. Cache contiguous K/V in both modes. Under Inductor the copy is a no-op: with kernel_profile=none, 200 LIBERO-10 benchmark frames on Jetson Thor (BF16, B=1, TORCHINDUCTOR_DETERMINISTIC=1) were bit-identical to the previous code on both the native and the Torch path. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Install the first agent-tuned kernel through the full pipeline: hotspot report, registry capture, FP64 calibration, Humanize2 tuning against the Torch reference (codex/gpt-6-astra:high, run 20261012T084037Z-cf9243b720), export and install into `tuned/thor/`, model-level verify and benchmark. The kernel applies to the captured B=1 pi05 shapes (576-token prefix, 50 action tokens) and falls back otherwise. Task timing per call (CUDA Graph, cold L2): prefix 1,079 us Torch / ~178 us existing Triton / 139 us tuned; denoise 304 / ~120 / 38 us. verify on 40 recorded inputs passed (worst ratio 0.80 of the limits). Jetson Thor (MAXN, jetson_clocks), Torch 2.13.0+cu132, Triton 3.7.1, pi05_libero_finetuned_v044, B=1, BF16, 10 denoise steps, LIBERO-10 1,600 frames after 10 warmup calls, Inductor with prefix and full-loop CUDA Graphs: kernel_profile=none 98.8 ms mean (decode 49.0 ms), thor 87.6 ms (decode 37.2 ms). Generated profile packages are digest-pinned evidence, so Ruff skips them, as it does for the verbatim Wan VAE. The proposal records the measurement, implementation status and the follow-ups: shape-range tasks, automatic region selection and fusion regions. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Installed profile packages are pinned by the SHA-256 of their bytes in profile.json. With core.autocrlf a Windows checkout rewrote them with CRLF and the profile refused to load. Check them out with LF everywhere. Signed-off-by: 肖一涵 <23371301@buaa.edu.cn>
Kernel tuning can only install registry operators, and registering one is manual work today. Design a resumable agent-in-the-loop flow that takes a caller-selected operator or fusion region, has a coding agent write its Torch reference, register it and route the call sites through the registry, and accepts the change only when controller-owned gates pass: diff scope, structure, ruff and tests, an unchanged eager ATen sequence, registry hits per call site, and kernel_profile=none outputs bit-identical to the base revision on every parity configuration. The result is a branch and evidence for human review; tuning stays in the 0009 flow. Signed-off-by: owpeter <pt909795@163.com>
The registration loop is developer tooling rather than a design proposal. Remove 0010 and its pointer from 0009; the tooling guide replaces it. Signed-off-by: owpeter <pt909795@163.com>
Specify the register tooling under scripts/kernel_tuning/registration: the caller selects an operator and its call sites, a coding agent writes the Torch reference, registers it, routes the call sites through the registry and adds tests, and CPU checks on scope, structure, ruff, tests and test effectiveness feed failures back until the change passes and a review branch is produced. GPU parity and tuning stay outside the loop. Signed-off-by: owpeter <pt909795@163.com>
Signed-off-by: owpeter <pt909795@163.com>
The registration agent's worked example came from `git show` of a hard-coded commit, which disappears from history under a squash or rebase merge. Copy the already-routed pi0.5 policy from BASE into the read-only context instead, and rewrite the registration README to match the current controller. Signed-off-by: owpeter <pt909795@163.com>
Tests now write the hand-written task package into a temporary directory, and the quick start generates a catalog task to check instead. Signed-off-by: owpeter <pt909795@163.com>
Keep the profile loader and install tooling; remove the committed profile, the Thor benchmark tables, run identity, and the relay note from 0009. Signed-off-by: owpeter <pt909795@163.com>
Signed-off-by: owpeter <pt909795@163.com>
Humanize2 fences each agent session with Linux Landlock, so tuning and registration cannot open a session on kernels built without it (for example Jetson L4T). Add --unfenced to run, tune-all and register run. Roles are declared with every scope; runs without the flag narrow the agent back to Humanize2's default permission, so fenced behavior is unchanged. The choice is frozen in the run or batch manifest, reused on resume and passed to every batch run, and a warning is printed before agents start. Signed-off-by: owpeter <pt909795@163.com>
Add a section on hosts without Landlock: how to check for it, where to pass --unfenced for run, tune-all and register run, how the choice is frozen across resume, the isolation an unfenced agent needs, and how a refused session shows up in run evidence. Signed-off-by: owpeter <pt909795@163.com>
capture-ops derived each registry case's output layout from the Torch reference, while downstream operators' inputs were recorded from the implementation the model actually ran. When the two disagree (for example the Triton rotate_qk returns a contiguous query while the reference keeps the input view layout), a tuned kernel follows the reference layout and the next tuned kernel's layout gate no longer matches in production. Record outputs, aliases and dtypes from the model's own call (run once, its result returned unchanged). The reference must agree on everything except strides and offsets; differing layouts are kept as reference_outputs and the generated baseline copies the reference result into the production layout. Signed-off-by: owpeter <pt909795@163.com>
Register a torch reference for linear and route the pi0.5 fused gate/up GEMM, MLP down projection and native SigLIP q/k/v, out, fc1 and fc2 projections through it. Produced by the automated registration run 20261014T134024Z-ed6d7ff626 (codex gpt-6-astra:xhigh, unfenced) and passed its C1-C5 CPU checks. Signed-off-by: owpeter <pt909795@163.com>
The registered linear route skipped the packed gate/up GEMM for any projection that is not exactly nn.Linear, so tensor-parallel ColumnParallelLinear shards fell back to two GEMMs. BASE packs every non-quantized inference pair regardless of module type; restore that and test it for an overriding nn.Linear subclass and ColumnParallelLinear. Signed-off-by: owpeter <pt909795@163.com>
Register a torch reference for gelu_tanh and route the native SigLIP MLP activation through it when the activation is exactly a tanh GELU. Produced by the automated registration run 20261014T144936Z-50adc4a729 (codex gpt-6-astra:xhigh, unfenced); it passed C1-C5 on attempt 2. Signed-off-by: owpeter <pt909795@163.com>
Transformers builds GELUTanh for both gelu_pytorch_tanh and gelu_python_tanh; the latter evaluates a Python formula that differs from F.gelu(approximate="tanh") in the last bit. Require the activation to be the native functools.partial before using the operator, mirror the real class in the test double, and check the installed Transformers directly. Signed-off-by: owpeter <pt909795@163.com>
Commit the agent-tuned split_kv_attention, linear (prefix down_proj) and rotate_qk kernels installed from their evaluator-verified exports. rotate_qk composes separately tuned prefix and decode variants with disjoint verified layouts; the decode variant was tuned against the production output layout. Record the end-to-end measurement in proposal 0008: on Jetson AGX Orin with pi05_base, B=1, mixed FP32/BF16, eager prefix and a full denoise-loop CUDA Graph, mean latency is 537.8 ms against 609.8 ms for the existing Triton path and 825.5 ms for Torch. kernel_profile still defaults to none. Signed-off-by: owpeter <pt909795@163.com>
Signed-off-by: owpeter <pt909795@163.com>
Merging main left the compiled-helper warmup referencing the RoPE and tanh-GELU helpers that now resolve through registry operators, and made `kernels` a required argument of `_fused_linear_pair` that main's callers and its packed-weight benchmark wrapper do not pass. Default `kernels` to the unprofiled selection, forward keyword arguments in the benchmark wrapper, and give the registry tests' policy stubs main's `_get_optimizations` hook. Signed-off-by: owpeter <pt909795@163.com>
The registration tests register `linear` as a worked example against the current checkout, but the routed pi05 projections now register a real `linear` operator and `tests/test_registry_linear.py`. Name the example `matmul_t` so it remains a genuine registry gap. Signed-off-by: owpeter <pt909795@163.com>
Stock Torch 2.10 Kineto events do not expose activity_type(), which the hotspot attribution needs to separate launches from device kernels. Raise an explicit error instead of an AttributeError, and test both behaviours. Signed-off-by: owpeter <pt909795@163.com>
Remove the obsolete capture --model path, ATen/RNG recording and replay branches, and their tests. Keep registry capture-ops, catalog capture, shared tensor/layout helpers, and the tuning preflight and evaluator. Update regression coverage and the canonical workflow documentation. Legacy full-discovery artifacts must be recaptured with capture-ops; registry captures need recalibration when numerical-source hashes change. Signed-off-by: owpeter <pt909795@163.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Type of change
Verification
Paste the exact commands you ran and their result. State which checks you did
not run (CUDA, real weights) rather than implying they pass.
Engine invariants
through
ActionDecoder/RLDecoderand the shared metadata.intentional difference is documented with numbers.
uv.lockis consistent.use the versioned API.
Checklist
commit policy,
and every author/coauthor has supplied their own DCO
Signed-off-by.CONTRIBUTING.mdand have a design document under
docs/proposals/.docs/en/benchmark.mdare updated when they change.
addresses, private paths, or machine-specific scripts are included.
CONTRIBUTING.md.