Activity

1. shoumikhin

   added

   module: cudaIssues related to the AOTI CUDA backendIssues related to the AOTI CUDA backend

   on Nov 5, 2026
2. shoumikhin commented on Nov 5, 2026

   

   shoumikhin

   on Nov 5, 2026

   ContributorAuthor

   More actions

   Repro pack for the CUDA backend failures in #24684

   Every open "needs an owner" or "needs a decision" row in #24684, and every ❌ cell, is below with the real error and a way to reproduce it.

   The CUDA backend rows have a small standalone script (under 110 lines each). It needs only torch and the CUDA build of ExecuTorch, and runs in under a minute. Where a script cannot show the problem, there is the LeRobot export command instead.

   Summary

   #	Failure	Policies	Repro	Fixed by
   1	float32 matmuls and convolutions are slower than eager PyTorch (Triton only)	act, diffusion, fineart_vla, pi0_fast; xvla in float32	standalone	needs an owner
   2	bf16 F.grouped_mm has no kernel choice on the A100 (sm_80)	cosmos3 (the recipe works around it)	standalone	needs an owner
   3	The flux3 export runs out of GPU memory during autotuning on the 80 GB A100	flux3	export command	needs an owner
   4	Scatter with a scalar source in deterministic mode	none yet	standalone	needs an owner
   5	wall_x and lingbot_vla_v2 checks on the A100 sit at their limit	wall_x, lingbot_vla_v2	export command	needs a decision (LeRobot)
   6	lingbot_vla_v2 does not export on the current LeRobot branch	lingbot_vla_v2	export command	in progress (LeRobot)
   7	torch.export cannot save the program (flat_apply)	dm05, wall_x	export command	pytorch/pytorch#202785
   8	torch.export cannot load the program (wrap_with_autocast)	lawam, molmoact2	export command	pytorch/pytorch#202925

   Policy to failure: act, diffusion, fineart_vla, pi0_fast: 1 (slow, not failing). flux3 (A100): 3. wall_x (A100): 5, and 7 on stock torch. lingbot_vla_v2: 6 on the current branch, 5 on the A100. dm05: 7. lawam, molmoact2: 8.

   1 and 2 have the same root: the backend asks Inductor for Triton kernels only (max_autotune_gemm_backends="TRITON" and max_autotune_conv_backends="TRITON" in CudaBackend.get_aoti_compile_options), so a program never needs libtorch. Its only non-Triton kernels are the fallback shims in get_supported_fallback_kernels (int4 matmul, sort, index, scatter and a few more), and none of them is a matmul or a convolution.

   Setup

   Versions: ExecuTorch 1.6.0 rc3 (the CUDA 13.2 wheel from the PyTorch test index), torch 2.15.0, Python 3.12, Linux x86_64, on an H100 (sm_90) and an A100 80 GB (sm_80).

   pip install torch==2.15.0 executorch==1.6.0 --index-url https://download.pytorch.org/whl/test/cu132
   export CUDA_HOME=<CUDA 13.2 toolkit>  PATH=$CUDA_HOME/bin:$PATH   # the export compiles with nvcc
   cd <a folder with no executorch checkout>
   python <script>.py

   The test index now serves 1.6.0 rc4 under the same version. The scripts were run on rc3.

   LeRobot export commands

   The export commands use the lerobot-export-stack branch of shoumikhin/lerobot (see #24353) as it was on Nov 3, pinned by the export-stack-2026-11-03 tag, and transformers 5.22.1:

   pip install "lerobot @ git+https://github.com/shoumikhin/lerobot@export-stack-2026-11-03" transformers==5.22.1
   lerobot-export --backend.type=executorch_cuda --policy.path=<checkpoint> --output_dir=<out> [--task="<task>"]

   Some rows use a randomly initialized checkpoint, since they test the compiler, not the policy. make_random.py builds one:

   python make_random.py eo1 <ckpt>/random_eo1

   make_random.py

   """Save a random-init checkpoint (policy + processors) for a LeRobot policy type.
   
   usage: python make_random.py <policy_type> <out_dir> [key=json ...]
   2 cameras of 224x224 and a 14-D arm unless the policy needs otherwise; identity normalization; seed 0.
   These are random weights: they test the compiler, not a trained policy.
   """
   import json
   import sys
   from pathlib import Path
   
   import torch
   
   from lerobot.configs.types import FeatureType, NormalizationMode, PolicyFeature
   from lerobot.policies.factory import get_policy_class, make_policy_config, make_pre_post_processors
   
   name, out = sys.argv[1], Path(sys.argv[2])
   overrides = {k: json.loads(v) for k, v in (a.split("=", 1) for a in sys.argv[3:])}
   torch.manual_seed(0)
   cfg = make_policy_config(name, device="cpu")
   V = lambda c, h, w: PolicyFeature(type=FeatureType.VISUAL, shape=(c, h, w))  # noqa: E731
   S = lambda d: PolicyFeature(type=FeatureType.STATE, shape=(d,))  # noqa: E731
   cams = {"observation.images.top": V(3, 224, 224), "observation.images.wrist": V(3, 224, 224)}
   state, act = 14, 14
   if name == "tdmpc":
       cams = {"observation.image": V(3, 84, 84)}
   elif name in ("vqbet", "gaussian_actor"):
       cams = {"observation.images.top": V(3, 96, 96)}
   elif name == "multi_task_dit":
       cams = {"observation.images.top": V(3, 224, 224)}
   elif name == "lingbot_vla_v2":
       state, act = 55, 55
   elif name == "wall_oss_05" and getattr(cfg, "camera_key_mapping", None):
       cams = {k: V(3, 224, 224) for k in cfg.camera_key_mapping}
   if hasattr(cfg, "primary_image_features"):
       cfg.primary_image_features = ["observation.images.top"]
       if hasattr(cfg, "wrist_image_features"):
           cfg.wrist_image_features = ["observation.images.wrist"]
   if not cfg.input_features:
       cfg.input_features = {**cams, "observation.state": S(state)}
   if not cfg.output_features:
       cfg.output_features = {"action": PolicyFeature(type=FeatureType.ACTION, shape=(act,))}
   if name != "tdmpc":  # TD-MPC's config refuses anything but its own action normalization.
       cfg.normalization_mapping = {k: NormalizationMode.IDENTITY for k in ("VISUAL", "STATE", "ACTION", "ENV")}
   for k, v in overrides.items():
       setattr(cfg, k, v)
   policy = get_policy_class(name)(cfg)
   if name == "tdmpc":
       # TD-MPC zero-initializes its reward and Q output layers, so every planned trajectory would score exactly 0.
       for layer in (policy.model._reward[-1], *(q[-1] for q in policy.model._Qs)):
           layer.reset_parameters()
   policy.save_pretrained(out)
   pre, post = make_pre_post_processors(cfg, dataset_stats=None)
   pre.save_pretrained(out)
   post.save_pretrained(out)
   print(json.dumps({"saved": str(out), "params_M": round(sum(p.numel() for p in policy.parameters()) / 1e6, 1)}))

   ---

   1. float32 matmuls and convolutions are slower than eager PyTorch (act, diffusion, fineart_vla, pi0_fast, xvla in float32; needs an owner)

   In float32 (TF32 off on both sides), the Triton kernels that the backend picks are about 2x slower than the cuBLAS and cuDNN kernels that eager PyTorch calls. In the policies this makes float32 bundles slower than eager: xvla in float32 runs at 359 ms per chunk against 105 ms eager on the H100 (742 against 200 on the A100), act at 0.8x, diffusion 0.8x and fineart_vla 0.7x on the H100. In xvla's kernel trace on the H100, one Triton convolution (triton_tem_fused_convolution) takes 3.0 ms per call and 30% of the GPU time, and the next 4 kernels are Triton matmul templates.

   Repro (fp32_speed.py): 4 convolutions, and 2 linear layers, exported and run with inputs and outputs on the GPU, timed against eager PyTorch.

   """Float32 convolutions and matmuls in an ExecuTorch CUDA program against eager PyTorch.
   
   Needs torch 2.15 and the CUDA build of ExecuTorch 1.6. Run from a folder that has no `executorch` checkout:
   
       python fp32_speed.py --out /tmp/fp32_speed
   
   TF32 is off on both sides, so both compute in strict float32. Inputs and outputs
   stay on the GPU on both sides, so the times compare only the kernels.
   """
   
   import argparse
   import os
   
   # ExecuTorch's own result type, so outputs stay on the GPU; read once when the bindings load.
   os.environ["EXECUTORCH_PYBINDINGS_TENSOR_OUTPUT"] = "executorch"
   import statistics
   import time
   
   import numpy as np
   import torch
   from torch import nn
   
   
   class Convs(nn.Module):
       def __init__(self):
           super().__init__()
           self.net = nn.Sequential(
               nn.Conv2d(3, 64, 7, stride=2, padding=3), nn.ReLU(),
               nn.Conv2d(64, 128, 3, padding=1), nn.ReLU(),
               nn.Conv2d(128, 256, 3, stride=2, padding=1), nn.ReLU(),
               nn.Conv2d(256, 256, 3, padding=1), nn.ReLU(),
           )
   
       def forward(self, x):
           return self.net(x)
   
   
   class Linears(nn.Module):
       def __init__(self):
           super().__init__()
           self.net = nn.Sequential(nn.Linear(1024, 4096), nn.GELU(), nn.Linear(4096, 1024))
   
       def forward(self, x):
           return self.net(x)
   
   
   def median_ms(fn, n=30, warm=5):
       for _ in range(warm):
           fn()
       times = []
       for _ in range(n):
           torch.cuda.synchronize()
           t = time.perf_counter()
           fn()
           torch.cuda.synchronize()
           times.append((time.perf_counter() - t) * 1e3)
       return statistics.median(times)
   
   
   def main():
       p = argparse.ArgumentParser()
       p.add_argument("--out", required=True)
       args = p.parse_args()
       os.makedirs(args.out, exist_ok=True)
       torch.backends.cuda.matmul.allow_tf32 = False
       torch.backends.cudnn.allow_tf32 = False
   
       from executorch.backends.cuda.cuda_backend import CudaBackend
       from executorch.backends.cuda.cuda_partitioner import CudaPartitioner
       from executorch.exir import ExecutorchBackendConfig, to_edge_transform_and_lower
       from executorch.exir.passes.memory_planning_pass import MemoryPlanningPass
       from executorch.exir.passes.propagate_device_pass import PropagateDeviceConfig
       from executorch.runtime import ExecuTorchResult, Runtime
   
       on_gpu = ExecutorchBackendConfig(
           propagate_device_config=PropagateDeviceConfig(skip_h2d_for_method_inputs=True, skip_d2h_for_method_outputs=True),
           enable_non_cpu_memory_planning=True,
           memory_planning_pass=MemoryPlanningPass(alloc_graph_input=False, alloc_graph_output=False),
       )
   
       cases = [("convs", Convs, (1, 3, 224, 224)), ("linears", Linears, (512, 1024))]
       for name, cls, shape in cases:
           for dtype in (torch.float32,):
               torch.manual_seed(0)
               model = cls().eval().to("cuda", dtype)
               x = torch.randn(shape, device="cuda", dtype=dtype)
               with torch.no_grad():
                   ep = torch.export.export(model, (x,))
                   tag = f"{name}_{str(dtype).split('.')[-1]}"
                   out = os.path.join(args.out, tag)
                   os.makedirs(out, exist_ok=True)
                   part = CudaPartitioner([CudaBackend.generate_method_name_compile_spec("forward")])
                   prog = to_edge_transform_and_lower(ep, partitioner=[part]).to_executorch(on_gpu)
                   pte = os.path.join(out, "model.pte")
                   with open(pte, "wb") as f:
                       prog.write_to_file(f)
                   prog.write_tensor_data_to_file(out)
                   ptd = os.path.join(out, "aoti_cuda_blob.ptd")
                   method = Runtime.get().load_program(pte, data_path=ptd).load_method("forward")
                   xg = ExecuTorchResult.from_host(x.cpu().numpy(), 0)
                   et_out = np.asarray(method.execute([xg])[0].to_host())
                   err = (torch.from_numpy(et_out) - model(x).cpu()).abs().max().item()
                   eager = median_ms(lambda: model(x))
                   et = median_ms(lambda: method.execute([xg]))
               print(f"{tag:18s} eager {eager:8.3f} ms   executorch_cuda {et:8.3f} ms   "
                     f"eager/et {eager / et:5.2f}x   max abs diff {err:.2e}", flush=True)
   
   
   if __name__ == "__main__":
       main()

   Output:

   H100:
   convs_float32      eager    0.625 ms   executorch_cuda    1.119 ms   eager/et  0.56x   max abs diff 2.61e-07
   linears_float32    eager    0.312 ms   executorch_cuda    0.687 ms   eager/et  0.45x   max abs diff 0.00e+00
   A100:
   convs_float32      eager    0.727 ms   executorch_cuda    1.286 ms   eager/et  0.57x   max abs diff 2.53e-07
   linears_float32    eager    0.878 ms   executorch_cuda    1.595 ms   eager/et  0.55x   max abs diff 2.62e-06
   

   2. bf16 F.grouped_mm has no kernel choice on the A100 (cosmos3; needs an owner)

   On the A100, Inductor has no Triton template for aten._grouped_mm in bf16, and the backend allows no other kind, so the lowering stops. On the H100 the same script lowers. The cosmos3 export recipe works around it by running each step with the fixed domain's projection (x @ W[domain] + b).

   Error in the policy (cosmos3 on the A100, before the workaround):

   torch._inductor.exc.InductorError: LoweringException: NoValidChoicesError: No choices to select. Provided reason: No choices exist for backend. please consider adding ATEN into max_autotune_gemm_backends config (defined in torch/_inductor/config.py) to allow at least one choice.
   

   Repro (grouped_mm_bf16.py):

   """bf16 F.grouped_mm lowered with ExecuTorch's CUDA backend.
   
   Needs torch 2.15 and the CUDA build of ExecuTorch 1.6. On an A100 (sm_80) the lowering fails with
   NoValidChoicesError; run it on an H100 for comparison. Run from a folder that has no `executorch` checkout:
   
       python grouped_mm_bf16.py
   """
   
   import torch
   import torch.nn.functional as F
   from torch import nn
   
   
   class GroupedLinear(nn.Module):
       def __init__(self, groups=4, k=256, n=512):
           super().__init__()
           self.w = nn.Parameter(torch.randn(groups, k, n) * 0.02)
           self.register_buffer("offs", torch.tensor([16, 32, 48, 64], dtype=torch.int32))
   
       def forward(self, x):
           return F.grouped_mm(x, self.w, offs=self.offs)
   
   
   def main():
       import sys
       del sys.argv[1:]  # the runner passes --out, which this script does not need
       from executorch.backends.cuda.cuda_backend import CudaBackend
       from executorch.backends.cuda.cuda_partitioner import CudaPartitioner
       from executorch.exir import EdgeCompileConfig, to_edge_transform_and_lower
   
       print(torch.__version__, torch.cuda.get_device_name(), torch.cuda.get_device_capability(), flush=True)
       model = GroupedLinear().to("cuda", torch.bfloat16).eval()
       x = torch.randn(64, 256, device="cuda", dtype=torch.bfloat16)
       with torch.no_grad():
           print("eager ok", tuple(model(x).shape), flush=True)
           ep = torch.export.export(model, (x,))
           part = CudaPartitioner([CudaBackend.generate_method_name_compile_spec("forward")])
           to_edge_transform_and_lower(
               ep, partitioner=[part], compile_config=EdgeCompileConfig(_check_ir_validity=False)
           ).to_executorch()
       print("lowered ok", flush=True)
   
   
   if __name__ == "__main__":
       main()

   Output:

   2.15.0+cu132 NVIDIA PG509-210 (8, 0)      # A100
   eager ok (64, 512)
   torch._inductor.exc.InductorError: LoweringException: NoValidChoicesError: No choices to select. Provided reason: No choices exist for backend. please consider adding ATEN into max_autotune_gemm_backends config (defined in torch/_inductor/config.py) to allow at least one choice.
   
   2.15.0+cu132 NVIDIA H100 (9, 0)
   eager ok (64, 512)
   lowered ok
   

   3. The flux3 export runs out of GPU memory during autotuning on the 80 GB A100 (flux3; needs an owner)

   The export of black-forest-labs/flux-3-action-droid (bf16) peaks at 85,320 MiB of GPU memory, during AOTInductor's autotuning. That is more than the A100's 80 GB. On the H100 (95 GB) it exports, and the finished program then runs in 61,694 MiB. The A100 export failed after 86 minutes:

   torch._inductor.exc.InductorError: RuntimeError: Failed to run autotuning code block: CUDA out of memory. Tried to allocate 2.98 GiB. GPU 0 has a total capacity of 79.25 GiB of which 1.50 GiB is free. Including non-PyTorch memory, this process has 77.74 GiB memory in use. Of the allocated memory 75.79 GiB is allocated by PyTorch, and 306.98 MiB is reserved by PyTorch but unallocated.
   error: The compile-model step failed (see above), so the bundle was not written.
   

   Export command:

   lerobot-export --backend.type=executorch_cuda --policy.path=black-forest-labs/flux-3-action-droid --output_dir=<out> --task="pick up the cube"

   4. Scatter with a scalar source in deterministic mode (no policy yet; needs an owner)

   With torch.use_deterministic_algorithms(True), Inductor does not generate a Triton kernel for scatter with a scalar source; it calls the ATen kernel instead, and the backend has no shim for it. The backend has one for a tensor source (aoti_torch_cuda_scatter_src_out), not for a scalar one. With deterministic mode off, the same model lowers. No policy in the table hits it.

   Repro (scatter_value_deterministic.py):

   """Scatter with a scalar source, lowered with ExecuTorch's CUDA backend while deterministic algorithms are on.
   
   Needs torch 2.15 and the CUDA build of ExecuTorch 1.6. Run from a folder that has no `executorch` checkout:
   
       python scatter_value_deterministic.py            # deterministic mode on: lowering fails
       python scatter_value_deterministic.py --off      # deterministic mode off: lowers
   """
   
   import sys
   
   import torch
   from torch import nn
   
   
   class ScatterValue(nn.Module):
       def forward(self, x, index):
           return x.scatter(1, index, 1.0)
   
   
   def main():
       from executorch.backends.cuda.cuda_backend import CudaBackend
       from executorch.backends.cuda.cuda_partitioner import CudaPartitioner
       from executorch.exir import to_edge_transform_and_lower
   
       deterministic = "--off" not in sys.argv
       torch.use_deterministic_algorithms(deterministic)
       print(torch.__version__, torch.cuda.get_device_name(), "deterministic", deterministic, flush=True)
       x = torch.zeros(4, 16, device="cuda")
       index = torch.tensor([[0, 3], [1, 4], [2, 5], [7, 9]], device="cuda")
       with torch.no_grad():
           ep = torch.export.export(ScatterValue(), (x, index))
           part = CudaPartitioner([CudaBackend.generate_method_name_compile_spec("forward")])
           to_edge_transform_and_lower(ep, partitioner=[part]).to_executorch()
       print("lowered ok", flush=True)
   
   
   if __name__ == "__main__":
       main()

   Output:

   2.15.0+cu132 NVIDIA H100 deterministic True
   RuntimeError: Method forward missing fallback kernels (1 total):
     - aoti_torch_cuda_scatter_value_out
   Please add them to the AOTI backend.
   
   2.15.0+cu132 NVIDIA H100 deterministic False      # with --off
   lowered ok
   

   5. wall_x and lingbot_vla_v2 checks on the A100 sit at their limit (needs a decision, LeRobot)

   2 exports of the same random weights on the A100 gave 0.095 and 0.047 against wall_x's limit of 0.075. The policy runs 10 bf16 flow steps, which LeRobot's rule gives a limit of 0.2 for other policies (pi0, pi0.5, eo1, wall_oss_05, dm05, fineart_vla, molmoact2). On the H100 it passes (0.022).

   lingbot_vla_v2 (random weights, the earlier branch head from 6) gives 0.078 on the A100 against the same 0.075 limit, and 0.0625 on the H100. This was not narrowed further, so the backend's bf16 kernels are also a candidate; eager PyTorch on the 2 GPUs was not compared.

   RuntimeError: The exported policy does not reproduce its test case (9.53e-02 > 0.075). Export it again on this device.
   

   Export commands for wall_x (needs torch with pytorch/pytorch#202785, see 7); lingbot_vla_v2 uses the commands in 6:

   python make_random.py wall_x <ckpt>/random_wall_x
   lerobot-export --backend.type=executorch_cuda --policy.path=<ckpt>/random_wall_x --output_dir=<out> --task="pick up the cube"

   6. lingbot_vla_v2 does not export on the current LeRobot branch (lingbot_vla_v2; in progress, LeRobot)

   On the branch as of Nov 3 at 10:53 AM Pacific, the export stops before tracing. The cells use an earlier head of the branch, with transformers 5.5.4, as #24353 does.

   ValueError: Exporting LingBot-VLA v2 needs a vision_grid (see `predict_action_chunk`).
   error: The trace step failed (see above), so the bundle was not written.
   

   Export commands:

   python make_random.py lingbot_vla_v2 <ckpt>/random_lingbot_vla_v2
   lerobot-export --backend.type=executorch_cuda --policy.path=<ckpt>/random_lingbot_vla_v2 --output_dir=<out> --task="pick up the cube" --policy.precompute_grid_thw=true

   7. torch.export cannot save the program (dm05, wall_x)

   SerializeError: Unsupported getattr attribute function_const_func_spec0 with type TreeSpec
   

   Fixed by pytorch/pytorch#202785 (not in torch 2.15). The yellow cells use torch 2.15.0 with it applied.

   python make_random.py dm05 <ckpt>/random_dm05
   lerobot-export --backend.type=executorch_cuda --policy.path=<ckpt>/random_dm05 --output_dir=<out> --task="pick up the cube"

   8. torch.export cannot load the program (lawam, molmoact2)

   Node.meta wrap_with_autocast_N is missing val field
   

   Fixed by pytorch/pytorch#202925 (not in torch 2.15).

   lerobot-export --backend.type=executorch_cuda --policy.path=jialei02/lawam-libero-sft-lerobot --output_dir=<out> --task="pick up the cube"