[SPIR-V] Add descriptor heap -fvk-resource-heap-stride / -fvk-sampler-heap-stride CLI flags

docs/ReleaseNotes.md

@@ -19,6 +19,14 @@ The included licenses apply to the following files:
 
 ### Upcoming Release
 
+#### SPIR-V
+
+- Added `-fvk-resource-heap-stride` and `-fvk-sampler-heap-stride`, which
+  override the computed `ArrayStrideIdEXT` of the resource and sampler
+  descriptor heaps with a literal `ArrayStride`. The value must be a power of
+  two in `[8, 256]`. Both flags require `-spirv` and
+  `-fspv-use-descriptor-heap` and are rejected otherwise.
+
 Place release notes for the upcoming release below this line and remove this
 line upon naming the release. Refer to previous for appropriate section names.
 

docs/SPIR-V.rst

@@ -2102,9 +2102,11 @@ objects as untyped variables in ``UniformConstant`` storage class:
 The concrete descriptor type is selected at each heap access. For image,
 sampler, and texel buffer resources, DXC forms a runtime array of that
 descriptor type, decorates the array with a byte stride, and uses
-``OpUntypedAccessChainKHR`` followed by ``OpLoad``. The stride is an
-``ArrayStrideIdEXT`` decoration referencing a specialization constant rather
-than a literal ``ArrayStride`` (see `Descriptor heap array stride`_ below):
+``OpUntypedAccessChainKHR`` followed by ``OpLoad``. By default the stride is an
+``ArrayStrideIdEXT`` decoration referencing a constant built from
+``OpConstantSizeOfEXT``; a literal ``ArrayStride`` is emitted only when the
+stride is overridden on the command line (see `Descriptor heap array stride`_
+below):
 
 .. code:: spirv
 
@@ -2224,10 +2226,10 @@ Three further restrictions on heap accesses produce diagnostics:
 Descriptor heap array stride
 ++++++++++++++++++++++++++++
 
-All resource heap runtime arrays share a single ``ArrayStrideIdEXT`` decoration
-rather than a literal ``ArrayStride``, because descriptor sizes are not known
-until pipeline creation. The shared value is built from ``OpConstantSizeOfEXT``
-and ``OpSpecConstantOp`` and evaluates to
+By default, all resource heap runtime arrays share a single ``ArrayStrideIdEXT``
+decoration rather than a literal ``ArrayStride``, because descriptor sizes are
+not known until pipeline creation. The shared value is built from
+``OpConstantSizeOfEXT`` and ``OpSpecConstantOp`` and evaluates to
 ``max(sizeof(image_descriptor), sizeof(buffer_descriptor))``. The sampler heap
 carries its own ``ArrayStrideIdEXT`` equal to ``sizeof(sampler_descriptor)``.
 
@@ -2238,9 +2240,10 @@ types the shader actually uses; a module will normally contain both the
 placeholder type and the distinct image types its heap accesses lower to.
 
 The stride is a property of the heap layout, not of the shader's loads. DXC
-decides it before translating any function, from the shader stage and the
-``-fspv-extension`` list. When acceleration structure descriptors may be on the
-resource heap, the formula expands to a three-way max
+decides it before translating any function, either from explicit user input
+(``-fvk-resource-heap-stride`` and ``-fvk-sampler-heap-stride``, see below) or
+by calculating it from the shader stage and the ``-fspv-extension`` list. When
+acceleration structure descriptors may be on the resource heap, the formula expands to a three-way max
 ``max(max(sizeof(image_descriptor), sizeof(buffer_descriptor)), sizeof(acceleration_structure))``.
 A ray-tracing shader that only heap-loads a texture still gets the three-way
 max. DXC adds the acceleration structure term when either
@@ -2256,8 +2259,29 @@ of the listed extension (``RayQueryKHR``, ``RayTracingKHR`` or
 ``RayTracingNV``); if several are listed, ``SPV_KHR_ray_query`` is preferred,
 then ``SPV_KHR_ray_tracing``, then ``SPV_NV_ray_tracing``. A shader that is not
 a ray-tracing stage and heap-loads a ``RaytracingAccelerationStructure``
-without such an extension is rejected, because its stride would not cover the
-acceleration structure descriptor.
+without such an extension or ``-fvk-resource-heap-stride`` is rejected, because
+its stride would not cover the acceleration structure descriptor.
+
+The computed stride can be replaced with a literal using
+``-fvk-resource-heap-stride <N>`` and ``-fvk-sampler-heap-stride <N>``. They
+emit ``OpDecorate <array> ArrayStride N`` on every runtime array of the resource
+heap and of the sampler heap respectively. ``N`` must be a power of two in the
+inclusive range [8, 256]. Both flags require ``-spirv`` and
+``-fspv-use-descriptor-heap``, and are rejected otherwise because they would
+have no effect.
+
+For a heap with an override, no ``ArrayStrideIdEXT`` is emitted and no
+``OpConstantSizeOfEXT`` is built. The flags are independent: the other heap
+keeps its computed stride.
+
+The literal is not validated against the descriptor sizes of the target
+implementation. It must match the stride at which the application lays out
+descriptors in the heap, and be at least the size of the largest descriptor
+that may appear in it; otherwise heap indices select the wrong descriptors or
+read out of bounds.
+``-fvk-resource-heap-stride`` also disables the acceleration structure check
+above, so the stride must cover acceleration structure descriptors when the
+shader loads them from the heap.
 
 HLSL Expressions
 ================

include/dxc/Support/HLSLOptions.td

@@ -454,6 +454,10 @@ def fvk_bind_sampler_heap : MultiArg<["-"], "fvk-bind-sampler-heap", 2>, MetaVar
   HelpText<"Specify Vulkan binding number and set number for the sampler heap.">;
 def fvk_bind_counter_heap : MultiArg<["-"], "fvk-bind-counter-heap", 2>, MetaVarName<"<binding> <set>">, Group<spirv_Group>, Flags<[CoreOption, DriverOption]>,
   HelpText<"Specify Vulkan binding number and set number for the counter heap.">;
+def fvk_resource_heap_stride : Separate<["-"], "fvk-resource-heap-stride">, MetaVarName<"<stride>">, Group<spirv_Group>, Flags<[CoreOption, DriverOption]>,
+  HelpText<"Override the resource descriptor heap array stride with a literal ArrayStride of <stride> bytes. Requires -spirv and -fspv-use-descriptor-heap. Must be a power of 2 in [8, 256].">;
+def fvk_sampler_heap_stride : Separate<["-"], "fvk-sampler-heap-stride">, MetaVarName<"<stride>">, Group<spirv_Group>, Flags<[CoreOption, DriverOption]>,
+  HelpText<"Override the sampler descriptor heap array stride with a literal ArrayStride of <stride> bytes. Requires -spirv and -fspv-use-descriptor-heap. Must be a power of 2 in [8, 256].">;
 // SPIRV Change Ends
 
 //////////////////////////////////////////////////////////////////////////////

include/dxc/Support/SPIRVOptions.h

@@ -110,6 +110,12 @@ struct SpirvCodeGenOptions {
   std::optional<BindingInfo> samplerHeapBinding;
   std::optional<BindingInfo> counterHeapBinding;
 
+  // -fvk-resource-heap-stride / -fvk-sampler-heap-stride
+  // When set, every runtime array of that heap gets a literal ArrayStride of
+  // this many bytes (a power of 2 in [8, 256]) instead of ArrayStrideIdEXT.
+  std::optional<uint32_t> resourceHeapStride;
+  std::optional<uint32_t> samplerHeapStride;
+
   bool signaturePacking =
       false; ///< Whether signature packing is enabled or not
 

lib/DxcSupport/HLSLOptions.cpp

@@ -357,6 +357,52 @@ handleFixedBinding(const InputArgList &args, OptSpecifier id,
   return true;
 }
 
+// Parses the single-integer descriptor-heap stride flag |id| in |args|. If
+// present, requires -spirv and -fspv-use-descriptor-heap, validates that the
+// value is a power of 2 in [8, 256] and stores it in |stride|. Returns true on
+// success (including when the flag is absent). Returns false and writes to
+// |errors| if a requirement is not met or the value is malformed or invalid,
+// using |name| as the pretty flag name.
+static bool handleHeapStride(const InputArgList &args, OptSpecifier id,
+                             std::optional<uint32_t> *stride,
+                             llvm::StringRef name, llvm::raw_ostream &errors) {
+  Arg *arg = args.getLastArg(id);
+  if (!arg) {
+    *stride = std::nullopt;
+    return true;
+  }
+
+  if (!args.hasArg(OPT_spirv)) {
+    errors << name << " requires -spirv\n";
+    return false;
+  }
+
+  // The stride is only consumed by the native heap lowering. Without it the
+  // override would be accepted and silently have no effect.
+  if (!args.hasFlag(OPT_fspv_use_descriptor_heap, OPT_fspv_use_emulated_heap,
+                    false)) {
+    errors << name << " requires -fspv-use-descriptor-heap\n";
+    return false;
+  }
+
+  llvm::StringRef value = arg->getValue();
+  uint32_t number = 0;
+  if (value.getAsInteger(10, number)) {
+    errors << "invalid " << name << " argument: '" << value << "'\n";
+    return false;
+  }
+  // Power of 2 in [8, 256] inclusive.
+  if (number < 8 || number > 256 || (number & (number - 1)) != 0) {
+    errors << name
+           << " must be a power of 2 between 8 and 256 (inclusive); got "
+           << value << "\n";
+    return false;
+  }
+
+  *stride = number;
+  return true;
+}
+
 // Check if any options that are unsupported with SPIR-V are used.
 static bool hasUnsupportedSpirvOption(const InputArgList &args,
                                       llvm::raw_ostream &errors) {
@@ -1175,6 +1221,16 @@ int ReadDxcOpts(const OptTable *optionTable, unsigned flagsToInclude,
     return 1;
   }
 
+  bool strideOk = true;
+  strideOk &= handleHeapStride(Args, OPT_fvk_resource_heap_stride,
+                               &opts.SpirvOptions.resourceHeapStride,
+                               "-fvk-resource-heap-stride", errors);
+  strideOk &= handleHeapStride(Args, OPT_fvk_sampler_heap_stride,
+                               &opts.SpirvOptions.samplerHeapStride,
+                               "-fvk-sampler-heap-stride", errors);
+  if (!strideOk)
+    return 1;
+
   for (const Arg *A : Args.filtered(OPT_fspv_extension_EQ)) {
     opts.SpirvOptions.allowedExtensions.push_back(A->getValue());
   }
@@ -1316,7 +1372,9 @@ int ReadDxcOpts(const OptTable *optionTable, unsigned flagsToInclude,
       !Args.getLastArgValue(OPT_fvk_u_shift).empty() ||
       !Args.getLastArgValue(OPT_fvk_bind_resource_heap).empty() ||
       !Args.getLastArgValue(OPT_fvk_bind_sampler_heap).empty() ||
-      !Args.getLastArgValue(OPT_fvk_bind_counter_heap).empty()) {
+      !Args.getLastArgValue(OPT_fvk_bind_counter_heap).empty() ||
+      !Args.getLastArgValue(OPT_fvk_resource_heap_stride).empty() ||
+      !Args.getLastArgValue(OPT_fvk_sampler_heap_stride).empty()) {
     errors << "SPIR-V CodeGen not available. "
               "Please recompile with -DENABLE_SPIRV_CODEGEN=ON.";
     return 1;

tools/clang/lib/SPIRV/SpirvEmitter.cpp

@@ -5657,8 +5657,9 @@ SpirvInstruction *SpirvEmitter::emitDescriptorHeapBufferAccess(
 
   const BufferEXTType *bufferDescriptorType =
       spvContext.getBufferEXTType(bufferSC);
-  const SpirvType *arrayType =
-      getDescriptorHeapRuntimeArrayType(bufferDescriptorType);
+  // Buffer descriptors are always on the resource heap.
+  const SpirvType *arrayType = getDescriptorHeapRuntimeArrayType(
+      bufferDescriptorType, /*onSamplerHeap=*/false);
   SpirvUntypedAccessChainKHR *untypedAccessChainPtr =
       spvBuilder.createUntypedAccessChainKHR(untypedUniformConstantType,
                                              arrayType, heapVar, index,
@@ -7357,13 +7358,18 @@ SpirvEmitter::doCXXOperatorCallExpr(const CXXOperatorCallExpr *expr,
         // The stride was decided without an acceleration structure term (not a
         // ray-tracing stage, no ray extension listed). Reject the load: the
         // heap would be indexed with a stride that may be too narrow.
+        // Exception: with -fvk-resource-heap-stride the stride is a literal
+        // chosen by the user, so there is no computed stride to widen.
         if (isRaytracingAccelerationStructure(resourceType) &&
-            !spvBuilder.resourceHeapStrideIncludesAccelStruct()) {
+            !spvBuilder.resourceHeapStrideIncludesAccelStruct() &&
+            !spirvOptions.resourceHeapStride.has_value()) {
           emitError("acceleration structure loaded from ResourceDescriptorHeap "
                     "requires the resource heap stride to account for "
                     "acceleration structure descriptors; compile with "
-                    "-fspv-extension=SPV_KHR_ray_tracing or "
-                    "-fspv-extension=SPV_KHR_ray_query",
+                    "-fspv-extension=SPV_KHR_ray_tracing, "
+                    "-fspv-extension=SPV_NV_ray_tracing, "
+                    "-fspv-extension=SPV_KHR_ray_query, or "
+                    "-fvk-resource-heap-stride",
                     expr->getExprLoc());
           return nullptr;
         }
@@ -7381,8 +7387,9 @@ SpirvEmitter::doCXXOperatorCallExpr(const CXXOperatorCallExpr *expr,
         const SpirvType *handleType =
             lowerTypeVisitor.lowerType(resourceType, SpirvLayoutRule::Void,
                                        llvm::None, baseExpr->getExprLoc());
-        const SpirvType *arrayType =
-            getDescriptorHeapRuntimeArrayType(handleType);
+        // The stride belongs to the heap being indexed.
+        const SpirvType *arrayType = getDescriptorHeapRuntimeArrayType(
+            handleType, isSamplerDescriptorHeap(decl));
         SpirvUntypedAccessChainKHR *untypedAccessChainPtr =
             spvBuilder.createUntypedAccessChainKHR(untypedUniformConstantType,
                                                    arrayType, var, index,
@@ -9829,14 +9836,24 @@ void SpirvEmitter::createSpecConstant(const VarDecl *varDecl) {
 }
 
 const SpirvType *
-SpirvEmitter::getDescriptorHeapRuntimeArrayType(const SpirvType *elemType) {
+SpirvEmitter::getDescriptorHeapRuntimeArrayType(const SpirvType *elemType,
+                                                bool onSamplerHeap) {
+  // -fvk-{resource,sampler}-heap-stride replaces the computed stride: the array
+  // carries a literal ArrayStride and no ArrayStrideIdEXT.
+  const std::optional<uint32_t> &cliStride =
+      onSamplerHeap ? spirvOptions.samplerHeapStride
+                    : spirvOptions.resourceHeapStride;
+  if (cliStride.has_value())
+    return spvContext.getRuntimeArrayType(elemType, *cliStride);
+
   // Descriptor sizes are defined by the client API, so the stride is an
   // ArrayStrideIdEXT operand rather than a literal. The sampler heap holds one
   // descriptor type. The resource heap is a flat array in which any resource
   // descriptor may sit at any slot, so all its runtime arrays share one stride
   // (see SpirvBuilder::getResourceHeapArrayStride) instead of the stride of the
-  // accessed type.
-  SpirvInstruction *strideId = isa<SamplerType>(elemType)
+  // accessed type. As with the literal override above, the stride follows the
+  // heap that is accessed, not the element type.
+  SpirvInstruction *strideId = onSamplerHeap
                                    ? spvBuilder.getSamplerHeapArrayStride()
                                    : spvBuilder.getResourceHeapArrayStride();
   return spvContext.getRuntimeArrayType(elemType, llvm::None, strideId);

tools/clang/lib/SPIRV/SpirvEmitter.h

@@ -439,10 +439,13 @@ class SpirvEmitter : public ASTConsumer {
   /// Translates the given varDecl into a spec constant.
   void createSpecConstant(const VarDecl *varDecl);
 
-  /// Returns the OpTypeRuntimeArray for a descriptor heap array of elemType,
-  /// decorated with ArrayStrideIdEXT referencing an OpConstantSizeOfEXT of the
-  /// element (descriptor) type.
-  const SpirvType *getDescriptorHeapRuntimeArrayType(const SpirvType *elemType);
+  /// Returns the OpTypeRuntimeArray for a descriptor heap array of elemType.
+  /// onSamplerHeap selects which heap the descriptor is loaded from. If that
+  /// heap has a -fvk-{resource,sampler}-heap-stride override, the array carries
+  /// that literal ArrayStride; otherwise it is decorated with ArrayStrideIdEXT
+  /// referencing the shared stride for the heap.
+  const SpirvType *getDescriptorHeapRuntimeArrayType(const SpirvType *elemType,
+                                                     bool onSamplerHeap);
 
   /// Emits the descriptor-heap buffer access for a buffer-like resource
   /// (StructuredBuffer/ByteAddressBuffer/ConstantBuffer/TextureBuffer + RW

tools/clang/test/CodeGenSPIRV/sm6_6.descriptorheap.ext.acceleration-structure.stride.error.hlsl

@@ -8,6 +8,21 @@
 // RUN:   -fspv-extension=SPV_KHR_ray_query                                  \
 // RUN:   -spirv %s | FileCheck %s --check-prefix=OK
 
+// Verify -fvk-resource-heap-stride suppresses the diagnostic: the stride is a
+// user-supplied literal, so there is no computed stride to widen. The
+// acceleration structure runtime array must carry a literal ArrayStride (not
+// ArrayStrideIdEXT) and no OpConstantSizeOfEXT may be emitted for the
+// acceleration structure type.
+// There is deliberately no -fspv-extension list: SPV_KHR_ray_query would make
+// the pre-scan widen the stride by itself, so compiling would not depend on the
+// override. The extensions this shader needs are enabled by default.
+// RUN: %dxc -T cs_6_6 -E main -fspv-use-descriptor-heap                     \
+// RUN:   -fspv-target-env=vulkan1.3                                         \
+// RUN:   -fvk-resource-heap-stride 256                                      \
+// RUN:   -spirv %s > %t
+// RUN: FileCheck %s --check-prefix=CLI < %t
+// RUN: FileCheck %s --check-prefix=ABSENT < %t
+
 // Verifies: an acceleration structure loaded from ResourceDescriptorHeap is
 // rejected when the resource-heap stride was not widened to include the
 // acceleration structure descriptor size.
@@ -22,7 +37,7 @@
 // shader also needs for RayQuery (SPV_KHR_ray_tracing alone does not provide
 // the RayQueryKHR capability), and must compile.
 
-// CHECK: error: acceleration structure loaded from ResourceDescriptorHeap requires the resource heap stride to account for acceleration structure descriptors; compile with -fspv-extension=SPV_KHR_ray_tracing or -fspv-extension=SPV_KHR_ray_query
+// CHECK: error: acceleration structure loaded from ResourceDescriptorHeap requires the resource heap stride to account for acceleration structure descriptors; compile with -fspv-extension=SPV_KHR_ray_tracing, -fspv-extension=SPV_NV_ray_tracing, -fspv-extension=SPV_KHR_ray_query, or -fvk-resource-heap-stride
 
 // The variable initialized from the rejected load must not cascade into a
 // fatal error that hides the diagnostic above.
@@ -32,6 +47,17 @@
 // OK-DAG:    %[[Accel:[a-zA-Z0-9_]+]] = OpTypeAccelerationStructureKHR
 // OK-DAG:                               OpConstantSizeOfEXT %uint %[[Accel]]
 
+// The acceleration structure runtime array carries the literal stride.
+// CLI-DAG:     %[[Accel:[a-zA-Z0-9_]+]] = OpTypeAccelerationStructureKHR
+// CLI-DAG:  %[[AccelArr:[a-zA-Z0-9_]+]] = OpTypeRuntimeArray %[[Accel]]
+// CLI-DAG:                                OpDecorate %[[AccelArr]] ArrayStride 256
+
+// No computed stride may appear anywhere in the module. This is a separate
+// check, because a NOT after the last DAG match only covers the rest of the
+// output, and the decorations come before the types.
+// ABSENT-NOT: ArrayStrideIdEXT
+// ABSENT-NOT: OpConstantSizeOfEXT
+
 RWBuffer<float4> output : register(u0);
 
 [numthreads(1, 1, 1)]