[WebGPU] Add buffer-region workspace preallocation

[Chi Lo (chilo-ms)]

Stack and merge target

Depends on #32071. The WebGPU-specific changes were restacked onto its updated head (e964fed7b7), and current main at the time (34d838fbcd) was merged into this branch.

This draft targets main. Until #32071 lands, the PR diff also includes its prerequisite workspace-planning changes; merge the parent first.

Summary

• Represent planned workspace as a provider-native buffer plus byte offset and usable size, while retaining the pointer-returning API for addressable allocations.
• Preserve WebGPU buffer offsets through tensor views, bind groups, segmented bindings, transfers, indirect dispatch, and graph capture/replay.
• Add workspace-only memory-pattern caching when ordinary activation patterns are disabled, including multiple independently tracked slots per kernel.
• Wire WebGPU MatMulNBits DP4A activation/scales scratch and conditional subgroup activation-prepack scratch into planned regions, with dynamic-allocation fallback.
• Add allocator high-water statistics, workspace-region regressions, and C++/ORT GenAI benchmarks with controlled cache/planner comparisons.

Integration notes

Conflict resolution preserves main's deferred pipeline dispatch, zero-initialized buffer reuse, and error propagation, along with #32071's diagnostic hooks and minimal-build exclusions. Workspace declarations use the current positional WorkspaceInputShape API. Plugin and minimal configurations retain dynamic scratch instead of requiring unavailable workspace interfaces.

Planning remains sequential and is disabled for multiple logical streams on one device. Ordinary WebGPU activation planning is unchanged. The base MatMulNBits kernel declares workspace for eligible DP4A and subgroup-prepack paths; this does not imply coverage of every MatMulNBits dispatch or fused operator.

Controlled WebGPU benchmark

The controlled benchmark report is the reference for these results. The superseded C++-only report has been removed to avoid presenting measurements with different cache settings and memory windows as equivalent.

Method: RTX 5090 Laptop GPU, Windows Release, Dawn Vulkan, driver 610.62. Both harnesses use enableInt64=1, graph capture off, NHWC, and explicit disabled/bucket storage-buffer caching. Each scratch/planned configuration uses five warmups, three measured requests, and two order-balanced fresh processes per phase. Latency and memory run in separate processes. The focused matrix comprises 96 workers: 80 successful and 16 OOM.

C++ measures a cached-prefill forward with repeated BOS tokens, one-token zero past cache, and full logits. GenAI uses seeded synthetic prompts and 128 generated tokens, with prefill and decode reported separately. Comparisons are controlled within each harness; these remain different workloads.

Latency

Values are scratch / planned, in seconds. GenAI decode covers the 127 tokens after the first generated token.

Model / prompt	Cache	C++ prefill	GenAI prefill	GenAI decode	GenAI full scenario
Qwen 2.5 1.5B / 8K	Disabled	1.887 / 1.792	1.828 / 1.822	1.657 / 2.070	3.491 / 3.898
Qwen 2.5 1.5B / 8K	Bucket	1.662 / 1.648	1.797 / 1.770	1.494 / 1.472	3.315 / 3.262
Qwen 2.5 1.5B / 12K	Disabled	OOM / OOM	OOM / OOM	N/A	OOM / OOM
Qwen 2.5 1.5B / 12K	Bucket	3.054 / 3.024	3.143 / 3.150	1.819 / 2.566	4.983 / 5.737
Qwen 2.5 7B / 4K	Disabled	1.770 / 1.668	1.615 / 1.599	2.280 / 2.760	3.903 / 4.366
Qwen 2.5 7B / 4K	Bucket	1.508 / 1.499	1.560 / 1.561	2.106 / 2.112	3.689 / 3.697

Memory

Values are scratch / planned, in MiB. A single external nvidia-smi stream per memory worker covers process launch through exit; the child-reported measurement window selects the post-warmup samples. Both harnesses therefore use the same device-wide counter. These are sampled maxima, not exact allocator high-water marks.

Model / prompt	Cache	C++ whole-process peak	C++ post-warmup peak	GenAI whole-process peak	GenAI post-warmup peak
Qwen 2.5 1.5B / 8K	Disabled	17,606 / 17,392	17,392 / 14,824	19,576 / 17,264	17,602 / 14,992
Qwen 2.5 1.5B / 8K	Bucket	4,941 / 5,101	4,941 / 5,101	4,933 / 5,093	4,933 / 5,093
Qwen 2.5 1.5B / 12K	Bucket	23,655 / 23,655	23,655 / 17,985	23,639 / 23,639	23,639 / 17,969
Qwen 2.5 7B / 4K	Disabled	21,151 / 20,517	21,151 / 17,559	21,067 / 20,475	20,577 / 18,257
Qwen 2.5 7B / 4K	Bucket	7,328 / 7,488	7,328 / 7,488	7,320 / 7,480	7,320 / 7,480

Interpretation:

• Storage-buffer caching explains most of the earlier C++/GenAI VRAM discrepancy. With bucket caching, their whole-process peaks agree within 8-16 MiB.
• The 24% reduction at 1.5B/12K is post-warmup only. Whole-process peaks are unchanged because the first recording pass still uses scratch allocations.
• Planning improves cache-disabled C++ prefill by about 5-6%, but GenAI decode is 21-25% slower in those configurations. At bucket-cached 12K, decode is 41.1% slower and the full scenario is 15.1% slower.
• Both disabled-cache 12K modes OOM in both harnesses; both bucket-cache modes succeed. Caching, not planning, enables 12K capacity in this comparison.
• Two process pairs do not establish precise small latency differences. Actual memory-sample spacing was median 15.99 ms, p95 16.09 ms, maximum gap 102.42 ms despite requesting 5 ms. The exact allocator/driver cause of decode regressions has not been established.

Validation

• Windows Release WebGPU/Vulkan builds succeeded for onnxruntime_test_all and onnxruntime_provider_test; the provider-test target was rebuilt for the controlled instrumentation.
• 22 targeted framework, memory-pattern, workspace-region, workspace-shape, and tensor-offset tests passed.
• 5 targeted WebGPU workspace-helper and MatMul execution tests passed without skips, including a rerun after the WASM guard fixes.
• The shared CUDA/WebGPU benchmark regression suite passed: 13 tests and 21 subtests.
• The controlled 1K smoke matrix passed all 16 workers. The full focused matrix recorded the 80 successful workers and 16 expected-capacity OOM outcomes above.
• Successful GenAI output hashes matched across cache/planner settings and latency/memory phases. Matching-option diagnostics confirmed planned-region consumption with both cache modes.
• All 24 numerical report rows were checked against raw data; memory peaks were independently recomputed from timestamped samples.
• Formatting and whitespace checks passed. The initial native-only/WASM helper compile issues were fixed in fc7463f156.

These are targeted checks, not a full-suite run or proof of planned-workspace consumption on every GPU path. The report includes reproduction commands, exact timing boundaries, provider options, runtime provenance, and measurement limitations.