[AlexanderSoftCode]

Context

Both GPU dropout paths ran a full Philox4x32-10 for every element, in forward and again in backward, and used one of its four output words. This replaces the four ElementwiseKernels with one RawKernel source compiled per dtype. It is one commit.

Kernel

Old behavior:

• SpatialDropout ran Philox for every pixel, though only one value per (sample, channel) matters. On the 16x16 to 4x4 maps in cifar10_breakdown, that is 16-256x redundant.
• float64 keep_prob made every element pay for a double comparison and division.
• SpatialDropout also did two 64-bit integer operations per element to recover its index.

What changed:

• Each Philox call now covers 8 elements using 16-bit draws, compared against an integer threshold. Threads own disjoint slices (element = thread_index * 8 + j).
• The kernel compiles per dtype (fp16, fp32, fp64), like conv's forward, so the output keeps the input dtype with no cast passes. This bends the "layer casts to a fixed dtype" rule in CLAUDE.md on purpose, so fp16 carries through without extra passes.
• Dropout stores no mask. Backward reruns the kernel from the same seed and offset, so there is no O(N) VRAM cost.
• SpatialDropout caches its (S, 1, 1, C) mask, because a rebuild would still need that buffer in each pass.

Results on an RX 9070 XT (cifar10_breakdown, batch 256, 3 repeats):

	Before	After
model.train() per step	5.217 ms	4.860 ms (-6.8%)
Dropout GPU time per step	0.429 ms	0.135 ms

That model has no plain Dropout layer. Timed separately, forward only, at rate 0.1:

Input shape	Dtype	Before	After
(128, 128)	fp32	7.5 µs	8.3 µs
(128, 16, 16, 64)	fp32	104 µs	28 µs
(128, 16, 16, 64)	fp16	102 µs	13 µs
(256, 32, 32, 64)	fp32	767 µs	226 µs
(256, 32, 32, 64)	fp16	782 µs	68 µs

Small tensors see no gain.

Dropped elements are exactly 0

The NumPy fallback multiplied by the mask, so 0 * NaN and 0 * inf produced NaN in dropped elements. The fallback now zeroes them first (_apply_mask), matching the GPU kernels and the rule already used in batchnorm_kernel.py. Kept NaN and inf values still pass through, so real problems stay visible.

This is free on GPU, but it makes applying the mask 3-6x slower on NumPy, about 6.7 ms per step on the benchmark's three spatial layers. I kept it so both backends agree.

Behavior changes

• Seeded GPU runs produce different masks than before, so they won't reproduce old runs or benchmark baselines.
• GPU rates round to the nearest 1/65536 (error at most 7.6e-6), so rates below 7.6e-6 drop nothing. Documented in both docstrings. The NumPy fallback keeps its float32 comparison.
• GPU dropout now raises TypeError for inputs that aren't fp16, fp32 or fp64.
• NumPy fallback: dropped NaN and inf inputs now give 0 instead of NaN.
• fp16 inputs lose the sign of negative zero in Dropout's GPU kernel. This is harmless because the values compare equal.

Layer dtypes under set_precision are unchanged from main, traced per layer on two test models. SpatialDropout receives and returns fp32 after a conv because Conv2d always outputs fp32.

Testing

• python3 -m unittest discover tests: 1188 tests pass on NumPy and CuPy (77 skipped), on ROCm in the devcontainer. Ruff is clean.
• New tests compare kernel output byte for byte against a pure-Python Philox4x32-10 reference, itself checked against Random123 known-answer vectors. They run in fp16 and fp32, across thread blocks, and through a real model step under each precision policy.
• The NaN/inf tests were confirmed to fail on the multiply-only code: SpatialDropout on GPU and both layers on NumPy. The NumPy failures are the regression from main; the GPU SpatialDropout failure was introduced and fixed within this branch.
• Nine deliberate kernel breaks (half-word order, overlapping threads, 16-bit threshold, no end guard, fixed counter, ignored offset high word, two fp16 faults, dropped values not zeroed) were each caught by at least one test.
• GPU checks ran on an RX 9070 XT only.

Not verified:

• CUDA and RTX 3050: nothing has compiled or run on NVIDIA, including the cuda_fp16.h path.
• Compile-failure paths: the fallback rebinding and the Dropout backward RuntimeError have no test.

Follow-ups (out of scope)

• make_built_layer passes seed= only to build(), so existing dropout tests that pass it are unseeded and their rate checks could fail by chance. The new tests set _seed_key directly.