Skip to content

fix[scripts]: emit real diffs from the DaCe determinism check - #2857

Draft
havogt wants to merge 1 commit into
GridTools:mainfrom
havogt:determinism-checks
Draft

fix[scripts]: emit real diffs from the DaCe determinism check#2857
havogt wants to merge 1 commit into
GridTools:mainfrom
havogt:determinism-checks

Conversation

@havogt

@havogt havogt commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Motivation

The test_next_dace_determinism session failed on a CI job with:

error: DaCe codegen is non-deterministic: 1/135 comparable program(s) differ

The uploaded artifact could not say what differed. write_diffs only wrote
the program name plus the relative paths whose (relpath, sha256) pair
differed, and run_determinism_check deletes run1/, run2/ and .dacecache
in its finally — so by the time the artifact was uploaded the two versions of
the file were gone:

testee_pyext
src/cpu/testee.cpp

Diagnosing it needed a local reproduction of the whole selection.

Change

_scan now also returns a (relpath, sha256) -> Path map, and write_diffs
uses it to render each differing source as a unified diff (capped at
MAX_DIFF_LINES).

One wrinkle: a name is compiled once per set of parameters, and a variant
carries no identity that survives across runs, so run1's variants cannot be
matched to run2's by digest. The previous placeholder gave up here — which is
exactly the case that occurred, since testee had 34 compiled variants of
which 2 differed. Variants are now paired greedily by smallest diff; pairing
them in digest order diffs unrelated variants against each other and reports
the difference between the variants (here: scan direction) rather than the
non-determinism.

Result

Same failure, replayed against the new code:

testee_pyext
src/cpu/testee.cpp

=== src/cpu/testee.cpp
-- variant: run1 77e1d8db7768... vs run2 345392b365eb...
--- run1/src/cpu/testee.cpp
+++ run2/src/cpu/testee.cpp
@@ -14,10 +14,10 @@
+    int * gpu___gtir_scan_output_carry_0;
+    DACE_GPU_CHECK(cudaMallocAsync((void**)&gpu___gtir_scan_output_carry_0, ...));
     int * gpu___gtir_scan_output_carry_1_0;
     DACE_GPU_CHECK(cudaMallocAsync((void**)&gpu___gtir_scan_output_carry_1_0, ...));
-    int * gpu___gtir_scan_output_carry_0;
-    DACE_GPU_CHECK(cudaMallocAsync((void**)&gpu___gtir_scan_output_carry_0, ...));

The underlying bug this surfaced

Not fixed here (it is a DaCe issue), but for the record — the differing program
is testee from test_scan.py::test_scan_nested_tuple_output, both forward
parametrizations. In dace/transformation/interstate/gpu_transform_sdfg.py:

for inodename, inode in set(input_nodes):
for onodename, onode in set(output_nodes):

input_nodes / output_nodes are lists built in deterministic order, then run
through set() to dedup. Data.__hash__ is hash(serialize.dumps(self)) — a
string hash, so iteration order follows PYTHONHASHSEED. That order is the
insertion order of the gpu_* clones into sdfg.arrays, hence the emitted
alloc/free order. Verified: identical output for a fixed seed, different output
across seeds; replacing both set(...) with an order-preserving dedup makes
the output seed-independent.

Testing

scripts/tests/python/test_dace_determinism.py — 46 pass, incl. four new cases
covering the unified-diff output, smallest-diff variant pairing, unpaired
variants, and truncation. Validated end to end by replaying the real
two-cache reproduction. pre-commit run clean.

https://claude.ai/code/session_01Sf2j16DZhoaJJgy2gWyKUR

The `diffs/` artifact only listed the relative paths whose digest differed
between the two runs, and `run_determinism_check` deletes both caches before
the CI job uploads it, so a failure could not be diagnosed from the artifact
alone. Render the differing sources as unified diffs instead.

A name is compiled once per set of parameters and the variants carry no
identity that survives across runs, so pair them by smallest diff — pairing
in digest order diffs unrelated variants and reports the difference between
them rather than the non-determinism.

Claude-Session: https://claude.ai/code/session_01Sf2j16DZhoaJJgy2gWyKUR
@havogt

havogt commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

cscs-ci run dace-determinism

@havogt

havogt commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Not sure if I like the change, but it's certainly more useful to have the diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant