Description
Running darnit audit --show-all multiple times against the exact same repository outputs controls in different orders between runs. This relates to the determinism efforts outlined in #418.
What We Did
- Audited the same repository twice in separate terminal sessions:
darnit audit <target> --framework reproducibility --show-all > run1.txt
darnit audit <target> --framework reproducibility --show-all > run2.txt
- Compared the text outputs with
diff -u run1.txt run2.txt.
What It Was Supposed To Do
Audit output should be completely deterministic across runs when running on the same unchanged codebase.
What We Got
The list of controls under --- Warnings --- randomly changes order between runs:
--- run1.txt
+++ run2.txt
@@ -20,8 +20,8 @@
--- Warnings ---
- ⚠ RE-02.01: WARN - Could not automatically verify - manual verification required
⚠ RE-03.01: WARN - Could not automatically verify - manual verification required
⚠ RE-01.02: WARN - Could not automatically verify - manual verification required
- ⚠ RE-01.01: WARN - Could not automatically verify - manual verification required
+ ⚠ RE-02.01: WARN - Could not automatically verify - manual verification required
⚠ RE-02.02: WARN - Could not automatically verify - manual verification required
+ ⚠ RE-01.01: WARN - Could not automatically verify - manual verification required
Root Cause & Proposed Fix
In packages/darnit/src/darnit/config/merger.py around line 424:
all_control_ids: set[str] = set(framework.controls.keys())
if user:
all_control_ids.update(user.controls.keys())
for control_id in all_control_ids:
effective.controls[control_id] = ...
Because all_control_ids is an unordered Python set, iteration order is randomized by PYTHONHASHSEED.
Sorting the control IDs or preserving original list order from framework.controls.keys() ensures stable, deterministic output across runs.
Reference report: https://github.com/Jaydeep869/darnit/blob/reproducibility-evaluation/reports/experiment_1_numpy.md#bug-report-2-non-deterministic-control-display-order-across-audit-runs-due-to-set-iteration-in-mergerpy
Description
Running
darnit audit --show-allmultiple times against the exact same repository outputs controls in different orders between runs. This relates to the determinism efforts outlined in #418.What We Did
darnit audit <target> --framework reproducibility --show-all > run1.txtdarnit audit <target> --framework reproducibility --show-all > run2.txtdiff -u run1.txt run2.txt.What It Was Supposed To Do
Audit output should be completely deterministic across runs when running on the same unchanged codebase.
What We Got
The list of controls under
--- Warnings ---randomly changes order between runs:Root Cause & Proposed Fix
In
packages/darnit/src/darnit/config/merger.pyaround line 424:Because
all_control_idsis an unordered Pythonset, iteration order is randomized byPYTHONHASHSEED.Sorting the control IDs or preserving original list order from
framework.controls.keys()ensures stable, deterministic output across runs.Reference report: https://github.com/Jaydeep869/darnit/blob/reproducibility-evaluation/reports/experiment_1_numpy.md#bug-report-2-non-deterministic-control-display-order-across-audit-runs-due-to-set-iteration-in-mergerpy