Skip to content

Non-deterministic control display order across audit runs due to set iteration in merger.py #428

Description

@Jaydeep869

Description

Running darnit audit --show-all multiple times against the exact same repository outputs controls in different orders between runs. This relates to the determinism efforts outlined in #418.

What We Did

  1. Audited the same repository twice in separate terminal sessions:
    darnit audit <target> --framework reproducibility --show-all > run1.txt
    darnit audit <target> --framework reproducibility --show-all > run2.txt
  2. Compared the text outputs with diff -u run1.txt run2.txt.

What It Was Supposed To Do

Audit output should be completely deterministic across runs when running on the same unchanged codebase.

What We Got

The list of controls under --- Warnings --- randomly changes order between runs:

--- run1.txt
+++ run2.txt
@@ -20,8 +20,8 @@
 --- Warnings ---
-  ⚠ RE-02.01: WARN - Could not automatically verify - manual verification required
   ⚠ RE-03.01: WARN - Could not automatically verify - manual verification required
   ⚠ RE-01.02: WARN - Could not automatically verify - manual verification required
-  ⚠ RE-01.01: WARN - Could not automatically verify - manual verification required
+  ⚠ RE-02.01: WARN - Could not automatically verify - manual verification required
   ⚠ RE-02.02: WARN - Could not automatically verify - manual verification required
+  ⚠ RE-01.01: WARN - Could not automatically verify - manual verification required

Root Cause & Proposed Fix

In packages/darnit/src/darnit/config/merger.py around line 424:

all_control_ids: set[str] = set(framework.controls.keys())
if user:
    all_control_ids.update(user.controls.keys())
for control_id in all_control_ids:
    effective.controls[control_id] = ...

Because all_control_ids is an unordered Python set, iteration order is randomized by PYTHONHASHSEED.

Sorting the control IDs or preserving original list order from framework.controls.keys() ensures stable, deterministic output across runs.

Reference report: https://github.com/Jaydeep869/darnit/blob/reproducibility-evaluation/reports/experiment_1_numpy.md#bug-report-2-non-deterministic-control-display-order-across-audit-runs-due-to-set-iteration-in-mergerpy

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions