Skip to content

fix(test): own test scratch dirs with tempfile::TempDir so they clean on drop - #476

Merged
MichaelTaylor3d merged 22 commits into
mainfrom
loop/leak-397-370
Sep 1, 2026
Merged

fix(test): own test scratch dirs with tempfile::TempDir so they clean on drop#476
MichaelTaylor3d merged 22 commits into
mainfrom
loop/leak-397-370

Conversation

@MichaelTaylor3d

@MichaelTaylor3d MichaelTaylor3d commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

DRAFT — DO NOT MERGE. Orchestrator gates.

Closes #397
Closes #370

What this does

Converts the hand-rolled std::env::temp_dir().join(..) test scratch directories to owned
tempfile::TempDir guards, so removal happens in Drop — including on an unwind, which is the
case that produced the leak. Every one of these sites ended with a manual remove_dir_all, which
is exactly the line a failing assertion skips: the runs a developer repeats were the runs that
leaked.

Counts — measured from the code, not from directory listings

count
std::env::temp_dir() call sites on main 65
converted to an owned TempDir guard 53
deliberately left 11

One further guard-lifetime defect was found by running the suite and is fixed here:
spend_audit_cli.rs::seeded_log bound the guard locally while returning the log that pointed
into it, which turned 10 tests red. See #370 for the shape.

main carried 65 sites and one pre-existing tempfile::Builder (the #361 fix in
content_serve.rs); this branch carries 11 and 55.

The tickets' own figures came from directory counts on one machine and did not match the code:
#370 estimated "roughly 50" sites across 8 named prefixes, and three of the prefixes reported as
leaking here — dig-webview-shared*, digstore-remote-test-decoy/*, dig-node-warmloc-engine*
in its literal form — do not exist in this repository at all. The first two belong to other
repos' harnesses; the third is the -engine tag of dig-node-warmloc-{tag}, which is converted.

The 11 left, individually

Genuinely leaks, NOT fixed here — none.

src/spend_audit.rs:1282 was previously listed here, deferred on the ground that the file was
owned by a live lane. That premise was false at the head it was written against#475, the
last PR to touch the file, merged 2026-09-01T00:35:35Z, 5h34m earlier, and none of the seven open
PRs touches it. Since Closes #370 would have retired the only ticket naming the site, it is now
converted: tmp_log() returns (SpendLog, tempfile::TempDir) and all 14 call sites bind the
guard for the life of the test. Run, not merely compiled: cargo test -p dig-node-service --lib spend_audit -> 47 passed; 0 failed. Binding the guard locally inside the helper would
reproduce exactly the seeded_log defect fixed above, which type-checks at every depth.

Also converted while the pattern was fresh, though it was benign:
dig-node-core/src/capsule_warm_locator_tests.rs:123node_with_pool_peer now returns
(Arc<NodeContent>, tempfile::TempDir) (3 callers). cargo test -p dig-node-core --lib capsule_warm_locator -> 3 passed; 0 failed.

Production code, not a test harness — two sites:

  • dig-node-core/src/lib.rs:537private_fallback_dir(), the degraded un-shared mode.
  • dig-node-service/src/server.rs:496 — the fail-closed ephemeral control-token dir. One per
    node process on a real host, by design.

Never creates a directory — three sites:

  • src/seed_export_cli.rs:139 — names a file that must be absent (json_is_refused_before_the_file_is_read).
  • src/spend_audit.rs:1575 — an absent ledger path (a_node_that_never_spent_reads_as_an_empty_ledger).
  • src/state.rs:1139 — a pure is_machine_state_dir path predicate.

Not code / already fixed — two sites:

Deliberate, and a guard would break the test — four sites:

  • dig-runtime/src/lib.rs:635, 930, 959, 990. These start the process-global runtime, which
    keeps reading the cache for the rest of the process; a guard dropped at the end of the test
    would pull the tree out from under it. They use fixed names, so the residue is bounded at
    four directories reused every run rather than one per run. The reason is now stated at each site.

Where the guard could not simply be a local binding

Three shapes needed more than a type change, and each is documented at the site:

  • The guard travels with the handle. peer.rs::fresh_pool_handle{,_on},
    tests/pool_connect.rs::start_pool, openrpc_drift_guard.rs::ephemeral_node,
    spend_audit_cli.rs::tmp_log and collateral_census.rs::seeded_store return
    (T, TempDir) — the started pool or store reads its files for its whole lifetime, so the tree
    must outlive the builder.
  • The guard rides on EnvHold in tests/server.rs — the value every caller already keeps
    alive for exactly the span the spawned server runs for.
  • The guard owns the PARENT in state.rs (ensure_dir_restricted / harden_state_dir) and
    updater.rs::unique_path. Their property under test is what happens when the directory does
    not yet exist; a guard over the directory itself would have created it and quietly retired
    the test.

Test

the_scratch_tree_is_removed_on_drop_and_on_an_unwind in profile_sync.rs. It captures one path
it owns and asserts on that path, rather than counting entries under the system temp root — a
disk-state assertion would be measuring every other test running concurrently on this box, and a
flaky one is worse than none.

The panic half is the load-bearing half: a helper that called TempDir::keep, or handed back
an unowned PathBuf, satisfies nothing there, while a merely tidier manual cleanup passes the
success case and fails it.

Residual — this reduces the class, it does not close it

Drop does not run on a panicking abort or on a killed process. Several of the directories
measured on this machine came from lanes killed mid-run, and no Drop-based fix can catch those.
Separately, Windows will not unlink a file another handle still holds, so a tree whose
wallet.sqlite a detached serve task still has open can survive its guard; content_serve.rs's
once-per-process sweep over trees idle longer than any live run is the existing answer to that
half, and it is unchanged here.

So the harness no longer leaks one directory per run on the ordinary and the failing path, which
is what the tickets asked for. It does not make the temp root self-cleaning.

Blast radius

Test-only, apart from the two production sites named above, which are not touched. The changed
symbols are test helpers plus their call sites: fresh_pool_handle{,_on} (10 callers, incl.
pex.rs), start_pool (3), transport_with_one_peer (3), tempdir in profile_sync.rs (27),
unique_path (4 + 5), tmp_log (3 + 2), seeded_store (10 + its integration twin), tmp_config
(4), scratch/dir/state_dir in the wallet crate, and EnvHold's shape in tests/server.rs.
No public API changes; no production behaviour changes.

Version

0.212.0, with Cargo.lock refreshed in the same commit.

Measured evidence — two consecutive cargo test -p dig-wallet --lib runs

Counting only the prefixes this crate owns (dig-tip*, dig-wallet-sup*, dig-wallet-svc*,
dig-watchlist*, dig-seed-export*) under %LOCALAPPDATA%\Temp:

run before after delta tests
1 284 287 +3 754 passed (1 load-induced timeout, see below)
2 287 290 +3 755 passed, 0 failed

The delta is a stable bound, not growth. The same three survive each run, and they are all the
same thing:

dig-wallet-svc-Ri5dmg/  wallet.sqlite  watched-keys.json
dig-wallet-svc-gRHJF6/  wallet.sqlite
dig-wallet-svc-eTMTFr/  wallet.sqlite

Every survivor contains an open wallet.sqlite. This is the residue stated in the PR and in #370's
own reference: Windows will not unlink a file another handle still holds, so remove_dir_all
fails and TempDir::drop swallows the error. It is bounded by the number of fixtures that leave a
database open, not by the number of tests.

For scale: those helpers are called ~40 times across the run. Roughly 40 scratch directories
created, 3 survive
— and all 3 from the one helper (service.rs::scratch) whose fixture opens a
wallet DB. Before this change every one of the ~40 persisted, because nothing removed them.

The one failure in run 1 is NOT this change

sync_supervisor::tests::stall_evidence_survives_the_end_of_a_session hit its own 120-second
wait: "timed out after 120s waiting for: the stall to be named despite the sessions turning over".
It passed alone (1 passed, run in isolation), passed in run 2, and passed in the full workspace
run. The box was compiling concurrently at the time. Recorded rather than dropped, because a
timing-sensitive 120s wait on a loaded machine is a real flake risk in CI and is worth its own
ticket if it recurs — it is not a directory-lifetime failure.

Full workspace

cargo test --workspace -j 2 on the same tree: 2,807 passed, 0 failed, zero compile errors and
zero code warnings.

The unwind half of the regression test is now independently mutation-proved

Previously only the success half was: disable_cleanup(true) trips the FIRST assertion, so the
unwind assertion never speaks. One mutation cannot prove both, because TempDir cleanup is a
single Drop on both paths. So a second, separate mutation was run — both blocks of
the_scratch_tree_is_removed_on_drop_and_on_an_unwind rewritten to the pre-fix idiom (a raw
create_dir_all plus a manual remove_dir_all at the end of the block, which is the nearest wrong
implementation: a merely tidier manual cleanup):

test ... FAILED
panicked at profile_sync.rs:1302: the tree must be gone after an unwind too -- this is the leak
test result: FAILED. 0 passed; 1 failed; 1032 filtered out

The success half REACHED its manual removal line and passed; the run failed at the unwind
assertion alone. Both mutations are recorded in the test's doc comment.

Version

0.216.0. origin/main merged in at 0.213.0; 0.209-0.215 are claimed by sibling PRs.
Cargo.lock refreshed in the same commit (cargo update -w --offline).

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

Gate — IN PROGRESS, not the verdict. Head read: f0224b7e44cd22f77f2cf35b499f6ba5849e84ef.

Item 1 (the count) — CONFIRMED, independently. Not sampled; full enumeration, same regex on both trees:

  • origin/main: git grep -c over *.rs for env::temp_dir()65
  • merge-base f280cbf65 (so the base did not move under the lane)
  • PR head → 12

65 − 12 = 53 converted. The PR body's table matches the code. The body enumerates all 12
remaining individually (my brief listed 10 categories; the body's extra two are
tests/content_serve.rs:40, a doc comment, and :87, #361's once-per-process sweep of the temp
root — neither is a creating call site). No undercount, no truncation.

Continuing with: the 12 justifications one by one, the seeded_log borrow-into-dropped-guard class
re-audit, and re-running the revert-proof mutation myself.

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

Gate — IN PROGRESS, not the verdict. Head f0224b7.

Item 2 (the 12 left) — 11 of 12 justifications CONFIRMED by reading the sites. One is not.

Confirmed correct to leave:

  • Production, not harness — both genuinely production. dig-node-core/src/lib.rs:537
    private_fallback_dir() is outside any #[cfg(test)], PID-keyed and documented as stable for
    the process lifetime; dig-node-service/src/server.rs:496 is inside
    resolve_state_dir_and_token()'s ephemeral closure, the fail-closed control-token dir on a real
    host. A self-deleting guard on either would be a real defect. Left correctly.
  • Never creates a directory — all three verified. seed_export_cli.rs:139 asserts
    !absent.exists() two lines later, so a TempDir would create what the test asserts is missing
    and invert it; spend_audit.rs:1575 passes an absent path to SpendLog::at and asserts an empty
    ledger; state.rs:1139 is a pure is_machine_state_dir path predicate with no fs call.
  • The 4 dig-runtime sites (lib.rs:635,930,959,990) each set DIG_IDENTITY_DIR/DIG_NODE_CACHE
    and then call build_runtime/dig_runtime_start — the process-global runtime, which keeps reading
    the cache after the test returns, so a guard dropped at test end would pull the tree out from under
    it. Each carries the reason inline. The names are fixed literals, so the residue is bounded at
    4 directories total, reused every run, not 4 per run. Confirmed.
  • content_serve.rs:40 is a doc comment and :87 reads the temp root for fix(test): serve integration test leaks a ~57MB temp dir per run — 1,123 dirs / 62.5 GB took the machine to ENOSPC #361's sweep.

GATING — the one exception: src/spend_audit.rs:1282.

The site is a genuine leak of exactly this class (create_dir_all, no removal, unique per call).
It was left on the stated ground that "spend_audit.rs is owned by a live lane / open PR in this
session"
. That premise does not hold at this head:

So there is no single-writer boundary to respect, and the file is free.

The consequence is what makes this gating rather than a nit: the PR says Closes #370, and #370 is
the only ticket tracking this instance.
On merge the one named, admitted, genuinely-leaking site
becomes both unfixed and untracked — a ticket closing on a premise that is false at its own head.

Either fix is fine and both are small; do not fix it by deleting the admission:

  1. Convert tmp_log() at spend_audit.rs:1282 using the same one-helper shape applied 53 times
    here — it returns SpendLog into a tree it creates, so it must return (SpendLog, TempDir) and
    every caller must bind the guard, which is precisely the seeded_log trap below; or
  2. drop Closes #370, and file a child naming spend_audit.rs:1282 so the residue stays tracked.

Still to come: the seeded_log borrow-into-dropped-guard re-audit, my own re-run of the revert-proof
mutation, and the version/lock check.

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

Gate — IN PROGRESS, not the verdict. Head f0224b7.

Item 3 (the seeded_log class) — the fix is right, and my independent re-audit found no
manifesting sibling.
I did not take the lane's word for it; I enumerated the class mechanically.

The fix at spend_audit_cli.rs:332 is the correct shape: seeded_log() -> (SpendLog, TempDir),
with all 10 callers binding let (log, _scratch) = seeded_log();.

Method — two passes, because the shape has two halves:

  1. Guard discarded at the call site. Every let (.., _) = f(); in the tree (19 hits). Checked
    each helper's return type: miss_setup -> (Bytes32,Bytes32,String),
    dpb -> (Vec<u8>,[u8;32]), reads_over -> (…,WalletDb,Arc<AtomicUsize>),
    spawn_capsule_rpc -> (String,Arc<Mutex<usize>>). No TempDir is discarded anywhere.
  2. Guard never returned in the first place — the actual seeded_log shape. Wrote a
    brace-matching pass over every .rs file: collect the 32 helpers whose signature returns a
    TempDir, then flag every function that calls one, returns a value, and has no TempDir in its
    own signature. 43 candidates, most name-collisions (dir, engine, state_dir). Read the ones
    that hold a live resource:
    • range_stream_wire.rs:85 holder_serving — correct, and the interesting one:
      Holder carries _cache: tempfile::TempDir (:80) explicitly so the node's cache outlives
      the builder. Its two other guards (server_dir, client_dir) back only in-memory
      NodeCerts, so dropping them is sound.
    • content_serve.rs:121 new stores the guard in Self (.dir).
    • dht.rs:927, download.rs:4899, holdings_wire.rs:132, peer_network.rs:33,
      peer_ping_identity.rs:50, dht_integration.rs:36 — all load a cert into memory and never
      read the tree again.
    • The compile_*_module/stage_real_module helpers return owned (Bytes32, Vec<u8>); nothing
      points into the tree.

Worth knowing about the class in general: it fails at runtime with NotFound, never at
compile time — which is why 10 tests went red rather than the build. The green suite is therefore
real evidence here, not merely an absence.

One non-gating note, same shape but currently benign:
crates/dig-node-core/src/capsule_warm_locator_tests.rs:123node_with_pool_peer binds
let dir = temp_dir("engine");, passes dir.path() into NodeContent::new(...), and returns only
Arc<NodeContent>. The guard dies at the end of the helper while the content object retains the
path. Nothing reads it today, so the suite is green and no directory is leaked — but it is one disk
read away from being the seeded_log failure, and it is the one place the pattern is not stated.
Returning (Arc<NodeContent>, TempDir) would make it uniform. Not a merge blocker.

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

Gate — IN PROGRESS, not the verdict. Head f0224b7.

Item 5 (the residual) — the PR body states it HONESTLY. Confirmed by reading, not by trusting.

The body has a section headed "Residual — this reduces the class, it does not close it", and it
names both mechanisms correctly: Drop does not run on a panicking abort or a killed process, and
Windows will not unlink a file another handle holds. It closes with "It does not make the temp root
self-cleaning."
It also states the measured bound (+3 per run, stable, all three holding an open
wallet.sqlite
) beside the ~40 created, rather than reporting only the improvement. That is the
opposite of the born-false-claim pattern — no gating finding here.

The single caveat is the one already filed above: the body's honesty about spend_audit.rs:1282
is undermined by Closes #370, which retires the only ticket carrying it.

Two blockers that are NOT about the code, flagged rather than fixed, per my brief:

  1. The version does not increment over main. Cargo.toml on this head is 0.212.0;
    origin/main is already at 0.213.0 (main moved: fix(wallet): retire the last largest-first rival, and make the unguarded sync setter unreachable #474, fix(config): a mistyped capability flag must be said out loud, not silently guessed #472, fix(sync): the background whole-store sync must not log a download it never stored #468 landed after this branch
    took its number). Check version increment will fail as-is. Needs a re-bump to at least
    0.214.0 on rebase — and note the recorded trap: git rebase --onto silently DROPS a bump
    commit whose patch is already upstream, with RC=0 and no conflict, so re-read the version
    from the file on disk afterwards
    , not from the commit log.
  2. mergeStateStatus=DIRTY, and all six required checks are ABSENT.
    check-merge-preconditions.sh --allow-draft (run unpiped, rc=1) reports
    Lint commit messages, Check version increment, Rustfmt, Clippy, Test + coverage,
    Release-script tests each as "ABSENT — no check-run attached to this SHA" — they have not
    run on f0224b7, so the four green CodeQL rows in the rollup are not the gate set. I verified
    the conflict is benign: git merge-tree --write-tree origin/main <head> conflicts only in
    Cargo.toml and Cargo.lock; every source file, including profile_sync.rs,
    sage/rpc.rs and sync_supervisor/tests.rs, auto-merges clean.

Remaining: my own re-run of the .disable_cleanup(true) revert-proof, then the verdict.

@MichaelTaylor3d MichaelTaylor3d left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHANGES-REQUIRED — head read: f0224b7e44cd22f77f2cf35b499f6ba5849e84ef.

One gating correctness finding, posted inline below, plus two mechanical merge blockers already
detailed in my interim comments. The conversion work itself is careful and verifies: the 65/53/12
count is exact, 11 of the 12 justifications hold on reading, the seeded_log fix is right and I
found no manifesting sibling, the revert-proof discriminates (I re-ran it), and the residual is
stated honestly in the body.

Comment thread crates/dig-node-service/src/spend_audit_cli.rs
@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

VERDICT: CHANGES-REQUIRED

Head read: f0224b7e44cd22f77f2cf35b499f6ba5849e84ef (resolved from the remote myself, not
from the dispatch brief). Reviewed in a private worktree at that SHA; the primary checkout and the
lane's dn-leak worktree were not mutated — I used dn-leak's target/ as a build cache only.

Ranked findings

1. GATING — Closes #370 retires the only ticket tracking src/spend_audit.rs:1282, and the
stated reason for leaving it does not hold at this head.
Full evidence in the inline thread. In
short: #475 (the last PR to touch the file) merged 5h34m before the deferral was recorded, no open
PR touches spend_audit.rs, and no lane worktree holds uncommitted work on it. Fix by converting
tmp_log() (it must return (SpendLog, TempDir) — a local guard reproduces the seeded_log
defect), or by dropping Closes #370 and filing a child.

2. BLOCKS MERGE, mechanical — the version does not increment. Cargo.toml = 0.212.0;
origin/main = 0.213.0 (#474, #472, #468 landed after this branch took its number). Flagged, not
fixed, per my brief. On rebase, re-read the version from the file on diskgit rebase --onto
silently drops a bump commit whose patch is already upstream, RC=0 and no conflict.

3. BLOCKS MERGE, mechanical — mergeStateStatus=DIRTY and all six required checks ABSENT.
check-merge-preconditions.sh --allow-draft, run unpiped, rc=1: Lint commit messages,
Check version increment, Rustfmt, Clippy, Test + coverage, Release-script tests are each
"ABSENT — no check-run attached to this SHA". The four green CodeQL rows in the rollup are not the
gate set. The conflict is benign: git merge-tree --write-tree conflicts only in Cargo.toml
and Cargo.lock; every source file auto-merges clean.

4. NON-GATING — crates/dig-node-core/src/capsule_warm_locator_tests.rs:123.
node_with_pool_peer drops its guard while the returned Arc<NodeContent> retains
dir.path(). Benign today (nothing reads it, so the suite is green and nothing leaks), but it is one
disk read from the seeded_log failure and the only place the pattern is not applied.

5. NON-GATING — the stall_evidence_survives_the_end_of_a_session flake does not gate. It
passed in isolation, in run 2, and in the full workspace run; the 120s wait is on stall detection,
not on disk. This PR does rewrite scratch() in that same file, but from a deterministic
(pid, seq) dir to a random one — no path by which that changes a stall timer. Worth its own ticket
if it recurs; it is not a directory-lifetime failure and it is not introduced here.

What I verified, and how

  • The count (item 1) — CONFIRMED, enumerated not sampled. env::temp_dir() over *.rs:
    origin/main 65, merge-base f280cbf 65 (the base did not move under the lane), head
    12. 65 − 12 = 53. The body enumerates all 12 individually.
  • The 12 (item 2) — 11 confirmed by reading the sites, detailed in my second interim comment.
    Both "production" sites are genuinely outside #[cfg(test)]; all three "never creates" sites would
    be inverted by a guard (seed_export_cli.rs:139 asserts !absent.exists() two lines on); the
    four dig-runtime sites use fixed literal names, so the residue is bounded at 4 total,
    reused every run
    , which I checked rather than accepted.
  • The seeded_log class (item 3) — re-audited independently, no manifesting sibling. Two passes:
    every let (.., _) = f(); in the tree (19 hits, none discards a TempDir), and a brace-matching
    scan for the actual shape — 32 guard-returning helpers, 43 candidate callers, each live-resource
    one read. holder_serving carries _cache: TempDir on its Holder deliberately;
    content_serve.rs::new stores the guard in Self; the cert helpers hold only in-memory
    NodeCerts. The class fails at runtime, never at compile time, so the green suite is real
    evidence here rather than an absence.
  • The revert-proof (item 4) — I re-ran the mutation myself, in my own worktree, restoring the
    file afterwards. Baseline: 1 passed; 0 failed; 1030 filtered out. With .disable_cleanup(true)
    added to the tempdir() helper: 0 passed; 1 failed, failing at profile_sync.rs:1278
    "the tree must be gone once the guard leaves scope". I checked the count, not the exit
    status (which was the shell's, from backgrounding). The test discriminates against the nearest
    wrong implementation.
  • The residual (item 5) — the body states it honestly. It has a section headed "this reduces
    the class, it does not close it"
    , names both mechanisms, reports the +3-per-run bound beside the
    ~40 created, and ends "It does not make the temp root self-cleaning." No born-false claim.

What I could not reach

  • I did not re-run cargo test --workspace -j 2. Disk is at 36 GB with three ENOSPC events
    today, so I scoped builds to -p dig-node-core --lib. The workspace claim of 2,807 passed is
    therefore unverified by me — and required Test + coverage has not run on this SHA either
    (finding 3), so nothing has yet confirmed it on CI.
  • I proved only the success half of the regression test. .disable_cleanup(true) fails at the
    first assertion, so my mutation does not demonstrate that the unwind half discriminates
    independently. That would need a mutation that cleans on the success path only; I judged it not
    worth the build budget given the panic half is a plain catch_unwind over the same helper.
  • .gitnexus was not used. Per the documented false-safe impactedCount: 0 on this repo, the
    blast radius above came from grep plus direct reads and the Python brace-matching scan.

Findings 2 and 3 clear on a rebase + re-bump. Finding 1 is the one that needs a decision.

@MichaelTaylor3d
MichaelTaylor3d marked this pull request as ready for review September 1, 2026 12:16
@MichaelTaylor3d
MichaelTaylor3d merged commit 12a3f49 into main Sep 1, 2026
15 checks passed
@MichaelTaylor3d
MichaelTaylor3d deleted the loop/leak-397-370 branch September 1, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant