Skip to content

fix(cli-core): use safe-write for atomic replace, upgrade to 0.2.0 - #989

Merged
kvinwang merged 1 commit into
masterfrom
safe-write-fsutil
Aug 4, 2026
Merged

fix(cli-core): use safe-write for atomic replace, upgrade to 0.2.0#989
kvinwang merged 1 commit into
masterfrom
safe-write-fsutil

Conversation

@kvinwang

@kvinwang kvinwang commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Problem

fsutil::write_atomic derives its temp file as a fixed <path>.tmp. That avoids the extension-replacing trap — the doc comment calls it out explicitly — but a deterministic name is still shared by every concurrent writer to that path, and the file is opened with create(true).truncate(true) rather than exclusively.

So two writers truncate each other's temp file, and the first to rename leaves the second holding a descriptor that now points at the published target — the loser writes into the live file, then fails its own rename with ENOENT.

Measured with four writers racing on one path, using a faithful copy of the current implementation:

fsutil::write_atomic  | failed writes 120/160 | corrupt results 18/40
safe_write 0.2.0      | failed writes   0/160 | corrupt results  0/40

This is the failure the module was written to prevent — its own docs note that the auth webhook fails closed on invalid JSON, so a torn allowlist denies keys to every app on the host.

Fix

The workspace already depends on safe-write from five other crates, and it creates a uniquely named temp file, which removes the shared name at the source. write_atomic_inner is replaced by a call to it.

Upgrading the workspace pin 0.1.3 → 0.2.0 is what makes that worthwhile. 0.1.x had the extension bug this module was written to avoid, plus no parent-directory fsync, no cleanup on failure, and a permission reset on overwrite.

What changes for the three call sites

before after
concurrent writers corrupt or fail all succeed, one wins
failed write content left in <path>.tmp nothing left behind
rewriting an existing file mode reset to 0o666 & !umask mode preserved

register_app_in_allowlist already held lock_exclusive, but write_state and write_token_file did not — and both run in the install flow the module docs describe as racy. write_token_file currently leaves a complete bearer token in vmm-auth-token.tmp if the write fails.

What is not changed

lock_exclusive is untouched. It solves a different problem — serializing a read-modify-write so two processes cannot each publish a complete file built from the same stale read — and safe-write explicitly does not do that. The module docs now state the split.

Two behaviours were checked against the actual call sites rather than assumed:

  • safe-write creates parent directories. All three call sites create theirs beforehand (install.rs:126-133), and the allowlist path is validated by the read_to_string that precedes the write, which is where the friendly "run dstackup install first" error comes from. Nothing can mask a mistyped path.
  • safe_write_with_mode applies the umask, where the old code chmod'd past it. The only mode in the tree is 0o600, which no realistic umask affects.

Verification

Two new tests, both confirmed to fail against the previous implementation rather than merely passing:

test old new
concurrent_writers_do_not_clobber_each_other FAIL — No such file or directory (os error 2) pass
rewrite_preserves_existing_permissions FAIL — widened a credential file to 664 pass

atomic_write_replaces_contents now asserts the directory is empty apart from the target, since the temp file no longer has a predictable name to check for.

16 tests pass in dstack-cli-core and 21 in dstackup; cargo fmt --check is clean and clippy reports 0 warnings. All five other crates that depend on safe-write (dstack-util, kms, gateway, vmm, dstack-attest) were checked against 0.2.0.

`fsutil::write_atomic` derived its temp file as a fixed `<path>.tmp`. That
avoided the extension-replacing trap (the doc comment says so explicitly), but
a deterministic name is still shared by every concurrent writer to that path,
and the file was opened with `create(true).truncate(true)` rather than
exclusively. Two writers therefore truncate each other's temp file, and the
first one to rename leaves the second holding a descriptor that now points at
the published target — so the loser writes into the live file and then fails
its own rename with ENOENT.

Measured with four writers racing on one path: 120 of 160 writes failed and 18
of 40 rounds left a target that was neither writer's content.

The workspace already depends on `safe-write` from five other crates, and it
creates a uniquely named temp file, so delegating to it removes the shared name
at the source. Upgrading the workspace pin to 0.2.0 is what makes that
worthwhile: 0.1.x had the extension bug this module was written to avoid, plus
no parent-directory fsync, no cleanup on failure, and a permission reset on
overwrite.

What this changes for the three call sites:

- concurrent writers no longer corrupt or fail. `register_app_in_allowlist`
  already held `lock_exclusive`, but `write_state` and `write_token_file` did
  not, and both run in the install flow that the module docs describe as
  racy.
- a failed write no longer leaves the content behind. `write_token_file`
  currently leaves a complete bearer token in `vmm-auth-token.tmp`.
- rewriting an existing file preserves its permissions instead of resetting
  them to `0o666 & !umask`.

`lock_exclusive` is untouched. It solves a different problem — serializing a
read-modify-write so two processes cannot each publish a complete file built
from the same stale read — and `safe-write` explicitly does not do that. The
module docs now say so.

The three current call sites all create their parent directory beforehand, and
the allowlist path is validated by the read that precedes the write, so
`safe-write` creating parent directories cannot mask a mistyped path. The only
`mode` in the tree is `0o600`, which is unaffected by the umask.

Two new tests, both confirmed to fail against the previous implementation:
concurrent writers ("No such file or directory") and permission preservation
("widened a credential file to 664").
Copilot AI lite review requested due to automatic review settings August 4, 2026 03:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a real race/corruption hazard in dstack-cli-core’s atomic file write helper by switching from a deterministic <path>.tmp temp file to the workspace-standard safe-write crate (now pinned to 0.2.0). This improves correctness under concurrent writers and better preserves file permissions on rewrites, which is particularly important for state/allowlist and credential-token writes.

Changes:

  • Replace the hand-rolled write_atomic_inner implementation with safe_write::safe_write / safe_write_with_mode.
  • Clarify module documentation to distinguish “atomic replace” from “read-modify-write serialization” (lock_exclusive).
  • Add regression tests for concurrent writers and permission preservation; update existing atomic-write test to assert no temp files remain without relying on a predictable temp name.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated no comments.

File Description
dstack/crates/dstack-cli-core/src/fsutil.rs Delegate atomic replace to safe-write, clarify locking semantics, and add regression tests for concurrency + permission preservation.
dstack/crates/dstack-cli-core/Cargo.toml Add safe-write as a direct dependency for dstack-cli-core.
dstack/Cargo.toml Bump workspace safe-write version from 0.1.3 to 0.2.0.
dstack/Cargo.lock Update lockfile for safe-write 0.2.0 and its dependency changes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@kvinwang
kvinwang merged commit f6af395 into master Aug 4, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants