Skip to content

Harden the shared Bun download cache against eviction and torn publishes - #19

Merged
arcaputo3 merged 3 commits into
agent/idiomatic-scalajs-mill-workflowsfrom
agent/bun-cache-hardening
Aug 26, 2026
Merged

Harden the shared Bun download cache against eviction and torn publishes#19
arcaputo3 merged 3 commits into
agent/idiomatic-scalajs-mill-workflowsfrom
agent/bun-cache-hardening

Conversation

@arcaputo3

Copy link
Copy Markdown
Contributor

Stack 6/N. Base is agent/bun-vendored-guard (#18), not main.

Why

An adversarial review pass over the stack found two ways the shared download cache introduced in
#14 could wedge or poison the now-default managed toolchain, plus three smaller sharp edges.

The bugs

An evicted cache was never repopulated. downloadedBunExecutable returned a plain PathRef
into the cache — Revalidate.Never — for a file that lives outside every Task.dest. Deleting
~/.cache/mill-bun (it is named and documented as a cache; evicting it is legitimate) left every
Mill run trusting a task result that points at a file that no longer exists. Nothing recovered
until the user guessed the right clean. The refs are now quick-signature (mtime + size)
Revalidate.Always, so recovery costs one stat per evaluation.

The cross-filesystem publish was not atomic. When the cache sits on a different filesystem
than out/ (a mounted CI cache volume; Docker with a bind-mounted workspace), the fallback was a
plain copy straight to the checksum-keyed final path. A concurrent reader could execute a
partially written binary, and a build killed mid-copy left a truncated file that every later build
trusts unconditionally — permanent poisoning, because existence at the path is the cache's proof
of validity. The fallback now copies to a temp name inside the cache directory and renames, so
the publish itself is always a same-filesystem atomic move. The copy also preserves file
attributes: the old os.copy.over silently dropped the executable bit.

Also fixed

  • Losing a publish race is judged by the outcome (cached exists), not the exception type —
    Windows reports it as a sharing violation, not FileAlreadyExistsException.
  • copyTree recreated relative symlinks as absolute paths into the source tree, so staged
    copies stopped being self-contained the moment the upstream task was cleaned. Raw link targets
    are now preserved verbatim (bun's node_modules/.bin entries are relative links).
  • A relative MILL_BUN_CACHE_DIR resolved against nothing and crashed with a raw
    IllegalArgumentException; it now resolves against the workspace root.
  • CI: the build job now sets MILL_BUN_USE_SYSTEM=true so the system-Bun path (findOnPath,
    version verification) keeps coverage after Pin Bun 1.4.0, compose the asset matrix, share the download cache #14 flipped the suite default to managed — that flip
    had made the job's setup-bun step install a binary nothing used. And pull_request CI is no
    longer restricted to PRs targeting main, so stacked PRs get checks.

Tests

Each fix was confirmed against the parent: the new test fails without the fix, passes with it.

  • an evicted download cache is repopulated, not trusted (integration, forked evals with a
    private MILL_BUN_CACHE_DIR) — fails on the parent at exactly the reviewed scenario: the
    post-eviction eval succeeds while pointing at a file that no longer exists.
  • relative symlink targets are preserved verbatim — fails on the parent (target absolutized).
  • cross-filesystem publish preserves permissions and leaves no temp debris and
    … tolerates losing the race pin the new publish contract.

Known limits (unchanged, deliberate)

  • musl detection can false-positive on a glibc distro that installs the musl compat package;
    the musl binary still runs there, and bunUseMusl is the documented override.
  • prepareWebStage symlinks node_modules on Windows, which needs Developer Mode or admin;
    the Windows CI leg planned for this stack will tell us whether a junction fallback is needed.

arcaputo3 and others added 2 commits August 26, 2026 17:45
An evicted cache entry was never repopulated: downloadedBunExecutable
returned a Revalidate.Never PathRef to a file outside every Task.dest,
so deleting ~/.cache/mill-bun left every build trusting a path that no
longer exists until the user guessed the right `clean`. Cache refs now
carry quick (mtime+size) signatures with Revalidate.Always — recovery
costs one stat per evaluation.

The cross-filesystem publish fallback copied straight to the
checksum-keyed final path, so a concurrent reader could execute a
partially written binary and a build killed mid-copy poisoned the cache
permanently. The fallback now stages next to the final path and renames
— the publish is always a same-filesystem atomic move — and preserves
the executable bit the old os.copy.over silently dropped. Losing a
publish race is judged by the outcome (cached exists), not the
exception type, since Windows reports a sharing violation rather than
FileAlreadyExistsException.

Also: copyTree preserves relative symlink targets verbatim instead of
absolutizing them into the source tree; a relative MILL_BUN_CACHE_DIR
resolves against the workspace root instead of crashing; the CI build
job pins MILL_BUN_USE_SYSTEM=true so the system-Bun path keeps coverage
now that the suite defaults to managed; and pull_request CI is no
longer restricted to PRs targeting main, so stacked PRs get checks.

Every fix was confirmed to fail on the parent commit before trusting
its pass: the eviction integration test (forked evals with a private
MILL_BUN_CACHE_DIR) fails there with a successful eval pointing at a
missing file, and the relative-symlink test fails with an absolutized
target.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BunTypeScriptTests extends upstream Mill's TypeScriptTests, so the
unqualified this.tsDeps() inside bunTestPackageJson resolved to the
Node toolchain defaults (ts-node, tsconfig-paths, @types/node) that the
outer trait deliberately replaced. Those names always survived the
outer-name filter, so a bare test module's package.json never equaled
the outer's and the reusesOuterInstall branch was unreachable — with
bunRequireLockfile on, every bare test module demanded its own
lockfile, contradicting the documented contract. The trait now pins
tsDeps to the outer module's.

The guard test passed anyway because both of its assertions also hold
in the broken standalone branch (the suite relaxes the lockfile
requirement, and installs never write into the source tree). It now
also asserts the returned install path is the outer module's
npmInstall.dest — confirmed failing before this fix and passing after.

Also: dev() on BunTypeScriptWebModule served straight out of the cached
webStage output while its sync thread mirrored live edits (but never
deletions) into it, and bundle builds from that same stage — a file
created and deleted during a dev session would ship in the next
production bundle. dev() now serves from a private copy in its own
command dest; copyTree preserves the node_modules symlink, so the copy
is cheap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arcaputo3
arcaputo3 deleted the branch agent/idiomatic-scalajs-mill-workflows August 26, 2026 22:11
@arcaputo3 arcaputo3 closed this Aug 26, 2026
Make bare test modules actually reuse the outer install
@arcaputo3 arcaputo3 reopened this Aug 26, 2026
@arcaputo3
arcaputo3 changed the base branch from agent/bun-vendored-guard to agent/idiomatic-scalajs-mill-workflows August 26, 2026 22:17
@arcaputo3
arcaputo3 merged commit 5b96f1a into agent/idiomatic-scalajs-mill-workflows Aug 26, 2026
@arcaputo3
arcaputo3 deleted the agent/bun-cache-hardening branch August 26, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant