fs: copy directory trees for fs.cp() on the thread pool - #65488
Open
codebytere wants to merge 2 commits into
Open
fs: copy directory trees for fs.cp() on the thread pool#65488codebytere wants to merge 2 commits into
codebytere wants to merge 2 commits into
Conversation
Collaborator
|
Review requested:
|
codebytere
force-pushed
the
perf/fs-cp-dir-threadpool
branch
from
August 22, 2026 15:18
3065c10 to
6d25d6a
Compare
The C++ fast path that fs.cpSync() takes when no filter is given created the destination directories with default permissions, so a 0700 directory came out of the copy as 0755 (with the default umask). The JavaScript implementation, which fs.cp(), fs.promises.cp() and fs.cpSync() with a filter still use, chmod()s every directory it creates to the mode of its source, and so did cpSync before the port. Set the source directory's permissions on each directory the copy creates (the destination root included); directories that already exist keep theirs, as before. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
codebytere
force-pushed
the
perf/fs-cp-dir-threadpool
branch
from
August 22, 2026 15:55
6d25d6a to
cce0cf5
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #65488 +/- ##
==========================================
+ Coverage 90.12% 90.16% +0.04%
==========================================
Files 752 751 -1
Lines 252315 253662 +1347
Branches 47444 47797 +353
==========================================
+ Hits 227395 228725 +1330
+ Misses 16217 16192 -25
- Partials 8703 8745 +42
🚀 New features to boost your workflow:
|
codebytere
force-pushed
the
perf/fs-cp-dir-threadpool
branch
2 times, most recently
from
August 22, 2026 19:42
73a8c02 to
0ecff36
Compare
codebytere
force-pushed
the
perf/fs-cp-dir-threadpool
branch
from
August 23, 2026 11:19
0ecff36 to
508ece6
Compare
codebytere
force-pushed
the
perf/fs-cp-dir-threadpool
branch
from
August 23, 2026 12:31
508ece6 to
0c17f68
Compare
codebytere
force-pushed
the
perf/fs-cp-dir-threadpool
branch
from
August 23, 2026 12:53
0c17f68 to
6a197c9
Compare
Collaborator
codebytere
force-pushed
the
perf/fs-cp-dir-threadpool
branch
from
August 24, 2026 00:22
6a197c9 to
7e01f9b
Compare
codebytere
force-pushed
the
perf/fs-cp-dir-threadpool
branch
from
August 24, 2026 02:15
7e01f9b to
724e8cd
Compare
codebytere
force-pushed
the
perf/fs-cp-dir-threadpool
branch
from
August 24, 2026 02:57
724e8cd to
e9dec5a
Compare
fs.cp() and fs.promises.cp() walked the tree in JavaScript with several thread pool round trips per entry (opendir batches, two stat()s, the copyFile(), a chmod()), all awaited in sequence: a 2 100-file tree took ~215 ms with ~110 ms of that on the main thread, against ~36 ms for fs.cpSync(), which copies the tree in C++ when no filter is given. Factor that C++ walk into CopyDirRecursive(), which records the error instead of throwing so that it can run on any thread, and run it as one ThreadPoolWork request (CpDirJob) for fs.cp()/fs.promises.cp() when the destination directory does not exist yet and nothing has to run per entry (no filter, no dereference, permission model off). Copying into an existing tree keeps the JavaScript walk and its rules for what may already be there. The same tree now takes ~30 ms with under 1 ms on the main thread. For that job the walk follows the JavaScript walk's rules rather than cpSync's: it creates every directory with mkdir() and every file with an exclusive uv_fs_copyfile() (honouring the copyFile() mode flags) and fails with EEXIST if anything has appeared in their place since the JavaScript check, so it never opens or follows something it did not create; sockets, FIFOs and unknown entries are reported back to JavaScript, which rejects them with the same SystemErrors as before; relative link targets are made absolute lexically as path.resolve() does. cpSync keeps merging into existing directories, skipping special files and canonicalizing link targets. The walk now uses the error_code overloads of std::filesystem throughout (directory iteration included), so an unreadable directory inside the tree is reported as EACCES by both cp() and cpSync() instead of terminating the process, which cpSync() has done since the walk moved to C++. Filesystem errors raised inside the walk keep their codes, with 'cp' or 'copyfile'/'mkdir' as the syscall. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
codebytere
force-pushed
the
perf/fs-cp-dir-threadpool
branch
from
August 24, 2026 07:11
e9dec5a to
45406cc
Compare
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs the directory walk of
fs.cp()/fsPromises.cp()as one thread pool request using the C++ implementationfs.cpSync()already has, instead of a JavaScript walk with several awaited round trips per entry; the first commit fixes that implementation giving created directories default permissions instead of the source directory's mode.benchmark/fs/bench-cp.js(new,fsPromises.cp()of 500 files), 30 runsfsPromises.cp()of a 2 100-file treecpSyncof a0700directory (umask 022)07550700The JavaScript walk does opendir batches, two
stat()s, thecopyFile()and achmod()per entry, each awaited in sequence, with the bookkeeping on the main thread.fs.cpSync()without afilterhas done the whole walk in C++ since #58461, but that walk creates directories with default permissions where the JavaScript walk (andcpSyncbefore the port) gives them the source directory's mode; the first commit fixes that (the mode is applied once the directory's contents are copied, so read-only source directories still copy), with a test that fails onmain.The second commit factors the walk into
CopyDirRecursive(), which records an error instead of throwing so it can run on any thread, and runs it as aThreadPoolWorkrequest when the destination directory does not exist yet and nothing has to run per entry (nofilter, nodereference, nomodeflags forcopyFile(), permission model off); the request creates every destination directory itself withmkdir()and fails withEEXISTif anything has appeared in its place since the JavaScript check, so it never writes through a late symbolic link; copying into an existing tree keeps the JavaScript walk and every rule it has for what may already be there (#58869 lists wherecpSync's walk differs). Sockets, FIFOs and unknown entries found by the request are handed back to JavaScript, which rejects them with the sameSystemErrors as before (cpSynckeeps skipping them), and relative link targets are made absolute lexically aspath.resolve()does (cpSynccanonicalizes them). The walk uses theerror_codeoverloads ofstd::filesystemthroughout, so an unreadable directory inside the tree is reported asEACCESby bothcp()andcpSync()wherecpSync()currently terminates the process; filesystem errors from inside the walk keep their codes and reportcpas the syscall, ascpSyncdoes.Refs: #58461
Tests: new
test-fs-cp-sync-directory-mode.mjs,test-fs-cp-async-special-files-in-tree.mjs(a socket and a FIFO inside the tree: rejected bycp(), skipped bycpSync(), same as onmain)test-fs-cp-unreadable-directory.mjs(aborts onmainforcpSync)test-fs-cp-async-destination-appears-late.mjsandtest-fs-cp-async-symlink-targets.mjs; alltest-fs-cp*pass; a differential run over the option matrix (dereference,verbatimSymlinks,preserveTimestamps,force/errorOnExist, fresh and pre-populated destinations, symlinks, a socket and a FIFO in the tree) produces the same trees and outcomes as before.Disclosure: the code, test, benchmark, measurements and this description were written by Claude Code, directed and reviewed by @codebytere.