Skip to content

Keep files deeper than maxdepth when mv deletes the source - #2233

Merged
martindurant merged 1 commit into
fsspec:masterfrom
darshjoshi:fix-mv-maxdepth
Oct 6, 2026
Merged

martindurant merged 1 commit into
fsspec:masterfrom
darshjoshi:fix-mv-maxdepth

Conversation

@darshjoshi

Copy link
Copy Markdown
Contributor

AbstractFileSystem.mv(..., recursive=True, maxdepth=N) copies only the paths within maxdepth, but then calls self.rm(path1, recursive=True) with no depth limit. The whole source tree is removed, so files deeper than maxdepth are deleted without ever having been copied.

Before / After

m = fsspec.filesystem("memory")
m.pipe({"/src/top.txt": b"top", "/src/sub/mid.txt": b"mid", "/src/sub/deeper/low.txt": b"low"})
m.mv("/src", "/dst", recursive=True, maxdepth=2)
print(sorted(m.find("/src")), sorted(m.find("/dst")))

Before:

[] ['/dst/sub/mid.txt', '/dst/top.txt']      # low.txt is gone

After:

['/src/sub/deeper/low.txt'] ['/dst/sub/mid.txt', '/dst/top.txt']

Cause

copy expands with maxdepth and skips directories, but mv removes the source with an unbounded recursive rm. Just passing maxdepth to rm is not enough, because it would then call rmdir on directories that still hold deeper files (on MemoryFileSystem that raises FileNotFoundError for implicit directories, or ENOTEMPTY for ones created with mkdir).

Fix

When recursive=True and maxdepth is set, mv expands the source with the same maxdepth before copying. After the copy it removes only those files, then removes a directory within that depth only if it still exists and is empty. Without maxdepth, behaviour is unchanged.

Note: with maxdepth, directories within that depth that are empty after the move (including ones that were already empty) are now removed. copy already doesn't recreate directories at the destination when maxdepth is set.

Filesystems that override mv with a native rename (local, sftp, ftp, etc.) are not affected and not touched.

Tests

  • Added test_mv_recursive_maxdepth_keeps_deeper_files in fsspec/implementations/tests/test_memory.py; it fails on current master (FileNotFoundError for the deeper file) and passes with the fix.
  • Full pytest fsspec: 1993 passed, 248 skipped, 2 xfailed.
  • ruff check, ruff format --check, codespell clean.
  • Changelog entry added.

Related: #2205 touches nearby lines of mv (same-path/nesting checks), so whichever lands second needs a small rebase. The destination layout at maxdepth=1 is #2161 and not changed here.

@martindurant
martindurant merged commit 5a54757 into fsspec:master Oct 6, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants