Context
bigfiles currently uses the ignore crate's parallel walker for directory traversal on all platforms. This is fast and portable, but on NTFS volumes there is a substantially faster approach: reading the Master File Table (MFT) directly.
Tools like everything.exe use this technique to enumerate every file on a volume in one sequential pass, often at hundreds of thousands of files per second, without ever doing a tree walk. The USN journal can be used on top of this for incremental updates.
Goal
Evaluate whether bigfiles should add an MFT-direct fast path on Windows, gated behind a --fast flag or auto-detected.
Open questions
- Is there a maintained Rust crate that exposes MFT reads?
ntfs (https://crates.io/crates/ntfs) is a starting point but may not cover the full API surface needed.
- MFT reads typically require administrator privileges. How do we degrade gracefully when not elevated?
- How do we preserve the gitignore /
--exclude semantics if we skip the tree walk entirely? (Likely: enumerate via MFT, then filter paths.)
- Benchmarks vs the current
ignore-based walker on representative trees (Windows C:\Users\<user>, a node_modules, a Steam library).
Why this matters
On Windows with large drives this is the difference between "scan finishes in 30 seconds" and "scan finishes in 2 seconds." It is also the single largest perf gap between bigfiles and the Windows-native tools users will compare it to.
Status
Not started. Tracking for future work; contributions welcome.
Context
bigfiles currently uses the
ignorecrate's parallel walker for directory traversal on all platforms. This is fast and portable, but on NTFS volumes there is a substantially faster approach: reading the Master File Table (MFT) directly.Tools like
everything.exeuse this technique to enumerate every file on a volume in one sequential pass, often at hundreds of thousands of files per second, without ever doing a tree walk. The USN journal can be used on top of this for incremental updates.Goal
Evaluate whether bigfiles should add an MFT-direct fast path on Windows, gated behind a
--fastflag or auto-detected.Open questions
ntfs(https://crates.io/crates/ntfs) is a starting point but may not cover the full API surface needed.--excludesemantics if we skip the tree walk entirely? (Likely: enumerate via MFT, then filter paths.)ignore-based walker on representative trees (WindowsC:\Users\<user>, anode_modules, a Steam library).Why this matters
On Windows with large drives this is the difference between "scan finishes in 30 seconds" and "scan finishes in 2 seconds." It is also the single largest perf gap between bigfiles and the Windows-native tools users will compare it to.
Status
Not started. Tracking for future work; contributions welcome.