[Perf]: DirectorySizeCache retains ~600 MB at large-drive scale and has no entry bound

[G00dS0ul]

Summary

DiskScannerEngine.DirectorySizeCache gains one entry per folder walked and has no upper bound on
entry count. Each entry additionally retains a full file path per distinct extension. On a large
drive this leaves roughly 600 MB of live, reachable objects after a scan has finished — memory
that no garbage collector can reclaim, because it is not garbage.

This is the half of the memory problem that #141 does not address. #141 removes the transient spike
during a scan; this issue is about what is still held after it.

Symptom

After a completed scan of a large drive, the backend's working set does not fall back to its idle
baseline. It stays elevated until the cache entries are actually removed — which happens only when
the TTL expires (ScanCacheTtlMinutes, default 15) or when a newer scan pushes this one out of the
5-most-recent-roots window (MaxCacheScans, default 5).

No amount of GC tuning changes this. The objects are reachable from a live singleton.

Measurements

Synthetic DirectorySizeCache built at realistic Windows path lengths and varied file sizes, each
scale point run in its own process so Process.PeakWorkingSet64 is uncontaminated. Retained figures
are GC.GetTotalMemory(forceFullCollection: true) deltas measured after a full blocking collection,
so they count live objects only.

Retained by cache shape — 400,000 folders, 4 extensions per folder

Cache shape	Retained	vs current
Current (Extensions with full LargestFilePath)	598 MB	—
Extensions kept, LargestFilePath dropped	279 MB	−53%
No Extensions at all	81 MB	−86%

Retained per entry — linearity check across an 80× range

Folders	Bytes per entry
5,000	1,533
20,000	1,542
50,000	1,548
100,000	1,551
200,000	1,562
400,000	1,568

Linear within +2.2%, so the cost scales predictably with folder count.

Extension density — 400,000 folders

Distinct extensions per folder	Retained
2	359 MB
4	598 MB
8	1,097 MB

At 8 distinct extensions per folder the retained cache alone exceeds 1 GB.

Caveat on these figures. The probe allocates a fresh Extensions dictionary per folder, which
models a first scan. In production the same dictionary instance is carried forward by reference
when an unchanged folder is re-scanned — both DiskScannerEngine.cs:230 and
DiskOperationsService.cs:106 do Extensions = oldEntry?.Extensions — so a re-scan retains less
than a naive per-entry multiplication suggests. Treat 598 MB as the first-scan figure.

Cause A — the cache has no bound on entry count

core/backend/Engine/DiskScannerEngine.cs:26

public ConcurrentDictionary<string, CacheEntry> DirectorySizeCache = new(StringComparer.OrdinalIgnoreCase);

A plain dictionary, one entry per folder walked, with nothing capping the count. Only two things
remove entries, and neither is a size limit:

• PruneStaleCacheEntries (DiskScannerEngine.cs:619-636) — drops entries whose CachedAtUtc
  is older than ScanCacheTtlMinutes.
• EnforceMaxCacheScans (DiskScannerEngine.cs:643-673) — keeps the N most-recent scan
  roots and evicts whole older scans. Its own comment at :640 says so: // This is "5 most-recent scans", not "5 folders".

So one scan of one large drive is a single scan root, inside the TTL, and is retained in full
however many folders it contains.

Three things that look like they might already bound this, and do not:

1. MaxCachedNodes does not apply here. It changes runtime behaviour in exactly two places,
   core/backend/Services/ScanCacheService.cs:57 and :569, both passing it to CreateMemoryCache
   (ScanCacheService.cs:62-69) where it becomes SizeLimit on ScanCacheService's own private
   MemoryCache. It never reaches DirectorySizeCache. (It is also read at
   core/backend/Models/SettingDtos/AppSettingDto.cs:33 for range validation.) Worth noting
   separately: that SizeLimit is denominated in the per-entry Size unit set at
   ScanCacheService.cs:146 (Size = Math.Max(1, 1 + node.Files.Count)), not in nodes — so
   "MaxCachedNodes = 50000" does not mean 50,000 cached nodes.
2. EnforceMaxCacheScans silently exempts entries with a null or empty ScanRoot. The filter at
   DiskScannerEngine.cs:649 drops them from the grouping, so the scan cap can never evict them.
   core/backend/Services/DiskOperationsService.cs:102 writes entries of exactly that shape.
3. The restore path is unbounded. DiskScannerEngine.cs:67 assigns the entire deserialized file
   in one statement, before PruneStaleCacheEntries() (:70) and EnforceMaxCacheScans() (:71)
   run. Startup peak is therefore whatever the file holds, regardless of settings.

Separately and possibly out of scope: the DI AddMemoryCache() at core/backend/Program.cs:25 is
registered with default options and has no SizeLimit at all. It is shared by
FileTypeScanner.cs:14, AgeHeatmapEngine.cs:13 and ScanDiffService.cs:22. That is a second
unbounded cache.

Cause B — a full file path is retained per (folder, extension)

core/backend/Models/FileTypeScanResult.cs:8

public string LargestFilePath { get; set; } = string.Empty;

Written per folder at core/backend/Engine/DiskScannerEngine.cs:360 (fte.LargestFilePath = f.FullName;). Measured at 53% of retained cache size.

No consumer anywhere reads a per-folder value. It is read in exactly two places, both inside a
single aggregation in FileTypeScanner.BuildFromMemory, and both exist only to compute a maximum:

core/backend/Engine/FileTypeScanner.cs:218-222

if (ext.Value.LargestFileBytes > prev.LargestFileBytes)
{
    prev.LargestFileBytes = ext.Value.LargestFileBytes;
    prev.LargestFilePath  = ext.Value.LargestFilePath;
}

The winner is one path per extension per scan root. Every losing folder's path is discarded and
never reaches any output. The surviving value leaves the backend once, at FileTypeScanner.cs:118,
as ExtensionBreakdownItem.LargestFilePath (Models/FileTypeScanResult.cs:48) — which is a
different property on a different class from the cache field above.

Prior art — and where this collides with it

DirectorySizeCache, backed by scanner_memory.json, is the project's designated durable source
of truth for scanned sessions. File Type Analytics reads from it by design — an arrangement put in
place to fix a bug where analytics reported roughly 3.5 GB out of the 27.2 GB actually scanned. That
is why BuildFromMemory reads CacheEntry.Extensions first and only falls through to
IScanCacheService when the resulting map is empty:

core/backend/Engine/FileTypeScanner.cs:229-232

if (extMap.Count > 0)
{
    return extMap;
}

The precedence is pinned by FileTypeScannerTests.cs:347
(Analyze_PrioritizesDirectorySizeCache_WhenBothCachesExist).

This is settled architecture, not an accident, which has a direct consequence for Step 2 below.
Evicting cache entries to save memory removes exactly the data that arrangement depends on: fewer
entries means a partial Extensions map, which means File Types under-reports again — the original
bug, re-introduced as a side effect of a memory fix. Worse, it fails quietly: the fallback at
FileTypeScanner.cs:234-272 returns a smaller result, not an error, so nothing surfaces the loss.

Two corollaries for whoever picks this up:

• Step 2 cannot be treated as a pure memory optimisation. It changes the completeness guarantee.
• "Move File Types onto IScanCacheService, which already has a working SizeLimit" is not the
  intended direction, even though it looks like the obvious one. The source of truth stays where it
  is.

Step 1 does not have this problem. It reduces bytes per entry while keeping every entry.

Proposed direction

Not yet planned in detail. Two steps, in order — the first is cheap and self-contained, the second
needs design work and carries the risk described above.

Step 1 — store the file's name, rebuild the path during aggregation

Replace the retained full path with the file's name, and reconstruct the full path at aggregation
time from the dictionary key, which already is the owning folder's absolute path (written as
dir.FullName at DiskScannerEngine.cs:224 and :407). Expected saving: −53% retained, with no
loss of accuracy, no loss of entries, and no change to the wire contract.

The codebase already performs exactly this reconstruction 40 lines further down the same method:

core/backend/Engine/FileTypeScanner.cs:245

var filePath = Path.Combine(node.Path, file.Name);

Two things this is not:

• It is not a rename. FileTypeEntry has no name field today, so one has to be added.
• The rebuild must use the key of the entry that won the comparison, not whichever key is in scope
  after the merge — otherwise the winning file is attributed to the wrong folder. No existing test
  would catch that mistake: the multi-directory aggregation tests (FileTypeScannerTests.cs:98-119
  and :347) seed no LargestFilePath at all.

Step 2 — bound DirectorySizeCache by entry count

The obvious lever, and the dangerous one — see Prior art above. Any eviction policy has to preserve
File Type completeness and the >80% re-scan hit rate at the same time, so it needs its own design
and its own regression test before it can be scoped. Directions worth evaluating, none evaluated
yet: evict whole scan roots rather than individual folders, spill cold entries to disk, or aggregate
leaf folders beyond a depth threshold.

Recommend treating this as unscoped until Step 1 has landed and the remaining footprint has been
re-measured.

Constraints any fix must respect

• docs/notion/prd.md:81 — "Cache hit on re-scan | > 80% | Subsequent reads served from cache".
  Bounding the cache trades directly against this. An eviction policy has to be measured against the
  hit rate, not only against megabytes.
• FR-D3 (prd.md:137) "Cache scan results to heavily reduce CPU load on subsequent reads" and
  NFR-P4 (prd.md:356) "Subsequent re-scans must serve cached results where unchanged".
• docs/notion/techspec-v2.md:2145 — "percentOfDisk values are accurate relative to
  totalScannedBytes". File Types and Extension Breakdown accuracy is not tradeable for footprint.
• Reshaping CacheEntry is an on-disk format break, and it fails silently. The type is
  serialized whole to %APPDATA%/GSAnalyzer/scanner_memory.json (DiskScannerEngine.cs:428-429,
  path built at :36-38) and read back at :63. On a shape mismatch the catch at
  DiskScannerEngine.cs:79-83 logs "Cache file corrupted, starting fresh" at Warning level and
  then deletes the file. So an unversioned shape change degrades to total silent cache loss on
  upgrade — which in turn silently violates the 80% hit-rate target above. Version the file or
  migrate it.
• Blast radius of a CacheEntry change: 3 production construction sites
  (DiskScannerEngine.cs:224, DiskScannerEngine.cs:407, DiskOperationsService.cs:102), plus one
  deserialization site with no new token that a grep for new CacheEntry misses
  (DiskScannerEngine.cs:63), plus 11 test sites across 6 files.
• Step 1 does not touch the wire contract. The single backend assertion on the DTO field is
  core/backend/GSSystemAnalyzer.Tests/Services/FileExtensionBreakdownTest.cs:90, and it asserts
  ExtensionBreakdownItem.LargestFilePath — not the cache field — so with the rebuild in place it
  passes unchanged, as do the five Flutter consumers (extension_breakdown_model.dart:56,
  extension_breakdown_screen.dart:496, :506, :533, :562, and csv_exporter.dart:52). Note
  that two of those are behavioural rather than cosmetic: :533 copies the path to the clipboard and
  :562 strips it to a parent directory and triggers a rescan.

Scope note

The cache itself is required by FR-D3 and NFR-P4. The 130–170 MB idle target is not in the
PRD. Searches of docs/notion/prd.md for memory|footprint|idle|MB|NFR|non-functional|performance
and of docs/notion/techspec-v2.md for working set|footprint|idle memory return no requirement
bounding the application's own memory footprint — techspec-v2.md:3012-3061 covers alerting on the
machine's RAM, not the app's. The target originates from the user report behind #141. Worth
resolving before this is scheduled.

Proposed acceptance criteria

• Idle working set returns to the agreed baseline within seconds of a completed scan on a large
  drive, and stays there.
• Retained DirectorySizeCache size at 400,000 folders is reduced by at least 50%.
• File Type Analytics still reports the full scanned byte total — no regression of the
  completeness fix that made DirectorySizeCache the source of truth. This needs an explicit
  test; nothing today would fail if the number quietly shrank.
• Cache hit rate on re-scan still meets prd.md:81 (> 80%), measured, not assumed.
• File Types, Extension Breakdown and Age Heatmap remain accurate per techspec-v2.md:2145.
• scanner_memory.json either keeps its current shape or is versioned/migrated — no silent
  cache deletion on upgrade.
• Existing backend tests green, in particular FileTypeScannerTests, AgeHeatMapEngineTest,
  FileExtensionBreakdownTest, ScanCacheServiceTests.

Relationship to #141

#141 fixes the spike: peak working set 2,328 MB → 665 MB and Large Object Heap residue
1,049 MB → 12.5 MB at 400,000 folders, by replacing the write path at
DiskScannerEngine.cs:428-435. That path currently materializes two full copies of the cache
before touching disk — a complete Dictionary snapshot (new Dictionary<string, CacheEntry>(DirectorySizeCache)) and then the entire JSON string — writes that
string to a temp file, and atomically moves it into place.

#141 does not touch retained memory. This issue is that half.