#269: assembling a target held three copies of it, not one — 3.06x peak to 1.13x

[Polichinel]

Measured before changing anything, and the issue's diagnosis turned out to be a third of the story.

What the measurement showed

A synthetic run through the real TargetLease.load — producer-format .tap.zip shards, real read_shard, tracemalloc and peak RSS agreeing within 2% — at 36 shards x 20,000 cells x 200 samples:

	before	after
peak, tracemalloc	3.06x the delivered frame	1.13x
peak, RSS delta	3.02x	1.06x

The 3x was three roughly equal thirds, and #269 named only the first:

1. every shard's bytes, resident together — the dict comprehension completed before the first was decoded;
2. every decoded per-shard frame, held for the stack;
3. the np.concatenate result, allocated while (2) was still alive.

The ratio held at both 12 and 36 shards, so it is the shape and not the scale. Fixing only the part the issue described would have reached about 2x.

Two false starts, recorded because both would have produced a confident wrong number

My first harness had the fake store return pre-built objects, so the download allocation fell outside the measurement window and peak read 2.04x. The obvious correction — return bytes(blob) — allocates nothing, because bytes(b) is b when b is already bytes; and so does b"" + b. It took a bytearray to get a genuine copy. The 3.06x only appeared once the store actually allocated, which is what a real download does.

The fix is one loop

frames_for_target takes fetch_shard_bytes(name) instead of a filled dict, drops each shard's bytes the moment they are decoded, and writes each shard into a manifest-sized buffer instead of stacking and concatenating. Peak is now the finished frame plus about one shard — the residual 0.13x is 2/n_shards, which is why 12 shards measures 1.36x and 36 measures 1.13x.

The buffer's slots are sized from expected_cell_count, a declaration this function already enforced per shard, and the enforcement runs before anything is written — so a shard whose row count disagrees is refused rather than straddling two months' slots.

Nothing about the product changed

The wire is byte-frozen (ADR-013) and tests/test_wire_fixture.py compares delivered artifacts against checked-in bytes. It passes unchanged, which is the whole claim: an assembly change with no output change.

At production scale

64,742 cells x 36 months, at ADR-013 §5's "~1000 samples per cell" — the one input taken from the contract rather than measured — one target's frame is 8.68 GB, so peak falls from about 26.6 GB to 9.8 GB per target, roughly 16.8 GB saved. Run-0's OOM kill recorded anon-rss:23778224kB; that incident's root cause was pandas on the historical leg and is not this, but the magnitude says this leg alone would have exhausted the same box.

The manager's historical frame is not the elephant

#269 also notes _historical_frame is held from _read through _save. By its declared dimensions — 64,742 cells x 438 months = 28,356,996 rows — that is about 108 MB at one float32 column, 1.2% of a single forecast target frame. Recorded in C-101 rather than filed as its own issue: a separate issue implying comparable cost would misdirect whoever picked it up.

Guarded

test_shards_are_fetched_one_at_a_time_not_all_up_front asserts the fetch/decode interleaving, not a byte count — a memory threshold in a test is a flake on a busy machine, while "fetch, decode, fetch, decode" is exactly the property that bounds the peak. Mutation-proven: restoring the up-front dict yields ['fetch','fetch','fetch','decode','decode','decode'] and it fails.

The register's own reference guard also caught me inventing a cross-reference to a register entry C-126 that does not exist — the 126 is an issue number.

Verification

ruff clean; 450 passed, 1 skipped, 39 xfailed.

Closes #269.