perf: an archive names its directories once and encrypts into one buffer - #59
Merged
Merged
Conversation
donislawdev
force-pushed
the
perf/archive-prefix-and-buffers-v2
branch
from
September 6, 2026 04:35
7b75a1a to
249aa65
Compare
Two findings from the performance report, both in internal/format/archive and neither moving a byte. P9: Layout.Path rebuilt the directory chain for every entry, through a fmt format verb, though the chain depends only on the depth. Prefix() builds it and the two hot callers hoist it out of their loops. Measured at depth 50 over 10 000 entries: 82.2 ms before, 30.6 ms once the verb went, and close to nothing with the prefix built once. The default depth is zero, where the prefix is empty and none of this was ever paid - so this is the ceiling of a setting rather than a run anybody has. P10: the two locked-entry writers allocated a fresh buffer on every Write. At 128 MB in 32 kB blocks that is about four thousand allocations. The buffer now lives as long as the entry. It cannot be done in place: p belongs to the caller and the zip writer passes the same slice on, so scrambling it would corrupt what somebody else is about to read. That is written next to both writers. The report never measured P10 - entryWriter is unexported - and the clock does not settle it either, since the processor time ranges overlap. The collector count does, and it is deterministic: a 128 MB locked zip went from 48 collections to 6 with aes-256 and 7 with zipcrypto. No new guard, deliberately. Path length against LongestPath and the locked archive's exact size were both already guarded, and adding a fourth defence beside three is the shape this project has thrown away seven times. Three mutations prove those guards catch the broken version. One mutation pattern went stale in the same step, quoting the call this change rewrote, and staleness.py caught it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
donislawdev
force-pushed
the
perf/archive-prefix-and-buffers-v2
branch
from
September 6, 2026 04:43
249aa65 to
f6997b1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sixth chunk of the 2026-09-05 performance report:
P9andP10. Neither moves a byte.P9- the directory chain, built onceLayout.Pathrebuilt the chain for every entry, through afmtformat verb, though it depends only onDepth.Prefix()builds it, and the two hot callers hoist it out of their loops.fmt.Fprintfper entry)So more than half of it was the format verb, which the report did not separate from the loop.
defaultDepthis 0, and at zero the prefix is empty - a flat archive never paid any of this. This is the ceiling of a setting, not a run anybody has.P10- one buffer per entry, not per writeBoth locked-entry writers allocated a fresh
make([]byte, len(p))on everyWrite. At 128 MB in 32 kB blocks that is roughly four thousand allocations.🔴 It cannot be done in place, and that is the only reason a buffer exists at all:
pbelongs to the caller and the zip writer passes the same slice on, so encrypting it where it lies would corrupt what someone else is about to read. Written next to both writers.Nobody had measured this -
entryWriteris unexported, so a probe outside the package cannot reach it - and the clock does not settle it either: the processor-time ranges overlap. The collector count does, and it is deterministic:System time fell visibly too (203 → 78 ms, 234 → 141 ms) and the wall clock slightly (585 → 570, 630 → 597). I do not claim a processor-time figure.
No new guard, on purpose
Both are already guarded:
archivedepth_test.gocompares path length against theLongestPatharithmetic, andarchivelock_test.gochecks exact size, a real archiver opening it, and determinism. Adding a fourth defence beside three is the shape this project has removed seven times.Instead, three mutations prove those guards catch the broken version: lost zero-padding in a level name, and - for both writers - the buffer handed on at full length instead of the length written. All caught.
layout.Path(...)call that was rewritten).staleness.pycaught it; retargeted and re-proven.Verification
go test -tags "$(cat .github/build-tags)" ./...- green, 83 packagespython tools/preflight.py --quick- all 12 checks passtry-named.pyon both guards - 5 mutations, all caught🤖 Generated with Claude Code