Skip to content

perf: an archive names its directories once and encrypts into one buffer - #59

Merged
donislawdev merged 1 commit into
mainfrom
perf/archive-prefix-and-buffers-v2
Sep 6, 2026
Merged

perf: an archive names its directories once and encrypts into one buffer#59
donislawdev merged 1 commit into
mainfrom
perf/archive-prefix-and-buffers-v2

Conversation

@donislawdev

Copy link
Copy Markdown
Owner

Sixth chunk of the 2026-09-05 performance report: P9 and P10. Neither moves a byte.

⚠️ Stacked on #58 (perf/plan-without-encoding), so the base is that branch rather than main. It has to be: docs/, tools/ and CLAUDE.md are single repositories with no branches, so a branch cut from main fails TestEveryGuardIsEitherProvenByMutationOrListedAsNotProven and TestEveryGuardCitedInTheSummaryIsJustifiedInRegression for guards that exist only on #58. Merge #58 first and this collapses to its own three commits.

P9 - the directory chain, built once

Layout.Path rebuilt the chain for every entry, through a fmt format verb, though it depends only on Depth. Prefix() builds it, and the two hot callers hoist it out of their loops.

variant median, depth 50, 10 000 entries
before (fmt.Fprintf per entry) 82.2 ms
the verb removed, still per entry 30.6 ms
prefix built once (what the code does) ~0 ms

So more than half of it was the format verb, which the report did not separate from the loop.

⚠️ defaultDepth is 0, and at zero the prefix is empty - a flat archive never paid any of this. This is the ceiling of a setting, not a run anybody has.

P10 - one buffer per entry, not per write

Both locked-entry writers allocated a fresh make([]byte, len(p)) on every Write. At 128 MB in 32 kB blocks that is roughly four thousand allocations.

🔴 It cannot be done in place, and that is the only reason a buffer exists at all: p belongs to the caller and the zip writer passes the same slice on, so encrypting it where it lies would corrupt what someone else is about to read. Written next to both writers.

Nobody had measured this - entryWriter is unexported, so a probe outside the package cannot reach it - and the clock does not settle it either: the processor-time ranges overlap. The collector count does, and it is deterministic:

GODEBUG=gctrace=1, 128 MB locked zip
aes-256     48 collections -> 6
zipcrypto   48 collections -> 7

System time fell visibly too (203 → 78 ms, 234 → 141 ms) and the wall clock slightly (585 → 570, 630 → 597). I do not claim a processor-time figure.

No new guard, on purpose

Both are already guarded: archivedepth_test.go compares path length against the LongestPath arithmetic, and archivelock_test.go checks exact size, a real archiver opening it, and determinism. Adding a fourth defence beside three is the shape this project has removed seven times.

Instead, three mutations prove those guards catch the broken version: lost zero-padding in a level name, and - for both writers - the buffer handed on at full length instead of the length written. All caught.

⚠️ One mutation pattern went stale from this change (it quoted the layout.Path(...) call that was rewritten). staleness.py caught it; retargeted and re-proven.

Verification

  • full go test -tags "$(cat .github/build-tags)" ./... - green, 83 packages
  • python tools/preflight.py --quick - all 12 checks pass
  • try-named.py on both guards - 5 mutations, all caught

🤖 Generated with Claude Code

@donislawdev
donislawdev force-pushed the perf/archive-prefix-and-buffers-v2 branch from 7b75a1a to 249aa65 Compare September 6, 2026 04:35
Base automatically changed from perf/plan-without-encoding to main September 6, 2026 04:43
Two findings from the performance report, both in internal/format/archive
and neither moving a byte.

P9: Layout.Path rebuilt the directory chain for every entry, through a
fmt format verb, though the chain depends only on the depth. Prefix()
builds it and the two hot callers hoist it out of their loops. Measured
at depth 50 over 10 000 entries: 82.2 ms before, 30.6 ms once the verb
went, and close to nothing with the prefix built once. The default depth
is zero, where the prefix is empty and none of this was ever paid - so
this is the ceiling of a setting rather than a run anybody has.

P10: the two locked-entry writers allocated a fresh buffer on every
Write. At 128 MB in 32 kB blocks that is about four thousand allocations.
The buffer now lives as long as the entry. It cannot be done in place:
p belongs to the caller and the zip writer passes the same slice on, so
scrambling it would corrupt what somebody else is about to read. That is
written next to both writers.

The report never measured P10 - entryWriter is unexported - and the clock
does not settle it either, since the processor time ranges overlap. The
collector count does, and it is deterministic: a 128 MB locked zip went
from 48 collections to 6 with aes-256 and 7 with zipcrypto.

No new guard, deliberately. Path length against LongestPath and the
locked archive's exact size were both already guarded, and adding a
fourth defence beside three is the shape this project has thrown away
seven times. Three mutations prove those guards catch the broken version.

One mutation pattern went stale in the same step, quoting the call this
change rewrote, and staleness.py caught it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@donislawdev
donislawdev force-pushed the perf/archive-prefix-and-buffers-v2 branch from 249aa65 to f6997b1 Compare September 6, 2026 04:43
@donislawdev
donislawdev merged commit 5910e69 into main Sep 6, 2026
35 checks passed
@donislawdev
donislawdev deleted the perf/archive-prefix-and-buffers-v2 branch September 6, 2026 04:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant