Repository navigation
Faster manifests and snapshots for saves of many files - #25
Open
ChakraFusion wants to merge 2 commits into
Open
ChakraFusion wants to merge 2 commits into
ChakraFusion wants to merge 2 commits into
Conversation
On a save of ~240k small files (Project Zomboid map chunks) a repeat manifest build took 13-19s in 2.4.0; now ~1.3s, with an identical hash. - Walk with filepath.WalkDir instead of filepath.Walk. Walk's extra Lstat per entry opens every file on Windows and dominated a warm build. - Reuse block read buffers (sync.Pool) instead of allocating 64KB+ per file: a cold build allocated ~15GB of garbage, now ~0.6GB. - Sort with sort.Slice when evicting from the hash cache. The insertion sort was quadratic in the entry count (map order is random). - Raise the cache budget floor to 256MB and scale it at 1MB per game, ceiling 512MB. 64MB was filled by one large save on its own, so the cache evicted and re-read that save on every pass. A budget, not an allocation: small libraries never come near it. - Share the whole-file hash string with the block hash for single-block files (most save files). - FileEntryForInfo: a cache lookup with the FileInfo a walk already has.
…compressing them again Every snapshot stays a complete zip that restores on its own. What changes is how it is written: a file whose content (SHA-256, as already recorded per snapshot and held by the hash cache) matches the previous snapshot of the same branch is copied in as its already-compressed entry (zip.Writer.Copy). Only new and changed files are compressed. On a 204k-file save: a full snapshot 80.5s, a follow-up 8.9s, same size. The archive walk also moves to filepath.WalkDir for the same reason as the manifest walk.
|
@ChakraFusion is attempting to deploy a commit to the sivadaboi's projects Team on Vercel. A member of the Team first needs to authorize it. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #21.
Measured on Project Zomboid (~240k small map-chunk files).
1. delta: hash cache and manifest walk that scale
A repeat manifest build went from 13–19 s to ~1.3 s, with an identical hash.
filepath.WalkDirinstead offilepath.Walk. Walk's extraLstatper entry opens every file on Windows, and that dominated a warm build.sync.Pool). A cold build allocated ~15 GB of garbage; it now allocates ~0.6 GB.sort.Slice. The old insertion sort was quadratic in the entry count.2. snapshot: copy unchanged files from the previous snapshot instead of compressing them again
Every snapshot is still a complete zip that restores on its own. Only the way it is written changes. A file whose SHA-256 matches its entry in the branch's previous snapshot is copied in as its already-compressed entry (
zip.Writer.Copy), so only new and changed files are compressed.On a 204k-file save, a full snapshot takes 80.5 s and a follow-up 8.9 s, with the same size.
Tests
reuse_test.go: a follow-up snapshot reuses unchanged entries and restores byte-identical.reuse_bench_test.go: an opt-in benchmark.go test ./...passes. CI on the fork: https://github.com/ChakraFusion/OpenSave/actions/runs/37162381473 (all green; one e2e shard needed a re-run forTestJourney_DeviceSpecificConfigNeverTravels, which also fails intermittently on plain v2.4.1)🤖 Generated with Claude Code