Skip to content

sync: batch small files, skip unchanged manifests, probe presence on its own - #26

Open
ChakraFusion wants to merge 3 commits into
Liquid-co:mainfrom
ChakraFusion:pr/sync-speed
Open

sync: batch small files, skip unchanged manifests, probe presence on its own#26
ChakraFusion wants to merge 3 commits into
Liquid-co:mainfrom
ChakraFusion:pr/sync-speed

Conversation

@ChakraFusion

@ChakraFusion ChakraFusion commented Oct 4, 2026

Copy link
Copy Markdown

Part of #21.

Based on the "Keep mine" PR (#22): its first commit is that PR. Only the last two commits are new here. I'll rebase once that PR is merged.

Why

A save of many small files (Project Zomboid: ~240k map chunks) cost one HTTP request per file. That is ~200 ms each on a real LAN, so a pull of 50 MB took hours. On top of that, every periodic pass fetched and compared the full manifest for every game, even when nothing had changed.

What

Batched pulls (protocol revision 2)

  • Small files are fetched up to 256 per request (4 MB), with 4 requests in flight, over a new /api/p2p/files route.
  • Every file is still written through the PatchWriter (verified, fsynced, renamed).
  • On loopback, 5000 files went from 42 s to 12 s.
  • Peers below revision 2, and relay connections, keep the per-file path.
  • A batch is bounded by what the responder counts, so a batch of small files is never refused as too large. A batch refused anyway is fetched again as two halves.

Unchanged games

  • When both devices still hold the hash they last agreed on, the manifest request carries ifHash and the peer answers "unchanged" instead of sending the manifest.
  • Older peers ignore the parameter.

Gzip and timeouts

  • Manifests are gzipped for clients that accept it, about 5.6× smaller.
  • Manifest and batch requests get a 10-minute timeout instead of 30 s. Over a VPN, a large manifest never arrived within 30 s, so that game never synced.

Smaller items

  • An in-sync pass also mirrors the peer's latest snapshot, so games that were identical from the start show history on every device.
  • Games whose save folder isn't on this device are skipped instead of retried every 20 s.
  • "syncing X with Y" is logged only when there's something to do.

Presence (second commit)

  • Peers are probed every 10 s on their own ticker, and once immediately at start.
  • A peer not heard from since start goes offline on its first missed probe, so a PC that was switched off overnight is no longer shown online after every start.

Compatibility

Mixed versions work in both directions:

  • Revision-2 peers batch.
  • Older peers get and send per-file requests and full manifests.
  • Gzip is negotiated through Accept-Encoding.

Tests

  • batch_test.go: small files are batched, older peers are not, a batch stays under what the responder counts, and a refused batch is split.
  • quick_test.go: the unchanged shortcut, real changes still sync, and an older peer's manifest is not fetched twice.
  • manifest_gzip_test.go, presence_test.go, missing_skip_test.go.
  • partial_safety_test.go: an interrupted batched pull never turns into deletions.
  • e2e/batch_bench_test.go: opt-in measurement.
  • CI on the fork (all jobs green): https://github.com/ChakraFusion/OpenSave/actions/runs/37162978426

🤖 Generated with Claude Code

ChakraFusion and others added 3 commits October 4, 2026 01:29
@ChakraFusion @claude
sync: "Keep mine" no longer deletes files; hold syncs that would dele…
97fdd89 
…te a large part of a save

"Keep mine" (and "Keep both", which ends the same way) recorded every local
file as shared with the peer before the peer had pulled any of it. When that
pull did not finish - a large save, a request that timed out, a device that
went away - the next sync read the files the peer was missing as deletions
made over there, and deleted them on the device whose version had just been
kept. On a save of ~240k files this removed tens of thousands; the folders
then differed again, the conflict came back, and the same answer did it
again.

- markResolvedLocal records as shared only what both sides verifiably hold
  (the intersection of the two manifests). A file only this device has is
  new to the peer, not deleted by it. If the peer cannot be asked, nothing
  is recorded as shared. Same for extra save locations
  (ResolveRootConflict).
- A sync that would delete more than 100 files and more than 10% of a
  save, on either side, is held as a conflict instead of applied. A deletion
  confirmed on purpose (the emptied-folder flow) is not held.
- The conflict carries uncapped counts of what differs (diffFiles stops at
  100), and the dialog says what each answer does: "Keep mine"/"Keep both"
  delete nothing; "Keep theirs" deletes N files only this device has, shown
  in red when that is many. Older builds without the counts fall back to
  counting the capped list.

TestKeepLocal_PeerThatNeverPulled_DeletesNothingHere fails on v2.4.1
(300 files -> 100); either change alone makes it pass.
Also adds tests that an interrupted pull or a peer holding part of a save
never turns into deletions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ChakraFusion @claude
sync: batch small files, skip unchanged manifests, probe presence on …
fe9ce64 
…its own

Performance
- Batched pulls (protocol revision 2): small files are fetched up to 256
  per request (4MB), 4 requests in flight, over a new /api/p2p/files
  route. Each file is still written through the PatchWriter (verified,
  fsynced, renamed). One request per file cost ~200ms on a real LAN; on
  loopback 5000 files went from 42s to 12s. Peers below revision 2 and
  relay connections keep the per-file path.
- Unchanged games: when this device still holds the agreed base, the
  manifest request carries ifHash and a peer that still holds it too
  answers "unchanged" instead of sending the manifest. Older peers ignore
  the parameter; their full answer is used, not fetched twice.
- Manifests are gzipped for clients that accept it (every Go client does,
  so older versions read them too): ~5.6x smaller with real hashes.
- Manifests and batches get a 10-minute client timeout instead of 30s; a
  large manifest over a VPN never arrived inside 30s, so those games
  never synced.
- An in-sync pass also mirrors the peer's latest snapshot, which only a
  pull did before; games identical everywhere from the start showed
  history on one device only.
- A batch is bounded by what the responder counts (each requested block at
  the file's full block size, at most the file's real size), so a batch of
  many small files with a few larger ones is never refused as too large; a
  batch refused anyway is fetched again as two halves.

Robustness
- Presence: paired LAN peers are probed every 10s on their own ticker.
  Probes used to ride along with the minute-long reconcile, which waits
  for a whole sync-all pass; peers reached over a VPN (no UDP broadcast)
  showed online/offline minutes late. Probe rounds are serialized.
- Games whose save folder is not on this device are skipped by sync-all
  and dropped from the retry queue instead of failing every 20s.
- "syncing X with Y" is logged only when there is something to do; most
  passes produced only "already in sync" lines, which rotated everything
  else out of the log within hours.
- The interrupted-pull safety test also runs over batched pulls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ChakraFusion
presence: do not trust "online" left over from the previous run
cf9269a 
Every peer started each run with whatever status the database held, and a
peer counts as offline only after three failed probes. For a PC switched
off overnight that meant being shown online after every start until three
probes - the first ten seconds after the background loops started - had
failed.

- The presence loop probes at once on start, then every 10s.
- The three-miss tolerance is for devices heard from during this run
  (answered a probe, or seen by discovery, a request or the relay since
  start). A peer with no sign of life since start goes offline on its
  first unanswered probe.

Adds an opt-in e2e measurement (OPENSAVE_PRESENCE_TIMING) of how long a
stopped device is shown online: 30s, by design, for one seen this run.
@vercel

vercel Bot commented Oct 4, 2026

Copy link
Copy Markdown

@ChakraFusion is attempting to deploy a commit to the sivadaboi's projects Team on Vercel.

A member of the Team first needs to authorize it.

This was referenced Oct 4, 2026
Open
Open

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ChakraFusion