Reduce sync work: faster snapshot projection + benchmarks#86
Draft
hahn-kev-bot wants to merge 13 commits into
Draft
Reduce sync work: faster snapshot projection + benchmarks#86hahn-kev-bot wants to merge 13 commits into
hahn-kev-bot wants to merge 13 commits into
Conversation
Previously AddSnapshots projected each snapshot by calling FindAsync per entity, which issued one database query per snapshot (and, on an initial sync of new data, every query returned null after a round-trip). Pre-load the projected rows that already exist for the batch with a single tracked query per object type. ProjectSnapshot then resolves existing entities from the change tracker and skips the lookup entirely for entities that have no projected row yet, collapsing N queries down to roughly one per distinct object type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zEmz7jPRPF6Lv8h6YAWBW
Adds a BenchmarkDotNet suite that measures CrdtRepository.AddSnapshots on its own, across 7 workloads mirroring DataModelSyncBenchmarks. Expensive DB seeding runs once in a template DB; each iteration forks the DB and recomputes the snapshot batch so no EF-tracked state leaks across iterations. - SnapshotWorker.ComputeSnapshotsToPersist: returns the exact snapshot list UpdateSnapshots would persist, without writing it. - DataModelTestBase: internal CreateRepository() and CrdtConfig accessors. - BenchmarkWorkloadBuilders: shared commit builders extracted from DataModelSyncBenchmarks (+ BuildUpdateExisting). - Program.cs: run both suites via BenchmarkSwitcher (handles --filter/args). - Remove leftover Console.WriteLine debug lines from AddSnapshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two experimental fast AddSnapshots implementations that keep the EF snapshot insert unchanged but populate projected tables with raw INSERT ... ON CONFLICT upserts instead of going through EF's change tracker: - FAST: one upsert command per entity row - FAST_JSON: one command per entity type, rows passed as a single JSON array expanded with SQLite json_each/json_extract FastProjection derives table/column names, primary key, the SnapshotId shadow FK, and value converters from the EF model (no per-entity code). It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction. CrdtRepository.AddSnapshots now selects via #if FAST_JSON / #elif FAST / #else. Program.cs adds a third FAST_JSON benchmark job and DataModelSyncBenchmarks enables [MemoryDiagnoser]. Benchmarks (CreateWords, 1000): both fast paths ~35% faster and ~32% fewer allocations than baseline; per-query vs JSON-batch shows no measurable difference against in-memory SQLite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The JSON-batch projection benchmarked identically to the per-query path against in-memory SQLite (same time and allocations), so remove it and keep only the per-query raw-SQL upsert path. FastProjection loses the useJsonBatch parameter and all json_each/json_extract code; CrdtRepository.AddSnapshots collapses to #if FAST / #else; the benchmark drops the FAST_JSON job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the EF change-tracker slow path and the #if FAST conditional so AddSnapshots always uses FastProjection. Deletes the now-dead slow-path helpers (ProjectSnapshot, GetEntityEntry, LoadExistingEntityIds, LoadExistingEntities). The benchmark collapses to a single job since FAST vs DEFAULT are now identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FastProjection is now an injected singleton (registered in AddCrdtDataCore and resolved into CrdtRepository via ActivatorUtilities) instead of a static class. Its per-type projected-table SQL metadata cache moves from a static field onto an internal ConcurrentDictionary on CrdtConfig, so it's shared across repositories/contexts and tied to config lifetime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Reduces the work done during sync (
AddRangeFromSync→SnapshotWorker.UpdateSnapshots→CrdtRepository.AddSnapshots) and adds a benchmark suite to measure it. On theCreateWordsworkload at 10k changes this branch takes sync from ~1.96 s / 976 MB allocated down to ~0.99 s / 538 MB — roughly 2× faster and ~45% less memory.What changed
Snapshot pre-load (
SnapshotWorker/DataModel)UpdateSnapshotsnow bulk-loads the relevant current snapshots (with theirCommit) into a cache keyed by entity id, andSnapshotWorkerreads full snapshots straight from that cache instead of issuing aFindSnapshotDB round-trip per cache hit.Fast raw-SQL projection (
FastProjection, new)INSERT ... ON CONFLICT(pk) DO UPDATE(one upsert per entity row) instead of going through EF's change tracker (FindAsync/SetValues/ graph tracking).SnapshotIdshadow FK, value converters — is derived from the EF model, so there is no per-entity code.FastProjectionis an injectable singleton; its per-type SQL metadata cache lives on an internalConcurrentDictionaryonCrdtConfig, shared across repositories/contexts. This replaces the previous EF change-tracker projection path inCrdtRepository.AddSnapshots, which is removed.Benchmarks (new
SIL.Harmony.Benchmarksproject)DataModelSyncBenchmarks(7 sync workloads) and anAddSnapshotsBenchmarksthat isolates the persist step,[MemoryDiagnoser]enabled. Run withdotnet run -c Release --project src/SIL.Harmony.Benchmarks.Testing
DataModelPerformanceBenchmarkstiming-threshold tests, which also fail onmain(environmental, not caused by this change).Notes for reviewers
ON CONFLICTafter aSELECTneeds the SQLiteWHERE truedisambiguator in the code history — the current per-query path usesVALUES). If other providers are ever targeted,FastProjectionwould need revisiting.Word.AntonymIdpointing at another newWordin the same batch) is not ordered; it's a nullableSET NULLFK and not exercised by current workloads.