[yuejiaointel]

Two changes for the concurrent (FINE_GRAIN) dynamic index, both on the save/load path:

1. Save without cleanup. save() no longer runs consolidate() + compact(). It writes the slots as they are (holes and soft-deleted IDs included) in a new format version 0.0.1.
   Why: compact moves every vector. When a caller saves from a forked child (e.g. a background snapshot), every page is then copied-on-write, so peak memory roughly doubles while saving.
2. Load a stream as the concurrent index: svs_index_load_stream_dynamic_ex. The C API could already save the concurrent index to a stream, and load it back as concurrent from a directory (svs_index_load_dynamic_ex), but loading from a stream always returned the non-concurrent index. The new function takes the same svs_dynamic_index_params_t as the directory version, so sync_kind = FINE_GRAIN restores the concurrent index.

Changes

• Concurrent save (include/svs/concurrent/dynamic_index.h): version 0.0.1; no consolidate/compact; writes one status byte per slot (Empty/Valid/Deleted) and the ID translator as is, so soft-deleted IDs stay mapped to their Deleted slots (the live index keeps them until consolidate). Throws on a Pending slot.
• Writes blocked during save, searches not: a new writers-only lock (outermost, with a gatekeeper like compact_gatekeeper_ so save isn't starved). add_points, delete_entries, consolidate, compact and replace_external_id take it shared; save takes it exclusive. Searches never take it.
• Loaders (both index types, directory and stream): accept 0.0.0 (all Valid, as before) and 0.0.1; stream loaders now check the version and reject unknown ones.
  • Concurrent loader keeps the soft-deleted IDs (same state as the live index right after delete_entries, so e.g. consolidate(ids) works after reload).
  • Non-concurrent loader drops them, matching its own delete (which erases the mapping immediately).
• Non-concurrent index save: unchanged (still consolidates/compacts, still 0.0.0). Doing the same for it is a follow-up.
• C API: new svs_index_load_stream_dynamic_ex; IndexBuilder::load_stream_dynamic picks the concurrent or plain dispatcher by sync kind (like load_dynamic), with a stream source added to the concurrent dispatcher. svs_index_load_stream_dynamic is unchanged (NONE). The FINE_GRAIN wrapper's save takes the shared lock, so searches keep running.
• Docs: include/svs/concurrent/README.md (locks), bindings/c/docs/C_API_Design.md, svs_c.h comments, data/serialization.toml (v0.0.1).

Compatibility

• New SVS reads old 0.0.0 files: yes, in all loaders.
• Old SVS reading new 0.0.1 files: no — it fails with an error (no silent misread in the cases tested). Only matters for a rollback after saving, or mixed versions.
• Files are larger: they keep holes (every slot is written, not just live vectors).
• Save no longer cleans the live index as a side effect, and a loaded index keeps its holes until the caller runs consolidate()/compact().

Note for callers that save from a forked child

save() takes the writers lock exclusive. In a forked child, a writer that was mid-call at fork() still holds that lock (its thread doesn't exist in the child), so the child's save() waits forever. Stop writers before forking. This is the same on main, where save's compact() waits on the storage lock.

Relation to update_entries (#467)

update_entries is a writer and must also take the writers lock (one line: auto writers_lock = lock_writers_shared(); before lock_storage_shared()). It isn't on main yet, so whichever of #467/#470 merges second adds that line. Tested locally with #468/#467 merged in: after save + svs_index_load_stream_dynamic_ex(FINE_GRAIN), svs_index_dynamic_update_points on a deleted ID reuses the same slot (no new hole, no growth), and 10 saves during a loop of update_entries all returned and reloaded.

Testing

• Full test suite, main vs this branch, same build/flags, declared order: both 202 test cases | 201 passed | 1 failed. The one failure, tests/svs/core/io/binary.cpp:138 ("Directory not empty"), fails the same way on main; an intermittent SIGSEGV at tests/svs/index/vamana/index.cpp:95 also occurs on main and passes on rerun.
• C API public build: 100% tests passed, 0 tests failed out of 26. pre-commit clean.
• Locally (tests not included in this PR): save with Deleted + Empty slots, reload into both index types (directory and stream): same statuses and search results, soft-deleted IDs kept (concurrent) / dropped (non-concurrent), source unchanged; 0.0.0 files still load; unknown version and corrupt status byte rejected; save while adding/searching; svs_index_load_stream_dynamic_ex with FINE_GRAIN / NULL / old function → FINE_GRAIN / NONE / NONE.
• Cross-version: files saved by main load in all four new loaders; 0.0.1 files fail with an error in main's loaders; non-concurrent saves still load in main.
• Fork copy-on-write check (200k × 128 float, 10% deleted, save to stream in a forked child): child private dirty memory grows by 665 MB on main vs 0.7 MB with this change.

Not in this PR

• Non-concurrent index save without cleanup (follow-up).
• concurrent/multi.h has its own save that still consolidates/compacts.
• A Pending slot left by an add_points that throws mid-way (existing behavior) makes later saves throw.
• An index with zero vectors can't be reloaded (same on main).

🤖 Generated with Claude Code