[diskann-garnet] Update data design doc

[Jack Moffitt (metajack)]

This update corrects some errors and gives specific key and value sizes for every term stored in Garnet.

[Copilot]

Pull request overview

Updates the diskann-garnet data design documentation to correct term descriptions and to provide more explicit key/value sizing for the data stored in Garnet.

Changes:

• Clarifies key prefixing/scratch-space expectations for Garnet operations.
• Documents key/value sizing for major term types (vectors, neighbor lists, quantized vectors, attributes, metadata, ID mappings).
• Updates internal-term descriptions (start point behavior, FSM representation, quantizer state storage).

Suppressed comments (2)

diskann-garnet/docs/data-design.md:129

• In the External ID Mapping section, "Interal" is misspelled and the note says "this key" when it refers to the value size. Also, describing IDs as a "bitstring" is misleading since these are byte strings.

*Key*: External ID bytes; this key is a variable length bitstring that the user assigned.
*Value*: Interal ID as bytes; this key is always 4 bytes in length.

diskann-garnet/docs/data-design.md:120

• Similar to other sections, this describes the external ID as a "bitstring". Since external IDs are arbitrary byte strings, using "byte string" is clearer and consistent with the rest of the document.

*Value*: External ID; this is variable length bitstring that the user assigned.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.55%. Comparing base (3218478) to head (df1ae7c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1328      +/-   ##
==========================================
- Coverage   91.55%   91.55%   -0.01%     
==========================================
  Files         522      522              
  Lines       99541    99541              
==========================================
- Hits        91139    91133       -6     
- Misses       8402     8408       +6     

Flag	Coverage Δ
miri	91.55% <ø> (-0.01%)	⬇️
unittests	91.23% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Harsha Simhadri (harsha-simhadri)]

Copilot please check if there are any remaining discrepancies between the updated design document and code

[Copilot]

Copilot please check if there are any remaining discrepancies between the updated design document and code

I checked the doc against the code. One genuine discrepancy: the Neighbor Lists section said max_neighbors "accounts for the graph slack factor," but in diskann-garnet the configured max degree M is the true maximum (see provider.rs max_degree doc comment). The slack factor instead lowers the build target (target_degree = M / GRAPH_SLACK_FACTOR in lib.rs), so the stored size is (M + 1) * 4 and the (16 + 1) * 4 = 68 example is correct. Fixed the description accordingly.

Everything else matches the code: 4-byte internal-ID keys and the dimension * size_of::<T>() vector value; FSM blocks are 8 kB / 64k IDs each with 1-bit-per-ID states and 8-byte _fsm keys; _qnt is a 4-byte key with the 1-byte backfill flag only for BIN (Q8 stores the table without it); BIN quantized-vector value is 1 bit/dim + up to 6 bytes overhead and Q8 is 20 bytes + 1 byte/dim; attributes are keyed by external ID; and the internal/external ID mappings and the start point at internal ID 0 all match.