Repository navigation
warren-core: TextKeys, the id map of text ids (M10 step 3) - #58
Merged
Merged
Conversation
… the base A text key storage needs what the fixed one did not: the bytes the fresh keys span, to reserve its arrays exactly; the build memory while it renumbers, since it moves every id's bytes; and a rebuild that consumes its base, to extend the base's arrays in place (DESIGN.md §5.7). IdMap::rebuild_into takes over the base's charge, so the result's charge is exactly what the new id map holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Three arrays (DESIGN.md §5.7, §14 item 41): id_bytes and id_offsets in dense-id order and id_order, the dense ids sorted bytewise at the target width. Dense to external is two offsets and a slice; external to dense is a binary search of id_order. A rebuild takes its node ids in an arena (TextIds) charged to the build memory, sorts an index array by bytes, merge-walks it against the base order, and appends the fresh ids to the base's bytes and offsets, in place when it consumes the base. Renumbering moves bytes and offsets and maps the order; compaction copies the live ids with an identity order. Shared views of the three arrays are validated: offsets from 0, monotone, ending at the bytes; the order a permutation and strictly increasing by bytes. TextIndex is the build's hash index over the live ids, two slots a node, SipHash keyed per index, hits confirmed by bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Three new segment kinds (8, 9, 10) for a text id map (DESIGN.md §5.7). id_bytes records the database encoding in header byte 7, which stays 0 and checked for every other kind; decoding refuses another encoding, and check_encoding checks a header read in place. Decoding validates the offsets and the order as from_text_segments does, and decode_text_within charges the build memory first, as the fixed id map's decode_within does. A new fuzz target, decode_text_id_map, joins CI's fuzz job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
The version fixture builds through a Keying: int8 ids, or each node as the nth of some text ids in byte order, the order-preserving bijection from a text id to its rank. Through random histories of rebuilds that drop and bring back nodes, a text-keyed graph has the int-keyed graph's dense ids, tombstones and CSRs, the same diff and shared-dense-id check, and the same compaction in either node order (DESIGN.md §5.7, Testing). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Distance between two text graphs aligns them by a merge of their id_orders (DESIGN.md §5.7), O(N) with no binary search per node, and the fixed types gain the same. A node tombstoned in this version stays the other version's own, as before; a test pins that, and text versions are checked to align as the int versions of their ranks do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
…r checks - TextKeys::compacted refuses dense ids out of id order or repeated (IdMapNotSorted), as the fixed keys do. - IdMap::rebuild_into's returned charge is always on the ids' budget and exactly the new map's bytes: a charge on another budget is released and the map charged in full, a short one topped up (an error if it does not fit). The precondition, and that the base is dropped on an error, are documented. - Text decoding checks the tombstones against the offsets' id space before the order, and decode_text_within charges the order check's bitmap by that id space. - check_id_bytes_header checks an id_bytes header's kind, width, length and encoding together; SegmentKind::check_header refuses IdBytes, so the encoding check cannot be skipped. DESIGN.md says where the bytes are validated against the encoding (pg_warren, step 4). - Growing an owned base in place charges its bytes again while it is reserved, since a reallocation may move it. - IdMap::renumber's doc and DESIGN.md give text renumbering's cost. - TextIndex asserts Sync at compile time, and the hasher is described as std's keyed default hasher. - New tests: a 1 MiB id, a rebuild that fails inside the extension leaves nothing charged, and a single-node map through every operation. - TextIds::len and is_empty, used only by tests, are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M10 step 3 (DESIGN.md §5.7, §14 item 41): text node ids in
warren-core. pg_warren doesn't change in this step; step 4 wires text graphs through it.What's added
TextKeys, anIdKeysimplementation over three arrays:id_bytes,id_offsetsandid_order.id_order. Dense → external is two offsets and a slice.TextIds: the rebuild's arena.TextRebuild: buildsid_orderanew and appends the fresh ids.TextIndex: a keyed SipHash index over the live ids. Every hit is confirmed by comparing bytes, and the index is charged toBuildMemory.IdOffsets,IdBytes,IdOrder).from_text_segmentsbuilds the map from shared regions.decode_text_id_maptarget, added to CI's fuzz matrix.Trait changes in
warren-coreIdKeys::rebuildnow takesFreshKeys.IdKeys::renumbernow takes the build memory.KeysRebuild::finish_intoandIdMap::rebuild_into, which keep the rebuild's peak memory to the design's 2·N·L + 28·N.align_idsnow merges the two id maps in id order, in O(N).Measured at 1M nodes
rebuild_intoDESIGN.md §5.7 "Built (step 3)" records the design changes.
Tests
int8in dense ids, tombstones, CSRs, diff, alignment and compaction.Run locally:
-D warningsand rustdoc-D warnings: clean.cargo test -p warren-core: 675 tests pass.Generated by Claude Code