warren-core: IdMap<K: IdKeys> with FixedKeys (M10 step 2)

DESIGN.md

@@ -420,7 +420,7 @@ M9 (0.5.0) moves loaded versions into dynamic shared memory so backends share on
 
 ### 5.7 Identity, lifecycle and visibility (M10 design)
 
-*Designed 2026-10-23 for M10 (0.6.0), after 0.5.0's release; built so far: step 1 (the spike) and step 5 (ownership transfer). The decisions are §14 items 41–49: items 44, 45 and 46 were decided by the owner on 2026-10-23, as recommended, and the rest are decided here. Claims about Postgres were read in the PG 15–18 sources (`tcop/utility.c`, `tcop/cmdtaglist.h`, `catalog/pg_shdepend.c`, `commands/event_trigger.c`, `utils/backend_progress.h`, `utils/adt/pgstatfuncs.c`, `utils/wait_event.h`, `utils/misc/guc_tables.c`); where a claim needed code to confirm it, step 1 of the work breakdown did so on PG 15, 16 and 18 (Spike results, below), and the text reflects what it found.* M10 has five items, which share little code. `text` node ids change the id map and everything that maps external ids. Ownership transfer, cleanup after a dropped role, and quotas change the catalog's lifecycle. Observability adds a per-backend table in shared memory beside the registry of §5.6. Each part states what Postgres allows an extension on PG 15–18 and where pg_warren stops. Nothing in M10 changes a kernel, the CSR layout, the segment format of existing id types, or the registry's format, so 0.6.0 needs no restart to keep sharing on PG 17 and 18 (§5.6).
+*Designed 2026-10-23 for M10 (0.6.0), after 0.5.0's release; built so far: steps 1 (the spike), 2 (`IdMap<K: IdKeys>`) and 5 (ownership transfer). The decisions are §14 items 41–49: items 44, 45 and 46 were decided by the owner on 2026-10-23, as recommended, and the rest are decided here. Claims about Postgres were read in the PG 15–18 sources (`tcop/utility.c`, `tcop/cmdtaglist.h`, `catalog/pg_shdepend.c`, `commands/event_trigger.c`, `utils/backend_progress.h`, `utils/adt/pgstatfuncs.c`, `utils/wait_event.h`, `utils/misc/guc_tables.c`); where a claim needed code to confirm it, step 1 of the work breakdown did so on PG 15, 16 and 18 (Spike results, below), and the text reflects what it found.* M10 has five items, which share little code. `text` node ids change the id map and everything that maps external ids. Ownership transfer, cleanup after a dropped role, and quotas change the catalog's lifecycle. Observability adds a per-backend table in shared memory beside the registry of §5.6. Each part states what Postgres allows an extension on PG 15–18 and where pg_warren stops. Nothing in M10 changes a kernel, the CSR layout, the segment format of existing id types, or the registry's format, so 0.6.0 needs no restart to keep sharing on PG 17 and 18 (§5.6).
 
 #### Text node ids (§14 items 41, 42)
 
@@ -448,7 +448,7 @@ Dense → external is O(1): two adjacent offsets and a slice. External → dense
 
 The exact-type rule stays (§9.3). A `varchar` argument to a function whose result holds ids is `datatype_mismatch`, with `::text` in the hint. Lookups whose result holds no id (`degree`, `has_edge`, `reachable`) accept `varchar`, as they widen `int2` and `int4` today. `node_filter`, and the `filter` of a `warren.step` (already `text[]`, parsed per id type at walk time), take text ids as they are. `vector_from_query`'s id column must cast implicitly to `text`: `varchar` does, integers do not. The library's PL/pgSQL builds its type witnesses from `info ->> 'id_type'`, which `text` satisfies.
 
-**Code (item 41).** Text cannot be a fourth `E`. `warren_core::graph::ExternalId` is `Copy` and fixed-width, and `pg_warren`'s `IdDatum` promises that an id can be read in place at a fixed width (§5.6). So the id map becomes generic over its *key storage*: `IdMap<K: IdKeys>`, with `FixedKeys<E: ExternalId>` (today's pairs and inverse, unchanged) and `TextKeys` (the three arrays). `IdKeys::Key` is `E` for a fixed type and the unsized `[u8]` for text, so `dense_id(&K::Key)` and `external_id(dense) -> Option<&K::Key>` serve both. Everything in `warren-core` that is generic over `E: ExternalId` becomes generic over `K`: the build's `rebuild_within` and `EdgeIntake`, `compact`'s renumbering, `diff`'s node delta and its shared-dense-id check, `distance`'s `align_ids`, and locality renumbering. Tombstones, CSRs, vectors and every kernel are untouched, since they see only dense ids. In `pg_warren`, `with_id_type!` gains a `Text` arm. `IdDatum` splits in two: the conversion between datum and key (borrowed for text: an argument is detoasted in place, and an id returned is a `text` datum built from its bytes), and the promise of an in-place layout (fixed types only). `external_ids` does the rest as today.
+**Code (item 41).** Text cannot be a fourth `E`. `warren_core::graph::ExternalId` is `Copy` and fixed-width, and `pg_warren`'s `IdDatum` promises that an id can be read in place at a fixed width (§5.6). So the id map becomes generic over its *key storage*: `IdMap<K: IdKeys>`, with `FixedKeys<E: ExternalId>` (today's pairs and inverse, unchanged) and `TextKeys` (the three arrays). `IdKeys::Key` is `E` for a fixed type and the unsized `[u8]` for text, so `dense_id(&K::Key)` and `external_id(dense) -> Option<&K::Key>` serve both. *Built (step 2):* `IdMap<K>` holds the keys and the tombstones, and `IdKeys` has what the shared logic needs of a key storage: `id_space`, `bytes`, lookups both ways (tombstoned nodes included), `in_order` (every key with its dense id, in key order, which `align_ids` and compaction walk), `renumber` by a permutation, `compacted` (the keys at given dense ids, already in key order, as dense ids `0..n`), and `rebuild`. The rebuild's merge walk is `IdMap`'s, shared by every key storage: it sorts and deduplicates the node ids (`IdKeys::Ids`, a `NodeIds`: `BuildBuffer<E>` for a fixed type, an arena for text), counts the fresh ones, and feeds each key, kept or fresh, in key order to a `KeysRebuild` that `IdKeys::rebuild` reserved exactly, then frees the ids before the rebuild finishes. Segment encoding and `from_segments` stay on `IdMap<FixedKeys<E>>`. Until step 4, `pg_warren` names `IdMap<FixedKeys<E>>` where it named `IdMap<E>`. Everything in `warren-core` that is generic over `E: ExternalId` becomes generic over `K`: the build's `rebuild_within` and `EdgeIntake`, `compact`'s renumbering, `diff`'s node delta and its shared-dense-id check, `distance`'s `align_ids`, and locality renumbering. Tombstones, CSRs, vectors and every kernel are untouched, since they see only dense ids. In `pg_warren`, `with_id_type!` gains a `Text` arm. `IdDatum` splits in two: the conversion between datum and key (borrowed for text: an argument is detoasted in place, and an id returned is a `text` datum built from its bytes), and the promise of an in-place layout (fixed types only). `external_ids` does the rest as today.
 
 **Build, diff and compaction.**
 - *Reading ids.* Node and edge ids are read in place from the tuple (`source::text_in_place`, as relations are since M7), each batch in a memory context reset after it.
@@ -643,7 +643,7 @@ Each step is a pull request that leaves `main` green.
    - `WaitEventExtensionNew` and the `MaxBackends` binding.
    - A pgrx-sized change-count slot under Miri.
    - The text id map at 10M nodes: build, sort, merge and hash-index time, lookup latency, and memory, at *L* = 24, 48 and 64.
-2. *`warren-core`: `IdMap<K: IdKeys>` with `FixedKeys`,* with no behaviour change: every existing test and proptest passes unchanged, and Miri.
+2. *`warren-core`: `IdMap<K: IdKeys>` with `FixedKeys`,* with no behaviour change: every existing test and proptest passes unchanged, and Miri. *Done:* tests changed only where the types did (`IdMap<FixedKeys<i64>>`, and `external_id` returning a reference).
 3. *`warren-core`: `TextKeys`:* the rebuild with its arena and hash index, the three segment kinds, validation, renumbering, compaction, `diff` and alignment, with the tests above.
 4. *`pg_warren`: text graphs end to end:* `IdType::Text`; the collation, type and encoding checks; the build's intake; the region's arrays and the loader; `external_ids`; every SQL function. Also the `text_ids` tests, and the soak at 10M nodes with synthetic IRIs (`bench/soak.py --ids text`), recorded in §12.
 5. *Ownership transfer:* `set_owner`, ownership re-checked after the lock, and `set_owner_race`. **Done** (see "Ownership transfer", *Implemented*).

crates/pg_warren/src/build/pipeline.rs

@@ -8,8 +8,9 @@
 //! CSRs are built where the plan shows it, else where it reserves.
 
 use warren_core::graph::{
-    BuildCharge, BuildMemory, BuildPlan, CsrLayout, EdgeIntake, EdgeList, ExternalId, GraphCsr,
-    GraphError, IdMap, Membership, NodeOrder, Permutation, RelationDictionary, RelationIndex,
+    BuildCharge, BuildMemory, BuildPlan, CsrLayout, EdgeIntake, EdgeList, ExternalId, FixedKeys,
+    GraphCsr, GraphError, IdMap, Membership, NodeOrder, Permutation, RelationDictionary,
+    RelationIndex,
 };
 
 use super::error::BuildError;
@@ -97,7 +98,7 @@ pub(crate) fn build<E: ExternalId>(
 /// contraction made them.
 pub(crate) struct Contents<E> {
     /// The id map, tombstones included.
-    pub(crate) id_map: IdMap<E>,
+    pub(crate) id_map: IdMap<FixedKeys<E>>,
     /// The relation dictionary.
     pub(crate) relations: RelationDictionary,
     /// The edges, in dense ids.
@@ -164,7 +165,7 @@ pub(crate) fn write_contents<E: ExternalId>(
 /// NULL, unknown or tombstoned endpoint.
 fn edges<E: ExternalId>(
     graph: &GraphDefinition,
-    id_map: &IdMap<E>,
+    id_map: &IdMap<FixedKeys<E>>,
     implicit: Option<RelationIndex>,
     relations: &mut Relations,
     source: &mut impl SourceReader<E>,
@@ -216,7 +217,7 @@ struct EdgeCollector<'b, 'm, E> {
     /// The one relation of a graph without relations.
     implicit: Option<RelationIndex>,
     relations: &'b mut Relations,
-    intake: EdgeIntake<'m, E>,
+    intake: EdgeIntake<'m, FixedKeys<E>>,
     /// Rows dropped for a NULL relation or weight.
     incomplete_rows: u64,
 }
@@ -275,7 +276,7 @@ mod tests {
     use crate::segment::{read_segment, MemorySegmentStore};
     use pgrx::pg_sys;
     use warren_core::graph::{
-        AnyCsr, EdgeIntake, GraphCsr, GraphError, IdMap, RelationDictionary, Tombstones,
+        AnyCsr, EdgeIntake, FixedKeys, GraphCsr, GraphError, IdMap, RelationDictionary, Tombstones,
     };
 
     fn graph() -> GraphDefinition {
@@ -353,7 +354,7 @@ mod tests {
             read_segment(&mut self.store, version, name).is_ok()
         }
 
-        fn id_map(&mut self, version: VersionId) -> IdMap<i64> {
+        fn id_map(&mut self, version: VersionId) -> IdMap<FixedKeys<i64>> {
             let tombstones = Tombstones::decode(&self.segment(version, "tombstones")).unwrap();
             IdMap::decode(&self.segment(version, "id_map"), tombstones).unwrap()
         }
@@ -384,9 +385,9 @@ mod tests {
         }
     }
 
-    fn neighbours(csr: &AnyCsr, id_map: &IdMap<i64>, external: i64) -> Vec<i64> {
+    fn neighbours(csr: &AnyCsr, id_map: &IdMap<FixedKeys<i64>>, external: i64) -> Vec<i64> {
         let dense = id_map.dense_id(&external).unwrap();
-        csr.neighbours(dense).map(|d| id_map.external_id(d).unwrap()).collect()
+        csr.neighbours(dense).map(|d| *id_map.external_id(d).unwrap()).collect()
     }
 
     #[test]
@@ -485,7 +486,7 @@ mod tests {
         }
         assert_eq!(after.dense_id(&20), None);
         assert!(after.tombstones().contains(1));
-        assert_eq!(after.external_id(1), Some(20));
+        assert_eq!(after.external_id(1), Some(&20));
         assert_eq!(after.dense_id(&5), Some(3));
         assert_eq!(after.dense_id(&40), Some(4));
 

crates/pg_warren/src/build/refresh.rs

@@ -60,7 +60,9 @@ fn refresh_graph(name: &str, promote: bool, label: Option<&str>) -> Result<Versi
 mod tests {
     use pgrx::prelude::*;
     use pgrx::spi::Error;
-    use warren_core::graph::{AnyCsr, EdgeIntake, GraphCsr, IdMap, RelationDictionary, Tombstones};
+    use warren_core::graph::{
+        AnyCsr, EdgeIntake, FixedKeys, GraphCsr, IdMap, RelationDictionary, Tombstones,
+    };
 
     use crate::fixture;
     use crate::segment::{read_segment, SpiSegmentStore};
@@ -78,7 +80,7 @@ mod tests {
         read_segment(&mut SpiSegmentStore, version, name).unwrap_or_default()
     }
 
-    fn stored_id_map(version: i64) -> Option<IdMap<i64>> {
+    fn stored_id_map(version: i64) -> Option<IdMap<FixedKeys<i64>>> {
         let tombstones = Tombstones::decode(&segment(version, "tombstones")).ok()?;
         IdMap::decode(&segment(version, "id_map"), tombstones).ok()
     }
@@ -247,8 +249,8 @@ mod tests {
         let uuid = refresh("SELECT warren.refresh('u')")?;
         assert_eq!(version_row(uuid)?.as_deref(), Some("1|f|2|2|1|0|0|4|t|"));
         let tombstones = Tombstones::decode(&segment(uuid, "tombstones")).ok();
-        let id_map =
-            tombstones.and_then(|t| IdMap::<[u8; 16]>::decode(&segment(uuid, "id_map"), t).ok());
+        let id_map = tombstones
+            .and_then(|t| IdMap::<FixedKeys<[u8; 16]>>::decode(&segment(uuid, "id_map"), t).ok());
         let mut first = [0; 16];
         first[15] = 1;
         assert_eq!(id_map.and_then(|m| m.dense_id(&first)), Some(0));

crates/pg_warren/src/cache/load/mod.rs

@@ -19,7 +19,8 @@ use std::sync::{Arc, Weak};
 use pgrx::PgSqlErrorCode;
 use thiserror::Error;
 use warren_core::graph::{
-    BuildCharge, BuildMemory, ExternalId, GraphCsr, GraphError, IdMap, RelationDictionary,
+    BuildCharge, BuildMemory, ExternalId, FixedKeys, GraphCsr, GraphError, IdMap,
+    RelationDictionary,
 };
 use warren_core::version::{GraphVersion, VersionError};
 
@@ -69,7 +70,7 @@ impl SqlError for LoadError {
 #[derive(Debug, Clone, PartialEq)]
 pub(crate) struct LoadedVersion<E> {
     /// Its id map, tombstones included.
-    pub(crate) id_map: IdMap<E>,
+    pub(crate) id_map: IdMap<FixedKeys<E>>,
     /// Its relation dictionary.
     pub(crate) relations: RelationDictionary,
     /// Its CSRs. A loaded graph reports no deduplicated edges: that count
@@ -79,7 +80,7 @@ pub(crate) struct LoadedVersion<E> {
 
 impl<E: ExternalId> LoadedVersion<E> {
     /// The version as `warren-core`'s view of it.
-    pub(crate) fn version(&self) -> Result<GraphVersion<'_, E>, VersionError> {
+    pub(crate) fn version(&self) -> Result<GraphVersion<'_, FixedKeys<E>>, VersionError> {
         GraphVersion::new(&self.id_map, &self.relations, &self.graph)
     }
 
@@ -99,7 +100,7 @@ impl<E: ExternalId> LoadedVersion<E> {
 /// memory it holds.
 pub(crate) struct Identity<E> {
     /// The id map.
-    pub(crate) id_map: IdMap<E>,
+    pub(crate) id_map: IdMap<FixedKeys<E>>,
     /// The charge for the id map.
     pub(crate) id_map_charge: BuildCharge,
     /// The relation dictionary.
@@ -165,13 +166,13 @@ mod tests {
     use crate::segment::{write_version, MemorySegmentStore};
     use crate::source::IdDatum;
     use warren_core::graph::{
-        AnyCsr, CsrLayout, EdgeIntake, GraphCsr, IdMap, RelationCsr, RelationDictionary,
+        AnyCsr, CsrLayout, EdgeIntake, FixedKeys, GraphCsr, IdMap, RelationCsr, RelationDictionary,
     };
 
     /// A version built in memory: ids 10, 20, 30, 40 (or as given) with
     /// the second tombstoned.
     struct Built<E = i64> {
-        id_map: IdMap<E>,
+        id_map: IdMap<FixedKeys<E>>,
         relations: RelationDictionary,
         graph: GraphCsr,
     }

crates/pg_warren/src/cache/shared/dsm/tests.rs

@@ -276,7 +276,7 @@ mod tests {
         assert_eq!(port()?.evict(&|evicted| *evicted == key), 1);
         end_message();
         assert_eq!(slot(key), Some((State::Draining, 0, 1)), "the piece still reads the segment");
-        assert_eq!(piece.external_id(99), Some(100), "and reads it");
+        assert_eq!(piece.external_id(99), Some(&100), "and reads it");
         drop(piece);
         end_message();
         assert_eq!(slot(key), None, "released once nothing read through it is held");

crates/pg_warren/src/cache/versions.rs

@@ -202,7 +202,7 @@ mod tests {
     use super::*;
     use crate::cache::load;
     use crate::segment::{write_version, MemorySegmentStore};
-    use warren_core::graph::{EdgeIntake, GraphCsr, IdMap, RelationDictionary};
+    use warren_core::graph::{EdgeIntake, FixedKeys, GraphCsr, IdMap, RelationDictionary};
 
     /// The cached (not merely held) version ids, ascending.
     fn cached(cache: &VersionCache) -> Vec<VersionId> {
@@ -213,7 +213,7 @@ mod tests {
     /// (uuid ids).
     fn store() -> MemorySegmentStore {
         let mut store = MemorySegmentStore::default();
-        let id_map = IdMap::<i64>::new().rebuild(vec![1, 2, 3]).unwrap();
+        let id_map = IdMap::<FixedKeys<i64>>::new().rebuild(vec![1, 2, 3]).unwrap();
         let mut relations = RelationDictionary::new();
         let relation = relations.intern("").unwrap();
         let mut intake = EdgeIntake::directed(&id_map, false);
@@ -222,7 +222,7 @@ mod tests {
         for version in [1, 2] {
             write_version(&mut store, version, &id_map, &relations, &graph, 64).unwrap();
         }
-        let uuids = IdMap::<[u8; 16]>::new().rebuild(vec![[1; 16]]).unwrap();
+        let uuids = IdMap::<FixedKeys<[u8; 16]>>::new().rebuild(vec![[1; 16]]).unwrap();
         let empty =
             GraphCsr::build(EdgeIntake::directed(&uuids, false).finish().0, 1, 1, true).unwrap();
         write_version(&mut store, 3, &uuids, &relations, &empty, 64).unwrap();

crates/pg_warren/src/catalog/version.rs

@@ -3,7 +3,7 @@
 //! catalog owner.
 
 use pgrx::prelude::*;
-use warren_core::graph::{ExternalId, GraphCsr, IdMap, TargetWidth};
+use warren_core::graph::{ExternalId, FixedKeys, GraphCsr, IdMap, TargetWidth};
 
 use super::retention::{retention, Kept, VersionSummary};
 use super::{CatalogError, GraphDefinition};
@@ -36,7 +36,7 @@ impl VersionStats {
     /// The statistics of a version with this id map and these CSRs, and
     /// the given dropped and deduplicated edge counts.
     pub(crate) fn measure<E: ExternalId>(
-        id_map: &IdMap<E>,
+        id_map: &IdMap<FixedKeys<E>>,
         graph: &GraphCsr,
         dropped_edges: u64,
         deduplicated_edges: u64,
@@ -49,7 +49,7 @@ impl VersionStats {
     /// over its relations, and the given dropped and deduplicated edge
     /// counts.
     pub(crate) fn new<E: ExternalId>(
-        id_map: &IdMap<E>,
+        id_map: &IdMap<FixedKeys<E>>,
         edge_count: u64,
         dropped_edges: u64,
         deduplicated_edges: u64,

crates/pg_warren/src/community_sql/contraction.rs

@@ -88,7 +88,7 @@ pub(crate) fn contract_version<E: ExternalId>(
 mod tests {
     use pgrx::pg_sys;
     use warren_core::compute::{ComputeMemory, Vector};
-    use warren_core::graph::{EdgeIntake, Membership, NodeOrder, Tombstones};
+    use warren_core::graph::{EdgeIntake, FixedKeys, Membership, NodeOrder, Tombstones};
     use warren_core::interrupts::Uninterrupted;
 
     use super::*;
@@ -183,7 +183,7 @@ mod tests {
             )
         }
 
-        fn id_map(&mut self, version: VersionId) -> IdMap<i64> {
+        fn id_map(&mut self, version: VersionId) -> IdMap<FixedKeys<i64>> {
             let tombstones =
                 Tombstones::decode(&read_segment(&mut self.store, version, "tombstones").unwrap())
                     .unwrap();