Context
Part of the "AI-native query layer" exploration for hotdata-langchain (single query_hotdata(...) tool routing across SQL/BM25/vector/point-lookup pathways — see internal plan notes). Point lookups are one of the four named retrieval pathways, and currently have no first-class, standalone path anywhere in the stack.
Note: the actual code change for this issue lands in hotdata-dev/runtimedb, not this repo. Tracked here to keep AI-native-layer roadmap items in one place.
Findings (verified via direct code investigation)
datafusion-vector-search-ext's PointLookupProvider::fetch_by_keys(keys, key_col, projection) (src/lookup.rs:66-76) is already a general-purpose abstraction with zero ANN-specific logic — three implementations exist (HashKeyProvider, SqliteLookupProvider, and the production FeatherLookupProvider: sorted Arrow IPC + binary search + mmap), none of which assume the keys came from a USearch search.
The actual coupling is in runtimedb:
- The lookup sidecar is only ever built inside the vector-index build/update pipeline (
runtimedb/src/vector/mod.rs, build_vector_index_from_batches / update_vector_index_from_batches, ~lines 443-660) — a table with no vector column has no sidecar today.
RegisteredTable / VectorIndexResolver (registry.rs) require a USearch Index to exist before a lookup provider can be attached.- RuntimeDB core has no other exact point-lookup mechanism outside this (only approximate Parquet row-group/bloom pruning and a coarse sorted-range index).
Scope to generalize
- Decouple lookup-sidecar lifecycle from the vector-index build pipeline in
runtimedb/src/vector/mod.rs, so it can be built for any table with a declared key column, independent of a vector column.
- Loosen (or add a parallel path in)
RegisteredTable / VectorIndexResolver so a lookup-only entry doesn't require a vector index.
- Add a small UDTF mirroring
VectorSearchVectorUDTF (udtf.rs:43-91), e.g. point_lookup('conn.schema.table', 'key_col', ARRAY[...]).
Estimated as a moderate, scoped refactor — not a redesign. (1) and (2) are entirely in runtimedb; (3) is small once those land.
Context
Part of the "AI-native query layer" exploration for
hotdata-langchain(singlequery_hotdata(...)tool routing across SQL/BM25/vector/point-lookup pathways — see internal plan notes). Point lookups are one of the four named retrieval pathways, and currently have no first-class, standalone path anywhere in the stack.Note: the actual code change for this issue lands in
hotdata-dev/runtimedb, not this repo. Tracked here to keep AI-native-layer roadmap items in one place.Findings (verified via direct code investigation)
datafusion-vector-search-ext'sPointLookupProvider::fetch_by_keys(keys, key_col, projection)(src/lookup.rs:66-76) is already a general-purpose abstraction with zero ANN-specific logic — three implementations exist (HashKeyProvider,SqliteLookupProvider, and the productionFeatherLookupProvider: sorted Arrow IPC + binary search + mmap), none of which assume the keys came from a USearch search.The actual coupling is in
runtimedb:runtimedb/src/vector/mod.rs,build_vector_index_from_batches/update_vector_index_from_batches, ~lines 443-660) — a table with no vector column has no sidecar today.RegisteredTable/VectorIndexResolver(registry.rs) require a USearchIndexto exist before a lookup provider can be attached.Scope to generalize
runtimedb/src/vector/mod.rs, so it can be built for any table with a declared key column, independent of a vector column.RegisteredTable/VectorIndexResolverso a lookup-only entry doesn't require a vector index.VectorSearchVectorUDTF(udtf.rs:43-91), e.g.point_lookup('conn.schema.table', 'key_col', ARRAY[...]).Estimated as a moderate, scoped refactor — not a redesign. (1) and (2) are entirely in
runtimedb; (3) is small once those land.