chore(db): drop unused ANN indexes on embedding - #7955
Conversation
Approximate retrieval is served by the compact embedding_search projection. The only vector ordering left on embedding is an exact rerank wrapped as (distance) + 0, which the planner cannot match to an index expression, and the binary_quantize expression indexes were never referenced by any query. The last app version that ordered by a bare distance has drained, so these ten indexes were maintained on every chunk write while never being scanned.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
 |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
The bulk-seed path required embedding_vector_hnsw_idx to exist, dropped it before seeding and rebuilt it after. With no ANN index left in the schema it threw before measuring anything. The benchmark seeds embedding directly and never populates embedding_search, so it now creates its own HNSW index after the load rather than borrowing a schema one, for both seed paths.
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 6 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
This branch was previously deployed
1 participant
Summary
embeddingthat are maintained on every chunk write but never scanned: the five per-width*_vector_hnsw_idxand the five*_binary_hnsw_idxexpression indexesembedding_searchprojection and its own HNSW indexes; the only vector ordering left onembeddingis the exact rerank, wrapped as(distance) + 0so the planner cannot match an index expressionbinary_quantizeexpression indexes were never referenced by any query in any shipped version — they appear only in the schema, and an existing integration test already asserts the distance expression never matchesbinary_quantize(embedding.embedding...embeddingDistanceTSDoc, which described an index this table no longer carriesWhy this is safe now
The previous release still had a branch that ordered by a bare distance (
orderBy(exact ? sql(distance) + 0: distance)), which did use the index. That branch was removed in the release currently serving traffic, so this drop lands one release after the last consumer drained — the contract half of the expand/contract the schema comment called for.Verified before dropping:
EXPLAINon pgvector confirms(v <=> $1) + 0turnsIndex Scan using ..._hnswintoSeq Scan → Sort, so the shipped query shape provably cannot use these indexesemb_doc_id_idx+1,386) showed +0 scans on bothembedding_vector_hnsw_idxandembedding_binary_hnsw_idxUses
DROP INDEX CONCURRENTLY IF EXISTSbehind the repo'sCOMMIT;/SET lock_timeoutconvention so no blocking lock is taken on a hot table.Type of Change
Testing
bun run check:migrations origin/stagingpasses (backward-compatible). Lint,check:audits(46 audits),docs-manifest:check, andtype-checkall pass. Knowledge vector/search suites pass (172 tests). Snapshot diff verified to contain exactly these ten index removals and no column or table changes.Checklist