You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Bound interactive vector candidates by readable documents, keep candidate IDs inline, and rerank a limited pool with original vectors. Use shortened half-precision candidates only for models trained for prefix retrieval; preserve all dimensions for other models.
Score keyword matches from a narrow text projection and sort scalar IDs/scores before hydrating content. Preserve lexical ranking, live source authorization, and workflow KB behavior. Keep Direct Search's 3s vector budget and Assistant's 8s budget, with stage diagnostics.
Make 0016 supersede 0015 and finish partially committed binary projections alongside the new vectors. Page source IDs before testing for missing rows, advance through completed pages, and log bounded progress. Fix standalone 0015 with the same pagination. Record both migrations only after 0016 succeeds so older deployment jobs cannot reinstall the old trigger.
Backfill before building new indexes, repair invalid interrupted builds, preserve valid indexes on retry, and serialize projection upgrades with active writers. Retain legacy projections during rollout and allow the migration job up to three hours for bulk work.
Type of Change
Bug fix
Testing
330 search/access unit tests, 142 DB unit tests, 374 application/ACL integration tests, 21 PostgreSQL permission tests, and 13 PostgreSQL migration tests passed. Migration tests cover cancellation, replay, valid/invalid indexes, synchronous writes, storage, and the actual runner upgrading an unjournaled partial 0015 through 0016. Another 220 tests passed around the latest staging merge.
Application stress suite: 19 passed, one optional live-network case skipped. On 300,000 synthetic chunks with PostgreSQL limited to 2 CPUs and 1 GiB RAM, common-term Direct/Assistant searches completed in 2.9–3.0s; two concurrent Assistant searches completed in 4.6s. All measured searches completed within their leg budgets; exact-neighbor recall was 100% across seven cases. Real application routes, authorization, and PostgreSQL execute; external embedding/source HTTP responses are controlled fixtures.
Large interrupted-upgrade benchmark: the actual runner completed 300,000 vector/keyword rows and nine indexes from a partial binary backfill in 14m49s on the same constrained database; the next migration run was a no-op in 3ms. The full 19-test retrieval suite passed again on those rebuilt indexes with every measured search complete and 100% exact-neighbor recall.
Lint, app/DB type checks, all 46 repository audits, API validation, docs manifest, block registry, and migration safety checks passed. Generated migration metadata retains staging's preceding migration.
The PR appears safe to merge with no outstanding findings.
Summary
This PR bounds vector and keyword retrieval, introduces half-precision candidate and keyword projections, and adds a resumable projection/index migration.
Propagates embedding-model identity into candidate-vector selection and uses shortened vectors only for compatible OpenAI models.
Filters candidate retrieval through readable document identities, reranks against original vectors, and preserves live source authorization during hydration.
Adds paginated, retry-safe vector and keyword projection backfills with concurrent index repair and superseded-migration journaling.
Extends PostgreSQL migration, authorization, retrieval, latency, recall, and interruption coverage.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Q[Search request] --> V{Retrieval legs}
V --> VC[Readable document identities]
VC --> HC[Bounded halfvec candidates]
HC --> RR[Original-vector reranking]
V --> KC[Keyword projection scoring]
RR --> LA[Live source authorization]
KC --> LA
LA --> H[Content hydration]
H --> R[Ranked results]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Summary
Type of Change
Testing
Checklist