fix(knowledge): share compact vector retrieval across access scopes by icecrasher321 · Pull Request #7944 · simstudioai/sim

icecrasher321 · 2026-09-17T21:10:44Z

Summary

Route workspace-scoped KB and workflow vector searches through the shared compact-candidate and full-vector reranking engine. Remove the separate broad full-vector query and per-KB result quotas.
Enforce workspace access and tags before candidate limits; preserve live user verification and current access checks during hydration. Keep small scopes exact and cap broad full-vector reranking at 1,600 candidates.
Preserve caller deadlines and truthful partial-result reporting. Diagnostics distinguish candidate selection, reranking, authorization, and hydration without claiming a physical index was used.

Type of Change

Bug fix

Testing

300 focused unit tests and 22 PostgreSQL integration tests pass, covering authorization, supported embedding widths, concurrent KB requests, and timeout reporting.
Large PostgreSQL benchmark of the retrieval engine: 27 passed, one optional live Go/provider test skipped. Synthetic corpus: 211,000 chunks; timed database limited to 2 CPUs / 1 GiB. Real application and SQL paths, controlled embedding and permission HTTP responses.
Final benchmark recovered all measured exact top-50/top-15 neighbors. Eight concurrent workspace searches completed vector retrieval in 0.24–0.88 seconds. EXPLAIN verifies compact HNSW use for broad scopes and bounded compact scans for the smaller full-width fixture.
Follow-up refill regression: if the nearest row loses access before hydration, a single-result workspace search returns the next authorized row from the same minimum 20-row page. Confirmed the regression fails before the fix; reran the focused unit and real PostgreSQL suites afterward.
An earlier shared-machine run had two timeout samples during concurrent local validation; the unchanged full-suite repeat passed. These synthetic measurements are not a production capacity guarantee. Selected samples evict PostgreSQL shared buffers, not the OS cache.
Type checking, lint, all 46 repository audits, block-registry validation, and docs-manifest validation pass.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-09-17T21:10:51Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 17, 2026 10:22pm UTC

icecrasher321 · 2026-09-17T21:11:51Z

@greptile

icecrasher321 · 2026-09-17T21:11:54Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-17T21:12:00Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 6 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 6 files

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic}

greptile-apps · 2026-09-17T21:16:30Z

Confidence Score: 5/5

The PR appears safe to merge; no actionable correctness, security, or repository-rule violations remain.

Summary

Routes workspace-scoped knowledge vector searches through the shared compact-candidate and exact reranking pipeline while preserving authorization during candidate selection and hydration.

Applies workspace access and structured tag filters before candidate limits.
Uses exact ranking for exhausted scopes and bounded compact-vector candidate selection for broad scopes.
Shares search deadlines across candidate selection, reranking, authorization, and hydration.
Expands unit and PostgreSQL integration coverage for access changes, pagination, embedding widths, concurrency, diagnostics, and timeout reporting.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Request[Workspace knowledge search] --> Scope[Resolve KB scope, access, and tags]
    Scope --> Probe[Probe bounded visible chunk identities]
    Probe -->|Scope exhausted| Exact[Exact full-vector ranking]
    Probe -->|Broad scope| Candidates[Compact-vector candidate selection]
    Candidates --> Rerank[Bounded full-vector reranking]
    Exact --> Hydrate[Recheck access and hydrate content]
    Rerank --> Hydrate
    Hydrate --> Results[Top-K authorized results]

_{Reviews (4) · Last reviewed commit: "fix(knowledge): batch workspace search r..."}

icecrasher321 · 2026-09-17T21:48:12Z

@greptile

icecrasher321 · 2026-09-17T21:48:16Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-17T21:48:23Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 7 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

icecrasher321 · 2026-09-17T22:09:44Z

@greptile

icecrasher321 · 2026-09-17T22:09:45Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-17T22:09:53Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 7 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

icecrasher321 · 2026-09-17T22:22:16Z

@greptile

icecrasher321 · 2026-09-17T22:22:18Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-17T22:22:24Z

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 7 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic}

icecrasher321 marked this pull request as ready for review September 17, 2026 21:12

cubic-dev-ai Bot reviewed Sep 17, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/__integration__/search-latency.integration.ts Outdated

cubic-dev-ai Bot reviewed Sep 17, 2026

View reviewed changes

icecrasher321 force-pushed the codex/unify-scoped-vector-retrieval branch from 0100011 to c787b5c Compare September 17, 2026 21:47

vercel Bot temporarily deployed to Preview September 17, 2026 21:47 Inactive

greptile-apps Bot reviewed Sep 17, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/__integration__/search-latency.integration.ts Outdated

cubic-dev-ai Bot reviewed Sep 17, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/queries.ts

icecrasher321 added 4 commits September 17, 2026 15:05

fix(knowledge): share compact vector retrieval across access scopes

5c2e6c7

fix(knowledge): align search fixtures with shared retrieval

e72dea2

fix(knowledge): preserve recall within bounded reranking

7828bde

fix(knowledge): validate bounded compact scan plans

454f44c

icecrasher321 force-pushed the codex/unify-scoped-vector-retrieval branch from c787b5c to 454f44c Compare September 17, 2026 22:09

vercel Bot temporarily deployed to Preview September 17, 2026 22:09 Inactive

cubic-dev-ai Bot reviewed Sep 17, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/queries.ts Outdated

fix(knowledge): batch workspace search refills

e1133d6

vercel Bot temporarily deployed to Preview September 17, 2026 22:22 Inactive

cubic-dev-ai Bot reviewed Sep 17, 2026

View reviewed changes

icecrasher321 merged commit 7acae58 into staging Sep 17, 2026
34 of 35 checks passed

icecrasher321 deleted the codex/unify-scoped-vector-retrieval branch September 17, 2026 22:32

waleedlatif1 mentioned this pull request Sep 18, 2026

v0.8.43: search improvements #7954

Merged

Conversation

icecrasher321 commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

icecrasher321 commented Sep 17, 2026

Uh oh!

icecrasher321 commented Sep 17, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 17, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

icecrasher321 commented Sep 17, 2026

Uh oh!

icecrasher321 commented Sep 17, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 17, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

icecrasher321 commented Sep 17, 2026

Uh oh!

icecrasher321 commented Sep 17, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 17, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

icecrasher321 commented Sep 17, 2026

Uh oh!

icecrasher321 commented Sep 17, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 17, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

icecrasher321 commented Sep 17, 2026 •

edited

Loading

vercel Bot commented Sep 17, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 17, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading