improvement(search): speed up organization integration and source queries

[waleedlatif1]

Summary

• Live-source candidate discovery (knowledgeReadAccessBatches) now starts from the sources a GitHub/Confluence grant could authorize and stops at each source's first candidate document, instead of DISTINCT-scanning every document in the owner with the access predicate evaluated twice per row
• Source eligibility lives in one place (access/live-sources.ts), shared by the grant resolvers and discovery, so discovery can never skip an authorizable source. KnowledgeAccessProvider.hasLiveSourceReaders becomes liveSourceConnectorCondition
• Search source summaries use per-source probes, not a per-viewer aggregate over every visible document. viewerDocumentCount is replaced by hasViewerDocuments: exact per-viewer counts are O(visible documents) no matter how they're indexed. The member status reads "Ready to search" instead of "N searchable documents"
• Adds partial index doc_connector_processing_status_idx for the failed/pending/processing probes behind the source status, progress and admin overview reads (concurrent build, migration 0345)
• Personal integrations no longer run the whole source overview (including its access probes) just to list configured provider types, and drop the unused searchableDocuments count
• Source document list runs one count scan instead of three, in parallel with the page query

Type of Change

• Improvement (performance)

Testing

• On a 727k-document local corpus (warm cache), queries executed through the app client with bound params:
  • Discovery: 2.7s → 38ms (plus ~0.6s of JIT compile the old query also paid)
  • Source states: 2.15s → 0.12s
  • Overview/progress probes use the new index with bound params
• Old vs new discovery and source-state queries return identical results; nothing leaks across tenants
• bun scripts/test-knowledge-acls.ts real Postgres integration suite (44 files, 374 tests) passes, including the GitHub live-grant, overview, pagination and progress suites. predicate.postgres.test.ts fails the same way on staging: its hand-built schema lacks access_rewrite_pending, and CI doesn't run it
• Unit suites, type-check, lint, check:audits, check:api-validation:strict and check:migrations all pass

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 14, 2026 11:21pm UTC

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no actionable correctness, authorization, migration, or compatibility regressions were identified.

Summary

This PR improves organization-search performance by narrowing live-source candidate discovery, replacing per-viewer document aggregates with per-source probes, consolidating connector document counts, and adding a matching partial processing-status index.

• Centralizes GitHub and Confluence live-source eligibility predicates for discovery and grant resolution.
• Replaces exact source document counts with an existence flag and updates internal contracts and status consumers.
• Avoids the full source overview when personal integrations need only configured provider types.
• Adds a concurrent, replay-safe partial index for non-terminal connector-document processing states.
• Preserves tenant, ACL, source-status, and connector-document filtering behavior in the reviewed paths.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Reader[Authorized reader] --> Scope[Resolve ACL and live-source scope]
  Scope --> Sources[Eligible GitHub or Confluence sources]
  Sources --> Candidate{Matching candidate document exists?}
  Candidate -- No --> Ordinary[Use ordinary stored ACL result]
  Candidate -- Yes --> Proof[Resolve live connector grants]
  Proof --> Batches[Yield authorized document predicates]
  Batches --> Search[Search and source summaries]
  Search --> Probes[Per-source document state probes]
  Probes --> Status[Ready, indexing, or sync-attention status]

Loading

_{Reviews (1) · Last reviewed commit: "improvement(search): speed up organizati..."}