feat(knowledge): fill the projection source and ACL in shards by waleedlatif1 · Pull Request #8112 · simstudioai/sim

waleedlatif1 · 2026-09-21T18:28:00Z

Summary

The projection source/ACL fill costs one graph insertion per row, which scales with cores, while the fill ran as one chain of runs on one connection. On a large deployment that is many days
The runner now takes an upper bound on its id range. A run may fill one shard of the chunk id space: ids are lowercase hex UUIDs, so the space is sliced on the first hex digit into contiguous ranges (the shard count must divide sixteen), and shards never fill the same page. The task admits one run per shard; each shard's continuation keeps its shard, so a sliced fill stays sliced until every slice is done
enqueueProjectionSourceAclBackfill(payload, shards) starts one chain per shard; the operator script takes --shards <n>. Without shards nothing changes: one chain over the whole space, one run at a time
The warm after the fill runs once nothing is left unfilled in any projection (one index probe per projection), so the shard that finishes last does it; two shards ending in the same moment could both warm, which repeats reads and nothing else
Search is unaffected: the fill is idempotent, every page still writes only rows still unset, and readability during the fill is decided as before

Type of Change

New feature

Testing

Runner: a new unit test asserts every page binds the range it was given (s.id < $2) so shards never meet
Application: shard ranges (contiguous slices, last shard unbounded, invalid counts and indexes refused), a run confined to its shard in both projections, a resumed shard keeping its upper bound, the warm left to whoever fills the rows another shard still holds, one run started per shard, an invalid shard count refused before anything starts
vitest for the backfill test file (15 tests) and the db package's runner tests, bun run lint, bun run check:audits and native tsc --noEmit pass

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

The fill's cost is graph insertion per row, which scales with cores while one run uses one. The runner takes an upper bound on its id range; a run may fill one shard of the chunk id space, sliced on the first hex digit so every shard is a contiguous range; the task admits one run per shard; the enqueue starts one chain per shard; and the warm after the fill runs once nothing is left unfilled anywhere.

vercel · 2026-09-21T18:28:08Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 21, 2026 7:27pm UTC

greptile-apps · 2026-09-21T18:32:25Z

Confidence Score: 5/5

The PR appears safe to merge, with no accepted new findings or outstanding previous findings.

Summary

This PR parallelizes the projection source/ACL backfill by dividing lowercase hexadecimal IDs into disjoint ranges and maintaining each range across bounded Trigger.dev continuation chains.

Adds validated shard range calculation and upper-bound filtering to every backfill page.
Raises worker concurrency to four and propagates shard identity through continuations.
Coordinates starts with per-shard tags, active-run discovery, and short-lived idempotency keys.
Defers table analysis and projection prewarming until no fillable rows remain globally.
Adds operator-facing --shards support and coverage for ranges, continuation behavior, completion, and enqueue coordination.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Operator runs backfill script] --> B{Trigger.dev enabled?}
  B -->|No| C[Run one inline full-range fill]
  B -->|Yes| D[Validate shard count]
  D --> E[Create one payload per shard]
  E --> F[Check latest tagged run]
  F -->|In flight| G[Leave range with existing chain]
  F -->|No active run| H[Trigger shard start]
  H --> I[Fill bounded pages in both projections]
  I --> J{Run budget exhausted?}
  J -->|Yes| K[Trigger tagged continuation with cursor]
  K --> I
  J -->|No| L{Any globally fillable rows remain?}
  L -->|Yes| M[Finish shard without warming]
  L -->|No| N[Analyze both projections]
  N --> O[Prewarm search projections]

_{Reviews (5) · Last reviewed commit: "feat(knowledge): key a fill start on the..."}

cubic-dev-ai

All reported issues were addressed across 6 files

_{Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic}

…ysis to whoever finishes A start's runs carry a per-shard idempotency key, so a retried start finds its runs rather than making more; a sliced start refuses a cursor, which belongs to one chain; the shard count is bounded by the runs the queue admits at once; a run reports its range done rather than the projection, and the run that finds nothing left anywhere analyzes and warms both projections once; the script reads its flag up front and refuses it where there is no worker to slice across.

waleedlatif1 · 2026-09-21T18:42:48Z

@greptile

waleedlatif1 · 2026-09-21T18:42:50Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-21T18:42:56Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 6 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…ng another A start no longer relies on an idempotency key, which would have blocked a legitimate restart for its window and collided across options. Every run of a chain carries the chain's tag, continuations included, and a start lists in-flight runs by that tag first: a range whose chain is still running is left to it, whether the start is sliced or not, and a start after a chain ended starts anew.

waleedlatif1 · 2026-09-21T18:55:53Z

@greptile

waleedlatif1 · 2026-09-21T18:55:54Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-21T18:56:00Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

…its start The in-flight lookup and the trigger are two calls, so two starts in the same instant could both find no chain. Each trigger now carries the chain's tag as a short-lived idempotency key, long enough to cover that instant and short enough never to hold a later restart.

waleedlatif1 · 2026-09-21T19:04:28Z

@greptile

waleedlatif1 · 2026-09-21T19:04:29Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-21T19:04:35Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

3 issues found across 6 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts">

<violation number="1" location="apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts:152">
P2: When a projection contains an orphaned row, `projectionsFilled` never returns true because the backfill intentionally cannot populate rows without a matching document. Ignore rows without a document in this completion probe so the final `ANALYZE` and prewarm still run for searchable data.</violation>

<violation number="2" location="apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts:193">
P2: When `shards` remains `1`, an invalid `payload.shard` bypasses validation and creates a run that fails only when the worker computes its range. Validate `payload.shard` before the trigger regardless of the `shards` argument.</violation>

<violation number="3" location="apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts:226">
P2: When an operator retries a terminal chain within two minutes, the same `idempotencyKey` suppresses the new backfill after the active-run lookup has stopped finding the old run. Use an idempotency key scoped to the individual start, or otherwise exclude terminal runs from the deduplication window.</violation>
</file>

…only rows the fill can finish A start's idempotency key is the chain's latest run, so two starts that saw the same state collapse into one while a start after another chain ended is its own; a start validates any shard it is handed; and the completion probe counts only unfilled rows whose document exists, since a row whose document is gone is not the fill's to finish.

waleedlatif1 · 2026-09-21T19:27:05Z

@greptile

waleedlatif1 · 2026-09-21T19:27:06Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-21T19:27:13Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 6 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts

Comment thread packages/db/script-migrations/0021_embedding_search_connector.ts

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

vercel Bot deployed to Preview September 21, 2026 18:44 View deployment

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts Outdated

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread apps/sim/background/projection-source-acl-backfill.ts

vercel Bot temporarily deployed to Preview September 21, 2026 18:55 Inactive

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts

vercel Bot temporarily deployed to Preview September 21, 2026 19:04 Inactive

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts Outdated

Comment thread apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts Outdated

Comment thread apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts

vercel Bot temporarily deployed to Preview September 21, 2026 19:27 Inactive

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

waleedlatif1 merged commit e07e7ba into staging Sep 21, 2026
34 checks passed

waleedlatif1 deleted the feat/sharded-projection-backfill branch September 21, 2026 19:34

waleedlatif1 mentioned this pull request Sep 21, 2026

v0.8.50: search #8114

Merged

Conversation

waleedlatif1 commented Sep 21, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Sep 21, 2026 •

edited

Loading

greptile-apps Bot commented Sep 21, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading