fix(knowledge): keep Slack searchable while its member crawl runs by waleedlatif1 · Pull Request #8005 · simstudioai/sim

waleedlatif1 · 2026-09-19T06:09:37Z

Summary

Members-mode search shows a document only while its member observation is under a day old, but a full Slack listing re-fetched every thread (a per-listing nonce in the listing hash) and took far longer than a day on a large workspace, so most of Slack went invisible and the crawl never completed a pass
New optional ConnectorConfig.listAccessibleScopes: before each member's listing, the member engine renews that member's observations under every channel they can still read, about twice a day (new knowledge_connector_member.scope_renewed_at watermark, expand-only migration). Access no longer depends on re-crawling content, and a channel the member left still lapses within a day
New optional ConnectorConfig.matchContentHash: Slack lists a thread under its root version (edit, reply count, latest reply, channel name) and stores version + text hash, so unchanged threads are not re-fetched. Re-read threads whose text is unchanged, including ones stored under the old text-only hash, only advance the stored hash, so nothing is re-embedded
Slack still re-reads threads active in the last 7 days (reply edits don't change the root), a rolling 1/28 of quiet threads each day, and every thread on an explicit full resync; verified empty threads use skippedRetryPolicy: 'source-change'
Fewer Slack calls per thread: permalinks are built from the workspace URL auth.test returns (chat.getPermalink only as a fallback), and conversations.info is fetched once per channel per run
A member listing that is still resuming no longer reports earlier content failures as this run's error; the generation's failures carry over once it completes, so the next full listing still retries them

Type of Change

Bug fix

Testing

Unit: lib/knowledge + connectors (5,466 passing); each new behavior fails its test when reverted
Postgres integration: new member-scope-renewal test, plus slack-empty-threads (both added to the CI integration list) and the surrounding connector suites, 77 passing
Checked the renewal page query plan on a production-sized copy: ordered index scan with primary-key lookups, no sort
bun run lint, check:audits, check:migrations origin/staging, type-check

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-09-19T06:09:42Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 19, 2026 7:07am UTC

greptile-apps · 2026-09-19T06:15:22Z

Confidence Score: 5/5

The PR appears safe to merge; the previous performance finding is addressed without introducing a demonstrated access, persistence, or rollout defect.

Summary

This PR keeps permission-scoped Slack knowledge searchable during long member crawls while reducing unnecessary thread hydration.

Renews member observations in batches based on currently accessible Slack channel scopes, with resumable progress and a completion watermark.
Introduces version-aware Slack thread hashes, scheduled refreshes, and hash-only persistence for unchanged hydrated content.
Reduces Slack API calls through per-run workspace and channel caching and locally constructed message links.
Adds additive database fields, API/log projections, and unit and PostgreSQL integration coverage for renewal and empty-thread behavior.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Member sync starts] --> B[List Slack channel scopes]
  B --> C[Gather scope-prefix batch]
  C --> D[Renew stale observations in accessible scopes]
  D --> E{Batch and scope listing complete?}
  E -- No --> F[Persist batch-start cursor]
  F --> A
  E -- Yes --> G[Advance renewal watermark]
  G --> H[Resume normal document listing]
  H --> I{Thread version current?}
  I -- Yes --> J[Keep indexed content]
  I -- Refresh due --> K[Hydrate Slack thread]
  K --> L{Text changed?}
  L -- No --> M[Update hash and source metadata]
  L -- Yes --> N[Reprocess and re-embed document]

_{Reviews (5) · Last reviewed commit: "fix(knowledge): batch scope renewal and ..."}

Members-mode search shows a document only while its member observation is younger than a day, but a full Slack listing re-fetched every thread and so took far longer than a day on a large workspace, leaving most of Slack invisible. Access is now renewed per channel the member can still read, and listings re-read only threads whose root changed, are active, or are due in a rolling 28-day refresh.

A Slack conversation listing that stops at its page cap now reports itself incomplete, and the member's renewal watermark only advances once every reachable channel has been listed and renewed.

waleedlatif1 · 2026-09-19T06:25:44Z

@greptile

waleedlatif1 · 2026-09-19T06:25:45Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-19T06:25:51Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

3 issues found across 23 files

Confidence score: 3/5

apps/sim/lib/knowledge/connectors/sync-primitives.ts: A current deferred hash can return unchanged before forceRehydrate is evaluated, causing a requested forced rehydration to be skipped; check forceRehydrate before accepting the matcher result.
apps/sim/lib/knowledge/connectors/member-observations.ts: Post-query scope filtering can consume the 10-minute renewal budget on stale observations before accessible rows are reached; push the scope-prefix predicate into SQL or otherwise filter before applying the budget.
apps/sim/lib/knowledge/connectors/listing-checkpoint.ts: Resuming an incomplete ordinary Slack listing during an explicit full resync restores fullSync as false, so resumed pages skip quiet-thread hydration; preserve the explicit full-resync state when resuming.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/knowledge/connectors/member-observations.ts">

<violation number="1" location="apps/sim/lib/knowledge/connectors/member-observations.ts:194">
P2: When a member has many stale observations outside their current scopes, this post-query filter can exhaust the 10-minute renewal budget before reaching accessible rows. Push the scope-prefix predicate into SQL, or query scopes separately, before applying the page limit so inaccessible observations do not consume renewal work.</violation>
</file>

<file name="apps/sim/lib/knowledge/connectors/sync-primitives.ts">

<violation number="1" location="apps/sim/lib/knowledge/connectors/sync-primitives.ts:291">
P2: When `forceRehydrate` is set and the connector matcher reports the deferred hash as current, this check returns `unchanged` before the force branch runs. Check `forceRehydrate` before accepting a matcher result so full resyncs always rehydrate deferred documents.</violation>
</file>

<file name="apps/sim/lib/knowledge/connectors/listing-checkpoint.ts">

<violation number="1" location="apps/sim/lib/knowledge/connectors/listing-checkpoint.ts:118">
P2: When an explicit full resync resumes an incomplete ordinary Slack listing, this line restores `fullSync` from the old checkpoint, which is `false`. Slack therefore skips quiet-thread hydration on the resumed pages, so the full resync does not reread every thread. Start a new full-sync checkpoint or carry the explicit full-sync request into the resumed generation before setting this context flag.</violation>
</file>

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

… identity change A Full resync now starts a new full-sync listing even when the connector does not rehydrate, so Slack rereads every thread instead of resuming an ordinary cursor. A member whose identity changes or whose token is rejected also loses its scope-renewal watermark, so renewal runs for the new identity right away.

waleedlatif1 · 2026-09-19T06:37:12Z

@greptile

waleedlatif1 · 2026-09-19T06:37:13Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-19T06:37:20Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 24 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

Accessible scopes are now listed page by page, and a renewal that does not finish within its budget stores its channel-listing cursor and start time, so the next run continues from there instead of re-reading the first pages and never reaching channels past them. The watermark records when the whole pass began, and an expired cursor restarts the pass once.

waleedlatif1 · 2026-09-19T06:45:59Z

@greptile

waleedlatif1 · 2026-09-19T06:46:01Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-19T06:46:08Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 24 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

Scope renewal now gathers container pages into batches before scanning the member's stale observations, so the scan runs once per batch instead of once per source page, and an unfinished batch resumes from where it was read. Content that hydrates unchanged under a new hash also refreshes its source URL, modified time and tags, and the member sync log records how many observations renewal kept fresh.

waleedlatif1 · 2026-09-19T07:06:07Z

@greptile

waleedlatif1 · 2026-09-19T07:06:08Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-19T07:06:15Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 26 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

vercel Bot deployed to Preview September 19, 2026 06:11 View deployment

greptile-apps Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread apps/sim/connectors/slack/slack.ts Outdated

Comment thread apps/sim/connectors/slack/slack.ts

waleedlatif1 added 2 commits September 18, 2026 23:24

fix(knowledge): keep scope renewal due when a channel listing is cut off

cef0b6d

A Slack conversation listing that stops at its page cap now reports itself incomplete, and the member's renewal watermark only advances once every reachable channel has been listed and renewed.

waleedlatif1 force-pushed the fix/slack-member-crawl-freshness branch from c78bd1c to cef0b6d Compare September 19, 2026 06:25

vercel Bot deployed to Preview September 19, 2026 06:27 View deployment

greptile-apps Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread apps/sim/connectors/slack/slack.ts

cubic-dev-ai Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts

Comment thread apps/sim/lib/knowledge/connectors/member-observations.ts

Comment thread apps/sim/lib/knowledge/connectors/sync-primitives.ts

Comment thread apps/sim/lib/knowledge/connectors/listing-checkpoint.ts

vercel Bot temporarily deployed to Preview September 19, 2026 06:36 Inactive

cubic-dev-ai Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts Outdated

vercel Bot deployed to Preview September 19, 2026 06:47 View deployment

greptile-apps Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts

cubic-dev-ai Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts

Comment thread apps/sim/connectors/slack/slack.ts

vercel Bot deployed to Preview September 19, 2026 07:07 View deployment

cubic-dev-ai Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts

waleedlatif1 mentioned this pull request Sep 19, 2026

improvement(knowledge): rank organization keyword search with Tin #8007

Merged

6 tasks

waleedlatif1 merged commit 6892dd7 into staging Sep 19, 2026
34 checks passed

waleedlatif1 deleted the fix/slack-member-crawl-freshness branch September 19, 2026 17:18

This was referenced Sep 19, 2026

v0.8.48: search improvements, file versions #8008

Closed

v0.8.48: prewarm, tin pg extensions, knowledge search speedups, ui consolidation, keyword search speedups #8096

Merged

Conversation

waleedlatif1 commented Sep 19, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 19, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 19, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 19, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 19, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Sep 19, 2026 •

edited

Loading

greptile-apps Bot commented Sep 19, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading