v0.8.42: knowledgebase queueing improvemnets, file search time fixes, outbox fixes

Author: waleedlatif1

.github/workflows/codeql.yml

@@ -62,7 +62,15 @@ on:
 
 concurrency:
   group: codeql-${{ github.ref }}
-  cancel-in-progress: true
+  # PR runs only. Superseding is what this is for: a PR push makes the previous
+  # scan of that PR irrelevant, and `refs/pull/N/merge` keeps each PR in its own
+  # group. The push and schedule triggers both resolve to `refs/heads/main`, so a
+  # blanket `true` let the daily cron cancel the merge scan of the same commit --
+  # a 2-minute window that finally landed on 3c8a4c4 (push 08:32:58 killed by the
+  # 08:34:47 cron), leaving main with a red rollup for a commit that the cron had
+  # in fact scanned clean. Non-PR events now queue instead: one pending run is
+  # held per group, so the cron simply waits out the merge scan.
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 permissions:
   contents: read

.github/workflows/test-build.yml

@@ -222,7 +222,10 @@ jobs:
         working-directory: packages/db
         env:
           KNOWLEDGE_ACL_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_auth_scim
-        run: bunx vitest run script-migrations/0016_backfill_search_vectors.postgres.test.ts
+        run: >-
+          bunx vitest run
+          script-migrations/0016_backfill_search_vectors.postgres.test.ts
+          script-migrations/0018_repair_workspace_file_content_revision.postgres.test.ts
 
       - name: Verify Search progress, pagination, and outbox scheduling in PostgreSQL
         working-directory: apps/sim
@@ -235,11 +238,36 @@ jobs:
           lib/knowledge/__integration__/search-source-pagination.integration.ts
           lib/knowledge/__integration__/search-reference-batching.integration.ts
           lib/knowledge/__integration__/embedding-insert-batches.integration.ts
+          lib/knowledge/__integration__/connector-lifecycle-locks.integration.ts
+          lib/knowledge/__integration__/connector-deferral.integration.ts
+          lib/knowledge/__integration__/stored-document-recovery.integration.ts
+          lib/knowledge/__integration__/connector-partition-work.integration.ts
+          lib/knowledge/__integration__/listing-continuation.integration.ts
           lib/knowledge/__integration__/kb-block-search.integration.ts
           lib/core/outbox/service.integration.ts
           lib/knowledge/__integration__/connector-upload.integration.ts
           lib/uploads/contexts/organization-logo/application.integration.ts
 
+      - name: Verify Confluence identity and directory sync in PostgreSQL
+        working-directory: apps/sim
+        env:
+          KNOWLEDGE_ACL_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_auth_scim
+        run: >-
+          bunx vitest run --mode integration
+          lib/knowledge/__integration__/confluence-identity.integration.ts
+          lib/knowledge/__integration__/directory-sync.integration.ts
+
+      - name: Verify Confluence audience migrations and permission queries in PostgreSQL
+        working-directory: apps/sim
+        env:
+          KNOWLEDGE_ACL_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_acl_test
+        run: |
+          bun -e 'import postgres from "postgres"; const sql = postgres(process.env.DATABASE_URL); await sql`CREATE DATABASE sim_acl_test`; await sql.end()'
+          bunx vitest run --mode integration lib/knowledge/access/group-membership.integration.ts
+          bunx vitest run \
+            lib/knowledge/access/predicate.postgres.test.ts \
+            lib/knowledge/connectors/external-directory.postgres.test.ts
+
   test-build:
     name: Lint and Test
     runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}

apps/docs/content/docs/knowledgebase/connectors.mdx

@@ -78,7 +78,7 @@ Each connector has source-specific fields that control what gets synced. Example
 
 - **Notion** — sync an entire workspace, a specific database, or a single page tree
 - **GitHub** — specify a repository, branch, and optional file extension filter
-- **Confluence** — enter your Atlassian domain and optionally filter by space key or content type
+- **Confluence** — enter your Atlassian domain and choose spaces, or **All** for all spaces accessible at each sync. Optionally filter by content type or label. PDF and Word (`.docx`, Word 97–2003 `.doc`) attachments on matching pages and blog posts are included as separate documents.
 - **Azure DevOps** — choose what to sync (wiki pages, work items, repository files, or all), with optional work item type/state filters, a custom WIQL query, and repository/branch/path filters
 - **Amazon S3** — point at a bucket with an optional key prefix and a customizable file extension allowlist; S3-compatible stores (Cloudflare R2, MinIO) are supported via a custom endpoint
 - **YouTube** — sync a channel (by `@handle` or ID) or playlist, with an optional published-after date filter and the option to exclude Shorts
@@ -88,6 +88,8 @@ Each connector has source-specific fields that control what gets synced. Example
 
 Configuration is validated on save — if a repository doesn't exist or a domain is unreachable, you'll see an error immediately.
 
+Confluence attachment indexing requires `read:attachment:confluence`. For a service account, include it when creating the scoped API token; see the [Confluence scope list](/search/confluence#using-a-service-account). Attachments are checked even when the parent page has not changed. Files over 100 MB appear as skipped; convert Word 6/95 files to `.docx` before attaching them.
+
 </Step>
 <Step>
 

apps/docs/content/docs/platform/self-hosting/environment-variables.mdx

@@ -234,9 +234,10 @@ width. Check what you pick there, or upgrade Ollama.
 | `COPILOT_API_KEY` | API key for Chat. Without it the Sim Chat block, scheduled prompt jobs, and Inbox cannot run |
 | `NEXT_PUBLIC_CHAT_DISABLED` | Set to `true` to hide the Chat module: the workspace lands on your first workflow, with no chats list, scheduled tasks, or editor Chat panel. Chat is shown when unset; `npx sim-setup` sets it for you if you skip the chat key |
 | `PII_URL` | Base URL of the Presidio service backing PII detection and redaction. The Helm chart wires it to its own `pii` Service when `pii.enabled`; on Compose point it at the PII service on your network. The default `http://localhost:5001` exists only in local development, and leaving it makes redaction fail |
-| `DURABLE_SECRET_PROVENANCE_ENFORCED_SURFACES` | Durable stores where a value whose secret provenance was never recorded fails the run instead of logging a warning. `all`, or a comma-separated subset of `memory`, `table-row`, `knowledge`, `workspace-file`. Unset (nothing enforced) by default |
 | `ADMIN_API_KEY` | Admin API key for GitOps operations and organization provisioning |
 
+Tracked memory, table rows, knowledge content, and workspace files require valid secret provenance before entering a model or a trusted runtime. Records with a null provenance tracking marker retain legacy compatibility.
+
 ## Enterprise Features
 
 Enterprise features are unlocked by configuration rather than billing on self-hosted deployments. One switch turns on the full set; per-feature flags below it override the switch either way.

apps/docs/content/docs/search/coda.mdx

@@ -19,14 +19,27 @@ Sign in to Coda with a dedicated integration account that can read the documents
 
 Use a standard REST API token. Do not select the MCP restriction.
 
-<Image className="mx-auto h-auto w-full max-w-2xl" src="/static/search/coda-api-token.jpg" alt="Coda API connections with the Generate new token dialog" width={1280} height={720} />
+<Image className="mx-auto h-auto w-full max-w-xl" loading="eager" src="/static/search/coda-token-creation.jpg" alt="Coda Generate new token dialog with a name and no restriction selected" width={600} height={281} />
 
 </Step>
 <Step>
 
-### Add the source
+### Add the credential
 
-In Sim, open **Settings → Sources → Add source → Coda**. Add or select a Coda service account credential and enter the token when prompted.
+In Sim, open **Settings → Sources → Add source → Coda**. If Coda is already listed, open it and select **Add connection**.
+
+Open **Service account → Add API token**. Paste the token, give it a recognizable display name, then select **Add API token**. You can also choose an existing Coda credential.
+
+<Image className="mx-auto h-auto w-full max-w-md" src="/static/search/coda-credential.jpg" alt="Add Coda API token form with token, display name, and optional description fields" width={516} height={435} />
+
+</Step>
+<Step>
+
+### Choose documents
+
+Select the documents to index, or leave the selection empty for discovery.
+
+<Image className="mx-auto h-auto w-full max-w-md" src="/static/search/coda-setup.jpg" alt="Coda connection setup with a saved credential, Documents selector, and optional Enterprise organization ID" width={516} height={458} />
 
 | Field | What to enter |
 |---|---|
@@ -40,7 +53,7 @@ The picker shows accessible documents the token owner has opened. Use IDs for ot
 
 ### Sync and verify
 
-Select **Connect & Sync**. Open the source's **Documents** and **Sync history** to check indexing. Content and permission changes appear after synchronization.
+Select **Connect & Sync**. Open the source's **Documents** and **Sync history** to check indexing. Content and permission changes appear after synchronization. Have a teammate with a matching verified email search for a shared document, and confirm that an unshared teammate cannot find it.
 
 To rotate the token, add the replacement credential, update the source, and verify a sync before revoking the old token.
 
@@ -59,8 +72,17 @@ Ordinary connections index visible canvas text and base-table rows. Enterprise c
 
 Enterprise mode excludes deactivated and deleted users. Direct-share guests absent from the organization directory are excluded; guests explicitly listed in a group or workspace may receive that membership's access. Ordinary connections cannot check Coda organization deactivation, so manage departing users' Sim membership too.
 
-Sim refreshes permissions even when content is unchanged. Unverified permissions never become broad access, and Sim admin status does not override Coda sharing. For setup failures or missing results, check the token's access, Enterprise admin role if applicable, the teammate's verified email, and **Sync history**.
+Sim refreshes permissions even when content is unchanged. Unverified permissions never become broad access, and Sim admin status does not override Coda sharing.
+
+These access rules apply to organization Search and **Admin access** sources. In a regular knowledge base, **Workspace access** shares all indexed content with that Sim workspace and ignores the Enterprise organization ID.
 
-In a regular knowledge base, **Workspace access** intentionally shares all indexed content with that Sim workspace and does not use the Enterprise organization ID. The permissions above apply to organization Search and **Admin access** sources.
+## Troubleshooting
+
+| Problem | Next step |
+|---|---|
+| Document missing from the picker | Open it in Coda with the token owner, or enter its document ID. |
+| Enterprise setup fails | Confirm the organization ID and Enterprise org-admin role. A workspace admin token is insufficient. |
+| Teammate sees no results | Check organization membership, their verified Sim email, the Coda share, and **Sync history**. |
+| Token expired or revoked | Replace the credential in the connection's **Settings**, then verify a sync. |
 
 See Coda's [public API](https://coda.io/developers/apis/v1) and [Enterprise Admin API](https://coda.io/developers/apis/admin/v1).

apps/docs/content/docs/search/confluence.mdx

@@ -7,7 +7,7 @@ import { Callout } from 'fumadocs-ui/components/callout'
 import { Step, Steps } from 'fumadocs-ui/components/steps'
 import { Image } from '@/components/ui/image'
 
-Search pages and blog posts from selected Confluence Cloud spaces. A Sim organization admin enables Confluence; **each teammate connects their own account**.
+Search pages, blog posts, and their PDF and Word attachments from selected Confluence Cloud spaces. A Sim organization admin enables Confluence; **each teammate connects their own account**.
 
 | Method | How it works |
 | --- | --- |
@@ -38,7 +38,9 @@ Open **Settings → Sources → Add source** and select **Confluence**. This ope
 
 ### Choose the account and spaces
 
-Under **Service account**, select a service account or [add one](#using-a-service-account). Enter the same **Confluence site** as the credential, then choose **Spaces**. **All** in the dropdown selects every space the account can currently browse; newly created spaces are not added automatically. Clear the picker search before selecting all.
+Under **Service account**, select a service account or [add one](#using-a-service-account). Enter the same **Confluence site** as the credential, then choose **Spaces**. **All** in the dropdown includes every space the syncing account can access at each sync, including newly accessible spaces. Clear the picker search before selecting all.
+
+If you selected all spaces before this behavior was introduced, reselect **All** and save. Previously saved selections remain a fixed list of spaces.
 
 To enter comma-separated keys such as `ENG, PRODUCT`, use the switch beside **Spaces**. Switching between the picker and manual entry keeps your selection.
 
@@ -63,7 +65,7 @@ After an admin configures Confluence, open **Integrations** and select **Connect
 If Confluence is allowed but no source exists, select **Connect** beside Confluence. To add another site later, open the Confluence row’s actions menu (**…**) and select **Add Confluence site**:
 
 1. Open **Your account** and select a saved account or **Connect Confluence account**. Authorize using the Atlassian email matching your verified Sim email.
-2. Enter the hostname under **Atlassian site**, then choose **Spaces**. Use **All** in the dropdown for the complete current list, or the arrows beside **Spaces** to enter comma-separated keys. You can select up to 1,000 spaces in this form.
+2. Enter the hostname under **Atlassian site**, then choose **Spaces**. Use **All** in the dropdown for all spaces accessible at each sync, or the arrows beside **Spaces** to enter comma-separated keys. You can select up to 1,000 individual spaces in this form.
 3. Select **Connect & Sync**. Sim saves the selected scope and starts indexing with your account.
 
 <Image className="mx-auto h-auto w-full max-w-md"
@@ -87,6 +89,7 @@ Use a **scoped API token** from an Atlassian service account:
 read:confluence-content.all
 read:page:confluence
 read:blogpost:confluence
+read:attachment:confluence
 read:space:confluence
 read:label:confluence
 search:confluence
@@ -98,7 +101,7 @@ read:user:confluence
 read:group:confluence
 ```
 
-Use all 12 scopes for account validation, pickers, content, permissions, and directory reads. Central indexing does not need write scopes.
+Use all 13 scopes for account validation, pickers, content, attachments, permissions, and directory reads. Central indexing does not need write scopes.
 
 4. Review and create the token, then copy it. Atlassian shows it only once.
 5. In Sim's source form, open **Service account → Add service account**. Paste the **API token**, enter **Site domain** (hostname only), and select **Add service account**. Continue in the source form with the same domain.
@@ -122,7 +125,9 @@ See Atlassian's [account setup](https://support.atlassian.com/user-management/do
 | **Filter by Label** | Optional comma-separated labels; content can match any listed label. |
 | **Metadata tags** | Labels, version, and last-modified tags. |
 
-Search manages the schedule and hides item limits. It indexes published/current content and each page's own text, including supported local callouts and code blocks. Archived content, comments, attachment contents, and expanded Include Page, Excerpt Include, or third-party macro output are excluded. Referenced pages can be indexed separately with their own permissions.
+Search manages the schedule and hides item limits. It indexes published/current content and each page's own text, including supported local callouts and code blocks. PDF, Word `.docx`, and Word 97–2003 `.doc` attachments on the selected pages and blog posts are indexed as separate documents with their parent content's permissions. Space, content-type, and label filters apply to the parent content. Attachment changes are checked on each sync, even when the parent text has not changed.
+
+Archived content, comments, other attachment formats, and expanded Include Page, Excerpt Include, or third-party macro output are excluded. Referenced pages can be indexed separately with their own permissions. Attachments over 100 MB are shown as skipped; convert older Word 6/95 files to `.docx` before attaching them.
 
 ## Manage access and sync
 
@@ -146,6 +151,7 @@ In **Sync history**, **Continuing** means a healthy listing needs another batch.
 | A new page, blog post, or label is missing | Confluence search can take time to update. Once the content appears in Confluence search with the selected label, sync again. |
 | A restricted page is missing | Both your account and the crawling account need access to the page and its ancestors. |
 | Embedded content is missing | Index the referenced page separately; remote macro output is excluded. |
+| PDF or Word attachments are missing | Check `read:attachment:confluence` and access to the parent page. Existing service-account tokens may need to be replaced with one that includes this scope. Attachment access failures are reported as a partial sync. |
 | **Reconnect** or email mismatch | Authorize with the Atlassian account matching your verified Sim email and grant all requested permissions. |
 
 Open a missing page as the affected teammate, check its space and page restrictions, then sync again after correcting access. See Atlassian's [content access](https://support.atlassian.com/confluence-cloud/docs/add-or-remove-page-restrictions/) and [permission inspection](https://support.atlassian.com/confluence-cloud/docs/inspect-a-users-permissions/) guides.

apps/docs/content/docs/search/gmail.mdx

@@ -133,7 +133,7 @@ Search schedules syncs hourly. The first sync lists every thread in scope and ca
 
 **Member accounts:** later syncs use each mailbox's Gmail change history, unless the configuration has a search filter. A full relisting runs about weekly, or sooner if Gmail no longer retains the saved history.
 
-**Service account:** each sync revisits the selected active mailboxes and resumes unfinished listings. If Google reports that a user's mailbox is not set up or returns a mailbox `failedPrecondition`, Sim records a warning and continues with the remaining users. The crawl stays incomplete and retries affected users on the next scheduled crawl; existing indexed mail is not deleted because a mailbox could not be read. Credential, delegation, and Directory failures still stop the crawl.
+**Service account:** each user's progress is saved separately, so a large mailbox or an account-specific access failure does not hold up other users. Temporary failures retry automatically. Unavailable mailboxes are revisited without deleting indexed mail solely because the mailbox could not be read. Sim continues discovering users and refreshing existing mail and permissions while unfinished work resumes. Results with stale or unverified permissions remain hidden. Credential, delegation, Directory, and scope failures still require attention.
 
 Updates, removals, and access refresh in the background. Empty mailboxes and filters with no matches complete normally with zero documents. Threads exceeding indexing size limits are skipped and reconsidered when they change.