fix(search): resume Google crawls around unavailable accounts (#7934)

* fix(search): resume Google crawls around unavailable accounts

* fix(search): harden crawl completion and recovery

* improvement(search): generalize connector partition progress

* fix(search): clear unverifiable placeholder permissions

Author: waleedlatif1

.github/workflows/test-build.yml

@@ -235,6 +235,11 @@ jobs:
           lib/knowledge/__integration__/search-source-pagination.integration.ts
           lib/knowledge/__integration__/search-reference-batching.integration.ts
           lib/knowledge/__integration__/embedding-insert-batches.integration.ts
+          lib/knowledge/__integration__/connector-lifecycle-locks.integration.ts
+          lib/knowledge/__integration__/connector-deferral.integration.ts
+          lib/knowledge/__integration__/stored-document-recovery.integration.ts
+          lib/knowledge/__integration__/connector-partition-work.integration.ts
+          lib/knowledge/__integration__/listing-continuation.integration.ts
           lib/knowledge/__integration__/kb-block-search.integration.ts
           lib/core/outbox/service.integration.ts
           lib/knowledge/__integration__/connector-upload.integration.ts

apps/docs/content/docs/search/gmail.mdx

@@ -133,7 +133,7 @@ Search schedules syncs hourly. The first sync lists every thread in scope and ca
 
 **Member accounts:** later syncs use each mailbox's Gmail change history, unless the configuration has a search filter. A full relisting runs about weekly, or sooner if Gmail no longer retains the saved history.
 
-**Service account:** each sync revisits the selected active mailboxes and resumes unfinished listings. If Google reports that a user's mailbox is not set up or returns a mailbox `failedPrecondition`, Sim records a warning and continues with the remaining users. The crawl stays incomplete and retries affected users on the next scheduled crawl; existing indexed mail is not deleted because a mailbox could not be read. Credential, delegation, and Directory failures still stop the crawl.
+**Service account:** each user's progress is saved separately, so a large mailbox or an account-specific access failure does not hold up other users. Temporary failures retry automatically. Unavailable mailboxes are revisited without deleting indexed mail solely because the mailbox could not be read. Sim continues discovering users and refreshing existing mail and permissions while unfinished work resumes. Results with stale or unverified permissions remain hidden. Credential, delegation, Directory, and scope failures still require attention.
 
 Updates, removals, and access refresh in the background. Empty mailboxes and filters with no matches complete normally with zero documents. Threads exceeding indexing size limits are skipped and reconsidered when they change.
 

apps/docs/content/docs/search/google-calendar.mdx

@@ -62,7 +62,7 @@ Admins can request member connections from **Settings → Sources → People →
 
 Open **Settings → Sources → Add source** and select **Google Calendar**. This opens **Connect Google Calendar service account**. To add another connection later, open **Google Calendar** from the Sources list and select **Connect service account**.
 
-This requires a Google Workspace customer and a super administrator to authorize domain-wide delegation. Consumer Gmail accounts cannot use this path.
+This requires a Google Workspace customer and a super administrator to authorize domain-wide delegation. Consumer Gmail accounts cannot use this path. In the Google Admin Console, confirm **Apps → Google Workspace → Calendar → Service status** is on for the users you want to index. An active Workspace account does not necessarily have Calendar enabled.
 
 <Steps>
 <Step>
@@ -131,7 +131,9 @@ Sim indexes event titles, descriptions, times, locations, and the selected atten
 
 Cancelled events, attachment contents, meeting recordings, and transcripts are not indexed. Status entries such as working location, out of office, focus time, and birthdays, and automatically generated reservation events from Gmail are not indexed. Events Google returns only as free/busy blocks, without searchable details, are not indexed. Events outside the selected date window are excluded. Private event details that Google withholds are not available in Search; see [Google's calendar sharing rules](https://developers.google.com/workspace/calendar/api/concepts/sharing).
 
-Search schedules syncs hourly. Event edits, cancellations, access changes, inactive or removed users, and events moving outside the date window are reconciled during completed background syncs. Central crawls page through each selected user and resume unfinished work before removing documents no longer listed. If an individual user's event listing returns a `403` with an explicit `forbidden` reason and no other reasons, Sim records a warning and continues with the remaining users. The crawl stays incomplete and retries affected users on the next scheduled crawl; unread calendars are not treated as empty. A `403` without a reason stops the crawl because its cause is unknown. Credential, delegation, Directory, and other provider failures still stop the crawl. The first sync may take longer, and results appear as indexing progresses; Search is not a live Calendar read.
+Search schedules syncs hourly. The first sync may span several background runs; results appear as indexing progresses. Central crawls save each user's progress so a large calendar or an account-specific access failure does not hold up other users. Temporary failures retry automatically. Unresolved accounts are revisited without treating unread calendars as empty; credential, delegation, Directory, and scope failures still require attention.
+
+Event edits, cancellations, access changes, inactive or removed users, and events moving outside the date window are reconciled in the background. Sim refreshes existing permissions while a longer crawl continues. Results with stale or unverified permissions remain hidden. Search is not a live Calendar read.
 
 ## Troubleshooting
 

apps/docs/content/docs/search/google-drive.mdx

@@ -151,7 +151,7 @@ Use **Sync now** to request an update, **Pause syncing** to stop scheduled syncs
 
 Sim exports Docs and Slides as text and Sheets as XLSX spreadsheets. Supported uploaded files use the knowledge-base document pipeline, including PDF and Office formats. Unsupported files and oversized exports cannot be indexed; Google limits Workspace exports to 10 MB. See [Drive export formats](https://developers.google.com/workspace/drive/api/guides/ref-export-formats) and [download limits](https://developers.google.com/workspace/drive/api/guides/manage-downloads).
 
-Search schedules syncs hourly. Central crawls revisit the selected users' files and permissions, including unchanged files, so permission changes and a new employee's older files are included. Unfinished crawls resume before deletion reconciliation. Content, deletions, and permissions refresh in the background; results are not a live read from Drive. **People** on the Sources settings page shows personal account connections; it does not list the central service-account credential.
+Search schedules syncs hourly. Central crawls save each user's progress separately, so a large Drive or an account-specific access failure does not hold up other users. Temporary failures retry automatically. Sim continues discovering users and refreshing files and permissions while unfinished work resumes. Unread files are not treated as deleted, and results with stale or unverified permissions remain hidden. Results are not a live read from Drive. **People** on the Sources settings page shows personal account connections; it does not list the central service-account credential.
 
 ## Troubleshooting
 
@@ -163,6 +163,7 @@ Directory permission warnings mean Sim could not fully verify group membership.
 | Directory access failed | Check all four delegated scopes and the **Directory administrator email** user's administrator privileges. A normal Google OAuth credential cannot supply this central Search path. |
 | Missing files in a central crawl | Check **Users**, folder and file-type filters, and whether selected active Workspace users can download the file and read its permissions. Opening a file alone does not prove either. Check Sync history for errors. Files reachable only by excluded or inactive accounts are not crawled; files with unverified permissions stay hidden. |
 | No text could be extracted | Images and scanned PDFs use OCR, but files with no extractable text cannot be indexed. Check whether the original contains readable text; a successful download does not guarantee searchable content. |
+| File contains a Git LFS pointer | The uploaded file references content stored elsewhere instead of containing the actual image or PDF. Replace it in Drive with the original file content, then sync again. Retrying the unchanged pointer cannot recover the content. |
 | User not found or inactive | Use a primary email in the same Google Workspace customer. Aliases, external or guest accounts, suspended users, and archived users cannot be selected for crawling. |
 | A teammate sees no results | Confirm they have joined the Sim organization and their verified Sim email matches the Drive permission or group membership. For member accounts, finish their personal Drive connection too. |
 | A public or shared-link file is missing | Check **Openly shared files**. Link-only sharing does not grant Search access. A named user or group permission can still make the file searchable. |

apps/sim/connectors/google-drive/company-crawl.test.ts

@@ -553,6 +553,14 @@ describe('Google Drive company-wide crawl', () => {
     ).rejects.toThrow()
     expect(mockFetch).not.toHaveBeenCalled()
   })
+
+  it('passes the crawl signal into the delegated token exchange', async () => {
+    fixture({})
+    const controller = new AbortController()
+    const ctx = { ...context(), signal: controller.signal }
+    await drive.listDocuments('directory-token', CONFIG, undefined, ctx)
+    expect(ctx.getDelegatedAccessToken).toHaveBeenCalledWith('alice@corp.com', controller.signal)
+  })
 })
 
 describe('Company-wide setup validation', () => {

apps/sim/connectors/google-drive/company-crawl.ts

@@ -6,6 +6,10 @@ import {
   GOOGLE_WORKSPACE_DRIVES_PAGE_SIZE,
   listGoogleWorkspaceDrives,
 } from '@/connectors/google-drive/workspace-drives'
+import {
+  type GoogleCompanyCursorAdapter,
+  googleCompanyUserContextSchema,
+} from '@/connectors/google-workspace/company-work'
 import {
   GOOGLE_WORKSPACE_USERS_PAGE_SIZE,
   getGoogleWorkspaceUser,
@@ -18,15 +22,7 @@ import { parseOptionalUnlimitedSafeInteger } from '@/connectors/utils'
 const CURSOR_PREFIX = 'gdrive-company:v1:'
 const MAX_CURSOR_BYTES = 384 * 1024
 const cursorSchema = z.object({
-  users: z
-    .array(
-      z.object({
-        id: z.string().min(1).max(256),
-        email: z.string().email().max(254),
-        customerId: z.string().min(1).max(256),
-      })
-    )
-    .max(GOOGLE_WORKSPACE_USERS_PAGE_SIZE),
+  users: z.array(googleCompanyUserContextSchema).max(GOOGLE_WORKSPACE_USERS_PAGE_SIZE),
   nextUsersPageToken: z.string().min(1).max(8192).optional(),
   scope: z.discriminatedUnion('kind', [
     z.object({
@@ -52,7 +48,7 @@ const cursorSchema = z.object({
 })
 type CompanyCursor = z.infer<typeof cursorSchema>
 
-type DelegatedTokenResolver = (subject: string) => Promise<string>
+type DelegatedTokenResolver = (subject: string, signal?: AbortSignal) => Promise<string>
 
 function delegatedTokenResolver(syncContext: Record<string, unknown>): DelegatedTokenResolver {
   if (typeof syncContext.getDelegatedAccessToken !== 'function') {
@@ -88,6 +84,21 @@ function writeCursor(cursor: CompanyCursor): string {
   return serialized
 }
 
+/** A single-user cursor preserves shared-drive traversal as well as the user's own Drive page. */
+export const googleDriveCompanyCursorAdapter: GoogleCompanyCursorAdapter = {
+  seed: (user) => writeCursor({ users: [user], scope: { kind: 'user' } }),
+  resume: (cursor) => {
+    const state = readCursor(cursor)
+    return state.users.map((user, index) => ({
+      user,
+      cursor: writeCursor({
+        users: [user],
+        scope: index === 0 ? state.scope : { kind: 'user' },
+      }),
+    }))
+  },
+}
+
 /** Positive caps can stop before later users; central sources must finish their selected corpus. */
 function validateCompanyConfig(sourceConfig: Record<string, unknown>): string[] {
   if (
@@ -197,7 +208,7 @@ export async function listGoogleCompanyDocuments(input: {
   }
 
   signal?.throwIfAborted()
-  const userToken = await resolveToken(user.email)
+  const userToken = await (signal ? resolveToken(user.email, signal) : resolveToken(user.email))
   const nextDrives = async (pageToken?: string): Promise<CompanyCursor> => {
     const drives = await listGoogleWorkspaceDrives(userToken, pageToken, signal)
     if (drives.driveIds.length > 0) {

apps/sim/connectors/google-drive/google-drive-errors.ts

@@ -12,6 +12,7 @@ import {
 import {
   ConnectorSourceError,
   type ConnectorSourceFailureCategory,
+  type ConnectorSourceReasonState,
 } from '@/connectors/source-error'
 
 const GOOGLE_ERROR_REASON_MAX_COUNT = 16
@@ -110,7 +111,8 @@ export class GoogleDriveApiError extends ConnectorSourceError {
     status: number,
     normalizedReasons: readonly string[],
     operation = 'drive.request',
-    reasonsComplete = true
+    reasonsComplete = true,
+    reasonState?: ConnectorSourceReasonState
   ) {
     const diagnosticReasons = safeGoogleErrorReasons(normalizedReasons).slice(
       0,
@@ -122,7 +124,7 @@ export class GoogleDriveApiError extends ConnectorSourceError {
       `Google Drive API request failed with HTTP ${status}${reasonSuffix}`,
       status,
       diagnosticCategory(kind, status),
-      { operation, reasons: diagnosticReasons }
+      { operation, reasons: diagnosticReasons, ...(reasonState ? { reasonState } : {}) }
     )
     this.name = 'GoogleDriveApiError'
     this.reasons = diagnosticReasons
@@ -144,7 +146,13 @@ export async function readGoogleDriveApiError(
   operation = 'drive.request'
 ): Promise<GoogleDriveApiError> {
   const details = await readGoogleErrorDetails(response)
-  return new GoogleDriveApiError(response.status, details.reasons, operation, details.complete)
+  return new GoogleDriveApiError(
+    response.status,
+    details.reasons,
+    operation,
+    details.complete,
+    details.reasonState
+  )
 }
 
 /**

apps/sim/connectors/google-workspace/api-errors.test.ts

@@ -32,6 +32,7 @@ describe('Google API diagnostics', () => {
       new Error('outer private detail', { cause: error })
     )
     expect(diagnostic).toMatchObject({ status, category, operation: OPERATION, reasons: [reason] })
+    expect(diagnostic?.reasonState).toBe('present')
     expect(JSON.stringify(diagnostic)).not.toContain(RESPONSE_SECRET)
     expect(JSON.stringify(diagnostic)).not.toContain('outer private detail')
   })
@@ -40,6 +41,7 @@ describe('Google API diagnostics', () => {
     const error = await readGoogleApiError(failure(403, RESPONSE_SECRET), OPERATION)
     expect(error.diagnostic?.reasons).toEqual([])
     expect(error.reasonsComplete).toBe(false)
+    expect(getConnectorFailureDiagnostic(error)?.reasonState).toBe('filtered')
     expect(JSON.stringify(error)).not.toContain(RESPONSE_SECRET)
   })
 
@@ -82,11 +84,13 @@ describe('Google API diagnostics', () => {
     )
     expect(absent.reasonsComplete).toBe(true)
     expect(absent.diagnostic?.reasons).toEqual([])
+    expect(getConnectorFailureDiagnostic(absent)?.reasonState).toBe('absent')
     const malformed = await readGoogleApiError(
       Response.json({ error: { errors: [{ reason: 123 }] } }, { status: 403 }),
       'calendar.events.list'
     )
     expect(malformed.reasonsComplete).toBe(false)
+    expect(getConnectorFailureDiagnostic(malformed)?.reasonState).toBe('malformed')
     const mixed = await readGoogleApiError(
       Response.json(
         { error: { errors: [{ reason: 'forbidden' }, { reason: RESPONSE_SECRET }] } },
@@ -96,6 +100,7 @@ describe('Google API diagnostics', () => {
     )
     expect(mixed.diagnostic?.reasons).toEqual(['forbidden'])
     expect(mixed.reasonsComplete).toBe(false)
+    expect(getConnectorFailureDiagnostic(mixed)?.reasonState).toBe('filtered')
     expect(JSON.stringify([absent, malformed, mixed])).not.toContain(RESPONSE_SECRET)
   })
 
@@ -120,8 +125,61 @@ describe('Google API diagnostics', () => {
     )
     expect(error.diagnostic?.reasons).toEqual(['forbidden'])
     expect(error.reasonsComplete).toBe(false)
+    expect(getConnectorFailureDiagnostic(error)?.reasonState).toBe('malformed')
     expect(JSON.stringify(error)).not.toContain(RESPONSE_SECRET)
   })
+
+  it('distinguishes an unreadable body from a malformed response', async () => {
+    const unreadable = new Response(
+      new ReadableStream({
+        start(controller) {
+          controller.error(new Error(RESPONSE_SECRET))
+        },
+      }),
+      { status: 403 }
+    )
+    const error = await readGoogleApiError(unreadable, OPERATION)
+    expect(getConnectorFailureDiagnostic(error)?.reasonState).toBe('unreadable')
+    expect(error.reasonsComplete).toBe(false)
+    expect(JSON.stringify(error)).not.toContain(RESPONSE_SECRET)
+    const malformed = await readGoogleApiError(new Response('not-json', { status: 403 }), OPERATION)
+    expect(getConnectorFailureDiagnostic(malformed)?.reasonState).toBe('malformed')
+  })
+
+  it('retains classification evidence beyond the diagnostic reason limit', async () => {
+    const reasons = [
+      'accessNotConfigured',
+      'appNotAuthorizedToFile',
+      'authError',
+      'badRequest',
+      'cannotDownloadFile',
+      'cannotExportFile',
+      'domainPolicy',
+      'download_restricted_for_revision',
+      'exportSizeLimitExceeded',
+      'failedPrecondition',
+      'fileNotDownloadable',
+      'fileNotExportable',
+      'forbidden',
+      'insufficientFilePermissions',
+      'insufficientPermissions',
+      'invalid',
+      'rateLimitExceeded',
+    ]
+    const error = await readGoogleApiError(
+      Response.json(
+        {
+          error: { errors: reasons.map((reason) => ({ reason })) },
+        },
+        { status: 403 }
+      ),
+      OPERATION
+    )
+    expect(error.rateLimited).toBe(true)
+    expect(error.reasonsComplete).toBe(false)
+    expect(error.diagnostic?.reasonState).toBe('filtered')
+    expect(error.diagnostic?.reasons).toHaveLength(16)
+  })
 })
 
 describe('Google API retries', () => {