Skip to content

fix(search): resume Google crawls around unavailable accounts - #7934

Merged
waleedlatif1 merged 4 commits into
stagingfrom
codex/google-suite-crawl-recovery
Sep 17, 2026
Merged

waleedlatif1 merged 4 commits into
stagingfrom
codex/google-suite-crawl-recovery

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Persist crawl and permission progress in a shared connector-partition store so slow or unavailable Google accounts do not block healthy users. Provider adapters validate bounded context; retries, cadence, and permission filtering are preserved.
  • Bound Google token retries, retain safe error diagnostics, avoid unnecessary OCR on Git LFS pointers, and reduce lifecycle lock contention during document recovery.
  • Update the Google Search guides and add PostgreSQL regression coverage to CI.
  • Apply the additive migration before deploying updated sync workers. New workers adopt existing provider checkpoints. Older workers safely restart new-format crawls on rollback, so mixed-version takeovers do not preserve progress.

Type of Change

  • Bug fix

Testing

  • 1,383 targeted unit tests passed; one database-gated test skipped.
  • 63 disposable PostgreSQL integration tests passed, including checkpoint replay, permission safety, locking, recovery, and atomic embedding writes.
  • Verified the actual migration against the previous schema, including old-code CRUD compatibility, metadata bounds, and forward/rollback checkpoint handling for all three Google providers.
  • Cleanup, typecheck, lint, 46 repository audits, API boundary validation, docs manifest, and migration safety checks passed.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Sep 17, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Sep 17, 2026 7:37pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge; all previous findings are resolved and the latest changes introduce no new actionable issues.

Summary

This PR makes Google Workspace central crawls durable on a per-user basis so unavailable accounts no longer block healthy users or cause unread data to be reconciled as absent.

  • Persists partitioned content and permission progress with bounded retry scheduling and legacy-checkpoint adoption.
  • Preserves permission safety during refreshes and document recovery.
  • Bounds delegated-token retries and propagates cancellation through token exchange.
  • Adds migration, PostgreSQL integration coverage, CI execution, and updated Google Search documentation.
  • The latest changes add the previously missing interface TSDoc and strengthen regression coverage for rejected ACL writes and stale permission evidence.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  S[Google company crawl scheduler] --> D[Discover Workspace users]
  D --> P[(Connector partition store)]
  P --> U[Select eligible user work]
  U --> T[Obtain delegated token]
  T --> L[List content or permissions]
  L --> C[Atomically save cursor and status]
  C -->|More work| P
  C -->|Temporary or account-specific failure| R[Persist bounded retry time]
  R --> P
  C -->|Partition complete| E[Continue healthy users]
  E --> F[Complete generation and reconcile safe absence]
Loading

Reviews (4) · Last reviewed commit: "fix(search): clear unverifiable placehol..."

Comment thread apps/sim/lib/knowledge/connectors/sync-content-pass.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/google-company-store.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 49 files

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread apps/sim/lib/knowledge/connectors/google-company-store.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/sync-content-pass.ts Outdated
Comment thread apps/sim/lib/knowledge/__integration__/google-company-work.integration.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/google-company-scheduler.ts
Comment thread apps/sim/lib/knowledge/documents/processing-recovery.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/partition-store.ts
Comment thread apps/sim/lib/knowledge/documents/ocr-source-validation.ts Outdated
Comment thread apps/sim/lib/knowledge/__integration__/listing-continuation.integration.ts Outdated
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 49 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic

@waleedlatif1
waleedlatif1 force-pushed the codex/google-suite-crawl-recovery branch from 8cff97e to 64cfcc4 Compare September 17, 2026 19:23
@waleedlatif1
waleedlatif1 force-pushed the codex/google-suite-crawl-recovery branch from 64cfcc4 to e07224b Compare September 17, 2026 19:23
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

Comment thread apps/sim/connectors/google-workspace/company-work.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 51 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread apps/sim/lib/knowledge/connectors/sync-content-pass.ts Outdated
Comment thread apps/sim/lib/knowledge/__integration__/listing-continuation.integration.ts Outdated
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 51 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit 4e49225 into staging Sep 17, 2026
35 checks passed
@waleedlatif1
waleedlatif1 deleted the codex/google-suite-crawl-recovery branch September 17, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant