improvement(file-search): publish complete indexes in bounded text chunks

[icecrasher321]

Summary

• Store complete file revisions in immutable text chunks capped at 8 KiB, with atomic publication and full-line regex verification.
• Keep the 25 MiB source/text policy, exclude incomplete files in full, and remove CSV/spreadsheet preview limits from indexing. Document parser safeguards in the File block docs.
• Preserve agent search-to-read workflows with matching extracted-text line numbers, shared complete parsing for Get Content, and bounded line-window selection.
• Bound query work, backfill, retries, and cleanup; pause new indexing when retired chunks back up. Preserve legacy tables for a separate retirement deploy after backfill and the rollback window.

Type of Change

• Improvement

Testing

• 1,297 unit tests for search, parsers, compiled-artifact reads, and agent search-to-ranged-read regressions; PostgreSQL 16/17 tests for publication, retry fencing, revision changes, deletion, cleanup/admission, pagination, scope, and regex boundaries.
• Local synthetic benchmark: 4,000 files / 225 MiB / 32,000 chunks. Five runs each: common searches 67–81 ms, rare searches 16–20 ms, no-match searches 12–16 ms. These are local measurements, not production capacity guarantees.
• 475 targeted application/API tests rerun after forwarding request cancellation through text extraction and artifact reads.
• Lint, app/database type checks, repository audits, API boundary validation, migration safety, and generated-doc checks.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 17, 2026 10:58pm UTC

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the previously reported compatibility and cancellation issues are fixed, and no actionable new regression remains.

Summary

This PR replaces partial workspace-file indexing with complete, immutable, bounded text chunks and aligns search and ranged content reads around the same extraction representation. Changes since the previous review also complete request-cancellation propagation through source downloads, compiled-artifact reads, and parsers.

• Publishes complete file revisions atomically as text chunks capped at 8 KiB.
• Excludes files wholesale when source, extraction, format, or parser safety limits prevent complete indexing.
• Preserves extracted-text line numbers between File Search and Get Content.
• Adds bounded query, retry, backfill, cleanup, and admission behavior.
• Supports PostgreSQL 16 and 17 through shared transaction-timeout configuration.
• Documents extraction limits, index coverage semantics, and search-to-read workflows.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Workspace file revision] --> B{Eligible and within source limit?}
    B -- No --> C[Mark revision skipped]
    B -- Yes --> D[Complete text extraction]
    D --> E{Extraction complete and within limits?}
    E -- No --> C
    E -- Yes --> F[Split into bounded text chunks]
    F --> G[Publish revision atomically]
    G --> H[File Search returns fileId and extracted-text line number]
    H --> I[Get Content reads same representation]
    I --> J[Select bounded line window]

Loading

_{Reviews (5) · Last reviewed commit: "fix(file-search): cancel abandoned text ..."}

[cubic-dev-ai]

All reported issues were addressed across 43 files

<sub>Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 46 files

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic</sub>

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 46 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic</sub>

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 54 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 55 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic</sub>