chore(file-search): retire legacy index storage

Author: icecrasher321

apps/sim/lib/workspace-files/search/README.md

@@ -43,11 +43,11 @@ Search results retain `fileId`, 1-based `lineNumber`, and bounded `text` preview
 1. Deploy the additive migration and new application/Trigger worker versions. Legacy index tables remain readable for the old deployment. The file trigger queues current revisions in the new table; this cutover intentionally allows temporary search unavailability while the new index builds.
 2. The dispatcher uses a separate `workspace-file-search-chunks-v2` backfill cursor. It seeds at most 1,000 active files per pass under a shared file lock, with idempotent inserts. Normal dispatch caps remain two outstanding jobs per workspace, 100 outstanding globally, and ten running workers. Reconciliation repeats hourly after a complete pass to repair missing metadata. Failed revisions remain visible as failed; they are not silently declared covered.
 3. Before retiring legacy storage, verify the new app and Trigger workers are fully deployed, old runs/retries have drained, the backfill cursor has completed, and scoped coverage is ready or explicitly excluded. Investigate failed or stale pending revisions. Check cleanup backlog and run representative exact/regex searches, including long lines and folder scopes.
-4. After the rollback window, ship a separate contract PR removing the legacy schema and dropping `workspace_file_search_segment` / `workspace_file_search_index` with a short lock timeout. Do not delete the entire old index row-by-row or backfill it inside the schema migration. Dropping obsolete tables reclaims their heap, indexes, and TOAST together. The `contract-pending` marker in `packages/db/schema.ts` tracks this step.
+4. After the rollback window and the checks above, deploy `0368_retire_legacy_file_search.sql`. It drops only `workspace_file_search_segment` and `workspace_file_search_index`, atomically, with a two-second lock timeout and without `CASCADE`. An unexpected dependency or lock conflict aborts the migration instead of removing dependent objects. Replay tolerates tables already retired. Dropping the tables reclaims their heap, indexes, and TOAST together; it does not delete current chunks or rebuild search.
 
-Until that contract deploy, legacy foreign-key cascades can still make a hard file/workspace deletion expensive. New-index cleanup is bounded; retaining the old schema cannot erase that legacy cost. The earlier timestamp-repair script detects the chunk schema and leaves obsolete legacy text for this contract step instead of deleting it in bulk. Legacy-table retirement remains a separate contract migration.
+The status enum, dispatch queue, and v2 backfill cursor remain in use. The timestamp-repair script still preserves provenance and repairs current revisions; its legacy deletion branch has been removed. No bulk deletion or replacement backfill runs in the contract migration.
 
-Rollback before retirement requires restoring the old trigger function as well as the old app/worker version, and reconciling legacy revisions written during the cutover. Do not assume retained tables are automatically up to date. Canonical revision joins prevent stale content from being returned.
+After retirement, rollback must stay on a chunk-compatible app and worker release. Restoring the old segment-based implementation requires recreating and rebuilding its retired storage; reverting application code alone is insufficient.
 
 ### Direct GIN writes
 

apps/sim/lib/workspace-files/search/chunks.integration.ts

@@ -187,6 +187,7 @@ describe('chunked workspace file search on PostgreSQL', () => {
       '0358_workspace_file_content_version_precision.sql',
       '0359_workspace_file_search_chunks.sql',
       ginWriteMigration,
+      '0368_retire_legacy_file_search.sql',
     ]) {
       await applyMigration(migration)
     }
@@ -208,7 +209,7 @@ describe('chunked workspace file search on PostgreSQL', () => {
   })
   beforeEach(async () => {
     await connection`TRUNCATE workspace, workspace_files, workspace_file_search_revision, workspace_file_search_build,
-      workspace_file_search_chunk, workspace_file_search_index, workspace_file_search_segment, workspace_file_search_dispatch_queue, workspace_file_search_backfill`
+      workspace_file_search_chunk, workspace_file_search_dispatch_queue, workspace_file_search_backfill`
     await connection`INSERT INTO workspace_file_search_backfill (id, completed_at) VALUES ('workspace-file-search-chunks-v2', now())`
     await addFile('file-1')
   })
@@ -222,6 +223,45 @@ describe('chunked workspace file search on PostgreSQL', () => {
     }
   })
 
+  it('retires legacy tables atomically, preserves current search, and safely replays', async () => {
+    await index('heading\nretirement needle\ntail')
+    await connection`CREATE TABLE workspace_file_search_index (file_id text PRIMARY KEY)`
+    await connection`CREATE TABLE workspace_file_search_segment (file_id text, content text)`
+    await connection`INSERT INTO workspace_file_search_index VALUES ('retired-file')`
+    await connection`INSERT INTO workspace_file_search_segment VALUES ('retired-file', 'retired text')`
+    await connection`CREATE VIEW legacy_dependency AS SELECT * FROM workspace_file_search_index`
+    try {
+      await expect(applyMigration('0368_retire_legacy_file_search.sql')).rejects.toMatchObject({
+        code: '2BP01',
+      })
+      expect(
+        (await connection`SELECT count(*)::int AS count FROM workspace_file_search_segment`)[0]
+          .count
+      ).toBe(1)
+      await connection`DROP VIEW legacy_dependency`
+      for (let attempt = 0; attempt < 2; attempt++) {
+        await applyMigration('0368_retire_legacy_file_search.sql')
+        expect(
+          (
+            await connection`SELECT to_regclass('workspace_file_search_index') AS legacy_index,
+          to_regclass('workspace_file_search_segment') AS legacy_segment`
+          )[0]
+        ).toEqual({ legacy_index: null, legacy_segment: null })
+        expect((await search('retirement needle')).results).toMatchObject([
+          { fileId: 'file-1', lineNumber: 2 },
+        ])
+        expect((await search('^retirement.*needle$', 'regex')).results).toMatchObject([
+          { fileId: 'file-1', lineNumber: 2 },
+        ])
+      }
+    } finally {
+      await connection`DROP VIEW IF EXISTS legacy_dependency`
+      await applyMigration('0368_retire_legacy_file_search.sql')
+    }
+    await connection`DELETE FROM workspace_files WHERE id = 'file-1'`
+    expect((await search('needle')).results).toEqual([])
+  })
+
   it('preserves search through disabling, draining, and replaying GIN pending-list maintenance', async () => {
     await connection`ALTER INDEX workspace_file_search_chunk_content_idx SET (fastupdate = on)`
     try {

packages/db/migrations/0368_retire_legacy_file_search.sql

@@ -0,0 +1,7 @@
+SET LOCAL lock_timeout = '2s';
+--> statement-breakpoint
+-- migration-safe: contract of #7947; deployed app, workers, and revision triggers use chunk storage. Retire only after the rollback window and completed backfill verification.
+DROP TABLE IF EXISTS "workspace_file_search_segment";
+--> statement-breakpoint
+-- migration-safe: contract of #7947; current revision metadata lives in workspace_file_search_revision, with no remaining runtime reader of this legacy table.
+DROP TABLE IF EXISTS "workspace_file_search_index";