[INF-1722] Add configurable migration controller

[thesyncim]

Ticket

• https://linear.app/stream/issue/INF-1722

Summary

Add a private, restart-safe controller dashboard and replace the high-volume complete-UPDATE replay path with exact primary-key upserts modeled on the proven crdb-to-pg approach.

Implementation

• Serve pgmigrate controller on loopback by default; non-loopback listeners still require a token.
• Keep DSNs write-only and run migration/verification in isolated child processes.
• Render durable lifecycle, row changes/s, tx/s, WAL apply/source rates, net lag trend, ETA, verification, findings, failures, and guarded resume/stop controls.
• Trust a complete durable controller configuration after a pod restart so the tokenless UI and Resume control stay synchronized.
• Replay complete keyed UPDATE tuples as INSERT ... ON CONFLICT (<actual target primary key>) DO UPDATE.
• Retain the target primary-key ordinal order from pg_index, and force safe batched DELETEs through one exact PK-to-CTID lookup per identity so the planner cannot choose an unrelated secondary index.
• Use array unnest, typed temporary COPY stages for large custom-type groups, and bounded VALUES fallback.
• Retain the exact ordered fallback for unchanged TOAST values, PK-changing updates, unsafe/non-keyed relations, RLS, triggers/rules/checks, and other side-effect-sensitive cases.
• Preserve source ordering for repeated identities and unique/exclusion-sensitive transitions.
• Commit target DML, applied LSN, transaction count, and row-change count atomically with synchronous_commit=on.
• Expose crash-atomic replay batch byte/change ceilings in CLI and UI. Replay remains one ordered applier; the limits do not weaken source transaction order.

Safety review

• Missing complete UPDATE rows self-heal through idempotent PK upsert; missing DELETE rows still fail closed.
• Duplicate PKs are split across statements so PostgreSQL never updates one conflict row twice in one command.
• Target catalog capability checks gate the fast path; unsafe relations cannot opt in accidentally.
• Every failed/uncommitted target batch leaves the authoritative checkpoint unchanged.
• Resume starts strictly after the last target commit and reuses fsynced CDC segments on the PVC.
• No target indexes are dropped or rebuilt.
• Cutover and sequence advancement remain CLI-only.
• c13 is unchanged.

Validation

• GOCACHE=/private/tmp/pgmigrate-final-gocache go test ./... -count=1
• GOCACHE=/private/tmp/pgmigrate-final-gocache go vet ./...
• GOCACHE=/private/tmp/pgmigrate-final-gocache go test -race ./internal/cdc ./internal/controller -count=1
• PGTEST_MAJORS=17 go test -tags=integration ./internal/cdc -count=1
• PostgreSQL 17 coverage includes crash/retry, atomic ordered replay, unchanged TOAST, PK changes, typed stages, missing/duplicate identities, rollback, and exact final digest.
• Batch sweep on the same workstation: 16 MiB ~111k changes/s; 32 MiB ~123k changes/s; 128 MiB ~106k changes/s. The reviewed default remains 32 MiB; production was not restarted for a losing tuning guess.

Live c3 validation

• Immutable deployed image: pgmigrate:v57@sha256:4b8acc92b34cb48e76d7d252996413160654c29f005c676dd14f637e995c460f (amd64 + arm64), pinned by GetStream/chat#16022. The embedded Go build revision is 40728517acd4c1003c2fe12584fa20da0d70258e with vcs.modified=false.
• Pod has zero restarts, remounted the same PVC, loaded the exact target checkpoint, audited retained CDC, and advanced strictly forward.
• Target pg_stat_statements proves the live path uses ON CONFLICT on exact PKs such as channels(app_pk,cid) and channel_members(app_pk,channel_cid,user_id).
• A 577-second messages DELETE outlier was traced to composite probes ordered (id,app_pk) while messages_pkey is (app_pk,id). A production EXPLAIN proved that ordering alone still selected messages_by_user; the final CTID shape plans as Index Scan using messages_pkey plus Tid Scan.
• v57 resumed from the unchanged pre-rollout checkpoint 1BF9D/81DDB620 / 81,522,814 rows / 8,700,697 transactions and advanced past the formerly stuck transaction to 81,634,615 rows / 8,711,932 transactions.
• Final browser-rendered steady state after the restart window normalized: 2,819 row changes/s, 292 tx/s, 20.3 MiB/s applied WAL, 14.6 MiB/s source WAL, 5.8 MiB/s net lag drain, zero current blockers, and 0s checkpoint staleness.
• A 45-second durable sample advanced 108,999 rows and 14,277 source transactions while lag fell by 704,629,384 bytes.
• Browser-rendered steady state: about 2,511 row changes/s, 306 tx/s, 23.9 MiB/s applied WAL, 13.6 MiB/s source WAL, 10.3 MiB/s net lag drain, and ~52h ETA.
• Dashboard is tokenless/configured through the loopback-only port-forward, Resume survived the pod restart, and it shows zero current blockers; the old divergence is historical.

Checklist

• Unit, vet, race, and PostgreSQL 17 integration coverage pass
• Replay data and progress remain atomic and crash-durable
• Exact durable checkpoint advanced forward after restart
• Live SQL uses the intended target primary keys
• Tokenless loopback UI validated in the browser
• Real replay rate, WAL rate, lag trend, and ETA visible
• Batch tuning measured rather than guessed
• No index drop/rebuild and no durability weakening
• c13 left unchanged