You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: nightly connector switch sweep with watermark-rewind handover - #5049
Nightly sweep. A Temporal schedule connectors-switch-sync-pipeline-sweep (02:00Z, overlap SKIP, catch-up 1h) starts switchSyncPipelineSweep, which lists every repo that still has a non-emitting, active, incremental GitHub shadow unit and runs switchSyncPipeline forward for it as a child workflow, one repo at a time. Each child gets only that repo's unswitched syncs, so already-switched units are not re-paused or touched again. The child uses the same per-repo workflow id as the pod script, so a manual switch and the sweep can never interleave: a repo whose switch is already running is counted as skipped. ParentClosePolicy.ABANDON lets a running switch finish even if the sweep is terminated. No cap: every sync that passes the clean gate is switched. Pausing the schedule in Temporal is the off switch.
Handover by watermark rewind instead of shadow replay. The forward switch no longer replays a unit's shadow rows through the results path. It reads the Nango sync's last run (finishedAt from the connection status), pauses Nango, and in one transaction flips emitEnabled, rewinds the unit's watermark to since = finishedAt - 1h with cursor = null and no covered windows (exactly what a completed run leaves behind), and deletes every shadow row of the unit. The reactivated unit then re-fetches from GitHub everything updated since Nango's last run, edits included, and the sink dedupes the overlap. Forks are cursor-walked in creation order and have no handoff gap, so their watermark is kept. A unit that is already emitting (rerun) only gets its shadow rows cleared. A Nango sync with no completed run is refused.
Why
The replay walked every remaining shadow row of a unit. Prod holds 2.6M shadow rows (7.9 GB), 2.25M of them older than five days, because the nightly diff only prunes matched rows whose event time falls inside the diffed day; re-captured old records never do. A fleet-wide sweep would have re-emitted all of them on night one and overwritten fresher Nango-delivered edits with stale captures. Rewinding since closes the same handoff gap with no replay code, and deleting the rows at switch time empties the table as units migrate.
Tests
Pure: sweep grouping of units into repos, handoverSince (1h overlap, forks exempt).
Workflow: sweep tallies switched/refused/unchanged/skipped/failed across children; forward switch order gate → quiesce → drain → re-gate → last run → pause → take over → activate, with the never-ran refusal and the heal path.
Activities: last-run read from the connection status, take-over rewinds or keeps the cursor walk.
DAL (CI DB): takeOverSyncUnit flips, rewinds a since walk, keeps a cursor walk, deletes the unit's shadow rows, and is a no-op on an emitting unit.
High Risk
Changes production sync cutover semantics and can rewind watermarks, pause/resume Nango, and bulk-delete shadow data across many repos on a schedule.
Overview
Replaces shadow-record replay on forward Nango→connector switches with a watermark rewind handover: read Nango connection status and CDP consumption to pick a handover time, pause Nango, then takeOverSyncUnit enables emit, rewinds since by one hour (cursor walks keep their watermark), and deletes all shadow rows in one transaction. Failures after pause resume Nango and reactivate the unit; already-emitting units only get shadow cleanup.
Adds a nightly Temporal sweep (connectors-switch-sync-pipeline-sweep, 02:00Z) that lists shadow-diff repos, groups unswitched syncs per repo, and runs switchSyncPipeline forward serially as child workflows with the same per-repo workflow id as the CLI (skip if already running). Removes unused shadow batch list/delete DAL helpers tied to replay.
Reviewed by Cursor Bugbot for commit 304f85f. Bugbot is set up for automated code reviews on this repo. Configure here.
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You have signed the CLA already but the status is still pending? Let us recheck it.
If activation fails on all three attempts before updating the unit, takeover has already committed and Nango is paused, but the connector remains paused too. The sweep records a failed child and moves on. Later sweeps cannot heal it because listShadowDiffUnits selects only active, non-emitting units (services/libs/data-access-layer/src/connectors/syncUnits.ts:382–383). Ingestion therefore remains stopped after the outage ends unless someone intervenes. Use durable activation retries after takeover, as below, or explicitly track interrupted handovers for recovery.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Refs linuxfoundation/lfx-cdp-insights-ops#833
What
Nightly sweep. A Temporal schedule
connectors-switch-sync-pipeline-sweep(02:00Z, overlap SKIP, catch-up 1h) startsswitchSyncPipelineSweep, which lists every repo that still has a non-emitting, active, incremental GitHub shadow unit and runsswitchSyncPipelineforward for it as a child workflow, one repo at a time. Each child gets only that repo's unswitched syncs, so already-switched units are not re-paused or touched again. The child uses the same per-repo workflow id as the pod script, so a manual switch and the sweep can never interleave: a repo whose switch is already running is counted as skipped.ParentClosePolicy.ABANDONlets a running switch finish even if the sweep is terminated. No cap: every sync that passes the clean gate is switched. Pausing the schedule in Temporal is the off switch.Handover by watermark rewind instead of shadow replay. The forward switch no longer replays a unit's shadow rows through the results path. It reads the Nango sync's last run (
finishedAtfrom the connection status), pauses Nango, and in one transaction flipsemitEnabled, rewinds the unit's watermark tosince = finishedAt - 1hwithcursor = nulland no covered windows (exactly what a completed run leaves behind), and deletes every shadow row of the unit. The reactivated unit then re-fetches from GitHub everything updated since Nango's last run, edits included, and the sink dedupes the overlap. Forks are cursor-walked in creation order and have no handoff gap, so their watermark is kept. A unit that is already emitting (rerun) only gets its shadow rows cleared. A Nango sync with no completed run is refused.Why
The replay walked every remaining shadow row of a unit. Prod holds 2.6M shadow rows (7.9 GB), 2.25M of them older than five days, because the nightly diff only prunes matched rows whose event time falls inside the diffed day; re-captured old records never do. A fleet-wide sweep would have re-emitted all of them on night one and overwritten fresher Nango-delivered edits with stale captures. Rewinding
sincecloses the same handoff gap with no replay code, and deleting the rows at switch time empties the table as units migrate.Tests
handoverSince(1h overlap, forks exempt).takeOverSyncUnitflips, rewinds a since walk, keeps a cursor walk, deletes the unit's shadow rows, and is a no-op on an emitting unit.Deploy
connectors-worker only. No migration.