You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: nightly mirror sweep of github-nango repos into shadow sync units - #5059
Every github-nango repo moves to the connectors pipeline. The switch is automated (nightly sweep, #5049), but mirroring a repo into shadow sync units is still a hand-fed pod script. This adds the nightly job that keeps a bounded pool of repos in shadow, refilled as the sweep drains it, most active repos first, in batches the operator grows over time.
Temporal schedule connectors-mirror-sweep on the connectors queue, daily at 03:00Z (after the 00:00Z diff and the 02:00Z switch sweep), overlap SKIP, 2-hour execution timeout. Workflow mirrorSweep({ batchSize, shadowCap }):
inShadow = distinct repos with at least one non-emitting, non-decommissioned unit on a live integration.
Candidates = active github-nango mappings with no sync units for that repo in that integration (name match), ordered enabled/not archived/not excluded first, then 30-day activity from Tinybird activityRelations (uniqExact(activityId) per channel), then owner/repo.
Mirror the top room repos in chunks of 100, 5 GitHub resolutions in flight, through the same mirrorRepo path the pod script now uses: credential → Nango connection → installation token → resolve repo on GitHub → eight shadow units with since = start of the UTC day 48 h back. emitEnabled stays false.
One summary log line per run.
Knobs (schedule input, no deploy)
batchSize: new repos per night. Starts at 500. Raise it as the diff and the sweep hold up:
Schedule creation in the worker swallows ScheduleAlreadyRunning and does not update, so operator-set values survive deploys.
shadowCap: repos in shadow at once, 5,000. Never exceeded.
Skips
A repo that cannot be mirrored is logged once per run with event: mirror_repo_skipped and one reason: no_installation_id, token_mint_failed, repo_not_found, repo_renamed (GitHub redirects to a different name; mirroring it under the new URL would never be switched, fix the mapping by hand), github_error, nango_error, already_mirrored (units already exist for that node id; the upsert would rewind the watermark). Credential lookup errors (DB outage, missing GitHub App secret) are not skips: they fail the chunk, the run stops at that chunk with chunksFailed: 1, and the next night refills. A malformed schedule input (non-positive or non-integer values, batchSize > shadowCap) fails the workflow immediately instead of running as a silent no-op. Skips are not stored: a broken repo is retried and logged every night until fixed. The pod script script:mirror-nango-repos shares the path and now exits 1 instead of re-upserting when a repo is already mirrored.
The connectors-worker deployment needs CROWD_TINYBIRD_ACTIVITIES_TOKEN (and CROWD_TINYBIRD_BASE_URL if not default). The Tinybird query has not been run against prod from this branch; the first scheduled run fails loudly if it is wrong and mirrors nothing.
The switch sweep is serial today (~40 s/repo); it may still be running at 03:00Z, so the pool refills a night later than the pace suggests. Sweep concurrency is a separate PR.
Expected pace from a 10-22 deploy with batch size doubling every second night: cap binds ~10-28, then ~1,200–1,700 repos/night as the sweep drains; all mirrored ~11-03.
Known rough edges
If a chunk activity is retried after a partial failure, repos mirrored on the first attempt are counted as already_mirrored skips on the retry, so that night's mirrored/unitsInserted under-count. Data is unaffected.
The lockfile re-resolves axios 1.19 → 1.20 for @sendgrid (the catalog pins ^1.20.0); incidental to adding axios to the worker.
Not in this PR
Sweep concurrency; a stored channel name in the already_mirrored message; ON CONFLICT DO NOTHING insert variant (the guard + the candidate query cover the sweep; a same-second rerun of the script and the sweep on one repo could rewind a same-day since).
Medium Risk
Writes integration sync units and calls GitHub/Nango at scale on a schedule; misconfiguration or Tinybird/query failures can block mirroring, but bounds and skip handling limit blast radius.
Overview
Adds an automated nightly mirror sweep so github-nango repos are copied into shadow sync_units without running a pod script, keeping a capped pool that the switch sweep can drain.
A Temporal schedule (connectors-mirror-sweep, 03:00Z) runs workflow mirrorSweep with batchSize / shadowCap. It counts repos already in shadow, picks unmirrored Nango mappings (prioritizing active repos and 30-day GitHub activity from Tinybird), then mirrors in chunks of 100 with bounded GitHub concurrency. Shared mirrorRepo handles Nango → token → channel resolve → eight incremental shadow units (48h lookback), with structured skip reasons and summary logging.
Data layer:listUnmirroredNangoRepos, countShadowRepos, and hasSyncUnitsForChannel support selection and deduplication. The script:mirror-nango-repos CLI is refactored onto mirrorRepo and exits non-zero on skips. Connectors worker gains @crowd/database (Tinybird) and axios for error classification.
Reviewed by Cursor Bugbot for commit 5a0ad0e. Bugbot is set up for automated code reviews on this repo. Configure here.
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You have signed the CLA already but the status is still pending? Let us recheck it.
Wait for every repo in the group to settle before failing the activity. If a credential lookup or insert rejects, Promise.all rejects while sibling calls continue. If those calls outlast the retry backoff, Temporal can retry the same chunk alongside them, exceeding the five-repo concurrency limit. Overlapping attempts can also both pass the channel-existence guard; upsertSyncUnits overwrites watermarks on conflict. Drain the pending calls before rethrowing the original error.
Wait for every operation in the group to settle before failing the activity. If a credential or database lookup rejects while peers are awaiting Nango or GitHub, Promise.all rejects immediately but leaves those peers running. Temporal can retry the chunk before they finish, exceeding the five-repository concurrency limit and racing the channel-id check and watermark-updating upsert, even with unique mappings. Use Promise.allSettled and propagate failures only after all peers finish; this prevents overlap caused by an early rejection.
If a credential lookup or database operation rejects, Promise.all fails immediately while the other mirrors keep running. If those calls outlast the retry delay, the next activity attempt can exceed the five-repository concurrency limit and race the same repositories through the existence check and watermark-updating upsert. Await all started mirrors with Promise.allSettled before propagating a failure so retries cannot overlap unfinished work from this failure path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-license-compliance[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Why
Every github-nango repo moves to the connectors pipeline. The switch is automated (nightly sweep, #5049), but mirroring a repo into shadow sync units is still a hand-fed pod script. This adds the nightly job that keeps a bounded pool of repos in shadow, refilled as the sweep drains it, most active repos first, in batches the operator grows over time.
Design:
docs/superpowers/specs/2026-10-20-issue-833-automated-mirroring-design.md(local).What runs
Temporal schedule
connectors-mirror-sweepon theconnectorsqueue, daily at 03:00Z (after the 00:00Z diff and the 02:00Z switch sweep), overlap SKIP, 2-hour execution timeout. WorkflowmirrorSweep({ batchSize, shadowCap }):inShadow= distinct repos with at least one non-emitting, non-decommissioned unit on a live integration.room = max(0, min(batchSize, shadowCap − inShadow)); zero → summary only.activityRelations(uniqExact(activityId)per channel), then owner/repo.roomrepos in chunks of 100, 5 GitHub resolutions in flight, through the samemirrorRepopath the pod script now uses: credential → Nango connection → installation token → resolve repo on GitHub → eight shadow units withsince= start of the UTC day 48 h back.emitEnabledstays false.Knobs (schedule input, no deploy)
batchSize: new repos per night. Starts at 500. Raise it as the diff and the sweep hold up:ScheduleAlreadyRunningand does not update, so operator-set values survive deploys.shadowCap: repos in shadow at once, 5,000. Never exceeded.Skips
A repo that cannot be mirrored is logged once per run with
event: mirror_repo_skippedand one reason:no_installation_id,token_mint_failed,repo_not_found,repo_renamed(GitHub redirects to a different name; mirroring it under the new URL would never be switched, fix the mapping by hand),github_error,nango_error,already_mirrored(units already exist for that node id; the upsert would rewind the watermark). Credential lookup errors (DB outage, missing GitHub App secret) are not skips: they fail the chunk, the run stops at that chunk withchunksFailed: 1, and the next night refills. A malformed schedule input (non-positive or non-integer values,batchSize > shadowCap) fails the workflow immediately instead of running as a silent no-op. Skips are not stored: a broken repo is retried and logged every night until fixed. The pod scriptscript:mirror-nango-reposshares the path and now exits 1 instead of re-upserting when a repo is already mirrored.Observability
event: mirror_sweep_summary—batchSize, shadowCap, inShadow, candidates, attempted, mirrored, skipped, unitsInserted, chunksFailed, durationMs.event: mirror_repo_skipped—integrationId, owner, repoName, reason, message.integration.sync_units(Metabase).Rollout
CROWD_TINYBIRD_ACTIVITIES_TOKEN(andCROWD_TINYBIRD_BASE_URLif not default). The Tinybird query has not been run against prod from this branch; the first scheduled run fails loudly if it is wrong and mirrors nothing.Known rough edges
already_mirroredskips on the retry, so that night'smirrored/unitsInsertedunder-count. Data is unaffected.@sendgrid(the catalog pins^1.20.0); incidental to adding axios to the worker.Not in this PR
Sweep concurrency; a stored channel name in the
already_mirroredmessage;ON CONFLICT DO NOTHINGinsert variant (the guard + the candidate query cover the sweep; a same-second rerun of the script and the sweep on one repo could rewind a same-daysince).Refs linuxfoundation/lfx-cdp-insights-ops#833