feat: contributors from crowd.dev git activity, refreshed daily (CM-1824)

[mbani01]

Summary

Second half of CM-1824, consuming the Tinybird snapshot shipped in #4886 (repo_commit_contributors_copy_ds, now live in lfx_insights).

Adds git_activity rows to repo_contributors: one commit-author row per repo and identity, for every repo crowd.dev tracks, with the CDP member id and the first/last commit timestamps (authored + co-authored). Bots, team members and organization profiles are already excluded upstream by the cleaned Tinybird source.

Changes

• Migration V1790954467: nullable cdp_member_id UUID on repo_contributors, plus repo_contributors_sync_state (source PK, watermark, updated_at).
• Sync module member-contributors/git-activity/: pages the Tinybird datasource in sorting-key order with a 4-column keyset cursor, maps channel to a packages-db repo (same canonicalisation as the governance sync) and username to an email or github-login identity, upserts with cdp_member_id; first_seen_at / last_seen_at use LEAST / GREATEST so channel spelling variants of one repo widen the window instead of overwriting it.
• Workflow + schedules on the security-contacts-worker queue, next to the governance sync:
  • git-activity-contributors-incremental-sync, 0 6 2-14,16-31 * *
  • git-activity-contributors-full-sync, 0 6 1,15 * *
• Tests for mapping, merging, watermark resolution, snapshot freshness, keyset paging and the sync paths.

How the incremental sync works

• The Tinybird copy pipe rebuilds the snapshot nightly at 03:30 UTC. Each row carries lastUpdatedAt = max(activityRelations.updatedAt), which moves on ingestion and on member merges.
• A run first reads max(computedAt) and throws if the snapshot is empty or not newer than the stored watermark, so a copy job that did not run is loud instead of silently reprocessing the same data.
• Daily runs read only rows with lastUpdatedAt > watermark - 1 day and upsert them. Nothing is deleted.
• The first run (no watermark) and the 1st/15th runs read the whole snapshot and reconcile: rows upserted get updated_at = NOW(), then git_activity rows with updated_at < run start are deleted. This closes the gaps a pure watermark cannot see (bot flag flips, repos disabled or re-enabled, identities absorbed by a merge, hard-deleted activities).
• A full run refuses to reconcile when more than 20% of the existing git_activity rows would be deleted, which is what a truncated-but-non-empty snapshot (truncate + Sequin backfill during the copy) looks like.
• The watermark is written only after the last page succeeded. A run that reads zero rows, reads rows but upserts none, or trips the reconcile guard throws and leaves state untouched.

Deployment

• CROWD_TINYBIRD_BASE_URL and CROWD_TINYBIRD_ACTIVITIES_TOKEN must be present in the security-contacts-worker deployment, and the token needs read access to repo_commit_contributors_copy_ds. To be checked after deploy.
• The first run is a full backfill of roughly 7M rows.

Depends on #4886 (merged).

[cursor]

PR Summary

Medium Risk
Large initial backfill and periodic full reconciles can bulk-delete git_activity rows; incorrect Tinybird snapshots are mitigated by freshness and removal guards but still affect contributor data quality.

Overview
Adds a git-activity contributor sync that pages the Tinybird repo_commit_contributors_copy_ds snapshot and upserts git_activity / commit-author rows into repo_contributors, including a new nullable cdp_member_id and a repo_contributors_sync_state watermark table.

The sync supports incremental runs (watermark minus one day, upsert only) and full runs (whole snapshot plus delete of untouched git_activity rows, with a 20% removal cap and snapshot freshness checks). Repo URLs are resolved like the governance sync; identities map to GitHub logins or emails. Temporal wires a new syncRepoContributorsFromGitActivity workflow on security-contacts-worker with daily incremental and twice-monthly full crons, registered at worker startup.

Unit tests cover mapping, merging, paging, watermark logic, and sync failure/success paths.

^{Reviewed by Cursor Bugbot for commit 55156e3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Copilot review overview

🟡 Changes recommended

Page-local aggregation can corrupt commit totals, and completed activity retries can fail due to the persisted watermark.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Prevent partial-count overwrites when source identities normalize together · New
• Make freshness check idempotent after watermark commit · New

What changed in this PR

Adds daily Tinybird-to-packages-db synchronization for repository commit contributors.

Changes:

• Adds contributor schema and watermark state.
• Implements full/incremental synchronization and mapping.
• Registers Temporal workflows, schedules, and tests.

File	Description
backend/​src/​osspckgs/​migrations/​V1790954467__repo_contributors_git_activity.sql	Adds contributor metrics and sync state.
services/​apps/​packages_worker/​src/​activities.ts	Exports the sync activity.
services/​apps/​packages_worker/​src/​bin/​security-contacts-worker.ts	Registers the new schedules.
services/​apps/​packages_worker/​src/​member-contributors/​activities.ts	Exposes the member-contributor activity.
services/​apps/​packages_worker/​src/​member-contributors/​git-activity/​mapRows.ts	Maps Tinybird rows to contributors.
services/​apps/​packages_worker/​src/​member-contributors/​git-activity/​readCommitContributors.ts	Implements Tinybird keyset paging.
services/​apps/​packages_worker/​src/​member-contributors/​git-activity/​syncGitActivityContributors.ts	Implements synchronization and reconciliation.
services/​apps/​packages_worker/​src/​member-contributors/​schedule.ts	Defines full and incremental schedules.
services/​apps/​packages_worker/​src/​member-contributors/​workflows.ts	Adds the Temporal workflow.
services/​apps/​packages_worker/​src/​member-contributors/​__tests__/​syncGitActivityContributors.test.ts	Tests mapping and synchronization behavior.
services/​apps/​packages_worker/​src/​workflows/​index.ts	Exports the workflow.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2ccbe53. Configure here.}

[mbani01]

Review Feedback Addressed

Commits: 0628316, d0e97f5

Changes Made

• syncGitActivityContributors.ts / mapRows.ts / migration: dropped commit_count. Not required by the ticket or any consumer, and not computable correctly from a snapshot keyed by raw channel + username: 5,899 canonical repos have more than one channel spelling in repos_to_channels, so page-local merging could not produce an all-time sum (per copilot-pull-request-reviewer, cursor)
• syncGitActivityContributors.ts: first_seen_at / last_seen_at now use LEAST / GREATEST in the upsert so channel variants widen the window instead of overwriting it, independent of page order or which variant an incremental run re-reads (per copilot-pull-request-reviewer, cursor)

No Change Needed

• syncGitActivityContributors.ts:96: the "not newer" retry failure is kept on purpose. The window is milliseconds at the end of the run, the data is already correct, and the failure self-heals at the next schedule. Treating computedAt == watermark as already applied would silently skip the more likely failure, a copy job that did not run (flagged by copilot-pull-request-reviewer)

Threads Resolved

3 of 3 unresolved threads addressed in this iteration.

[Copilot]

Copilot review overview

🟡 Changes recommended

Full reconciliation needs protection against deleting valid data when a nonempty but unexpectedly partial snapshot is received.

Review effort: Balanced
Findings: 1

Open (1)

• Prevent destructive reconcile on unexpectedly truncated snapshots · New

Resolved since last review (2)

• Prevent partial-count overwrites when source identities normalize together
• Make freshness check idempotent after watermark commit

[mbani01]

Review Feedback Addressed

Commit: 362bc8a

Changes Made

• syncGitActivityContributors.ts: the 1st/15th full reconcile now refuses to delete when more than 20% of the existing git_activity rows were left untouched by the run, which is what a truncated-but-non-empty Tinybird snapshot looks like. The run throws with the counts, nothing is deleted, the watermark is not advanced. (per copilot[bot])
• syncGitActivityContributors.test.ts: covers the guard passing on normal churn and refusing on an excessive drop.

Threads Resolved

1 of 1 unresolved thread addressed in this iteration.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The initial 7M-row reconciliation and production Tinybird access require final human operational validation.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Prevent destructive reconcile on unexpectedly truncated snapshots

[Copilot]

🟡 Changes recommended

A snapshot replacement during pagination can produce a mixed full run and delete valid contributor rows.

1 open finding

• Unpinned snapshot pagination can delete valid rows · New

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.