chore: remove v1 health score computation and columns from Tinybird (IN-1222)

[gaspergrom]

Do not merge until Joana Maia confirms consumer sign-off (LFX One, Org Lens, OD, Self Serve, CNCF) and a deprecation date — IN-1222 PRE2/PRE4.

Summary

• Removes the v1 computation from Tinybird:
  • health_score_copy pipe and health_score_copy_ds datasource. The copy ran at 00:50 UTC. Deleting the pipe also removes its schedule.
  • health_score_overview pipe, which served the v1 overview route.
  • health_score_security and health_score_contributions_outside_work_hours. Only the benchmark route called them. Insights PR 2 stops those calls first.
• Removes the v1 columns: healthScore, contributorHealthScore, popularityHealthScore, developmentHealthScore and securityHealthScore from project_insights_copy_ds. Also healthScore from insights_projects_populated_ds, and the v1 fields from project_insights, project_repo_insights, insightsProjects_filtered and projects_list.
• Removes the v1 slot from the OSI tuples, which moves every later field down one index. Insights PR 1 reads some of those fields at fixed positions, so insights PR 2 must be live first.
• Removes the v1 category-percentage node and the repo_health_score_copy_ds join from project_insights_copy.
• Keeps the 11 signal pipes behind the benchmark callouts, the two dependency pipes and all v2 pipes. Leaves health_score_active_contributors__checker alone. It is a live Tinybird pipe that this repo does not contain.
• Does not change the Contribution Time Distribution widget or its public route. Both read activity_heatmap_by_weekday_and_2hours_blocks.

Deploy notes

Prerequisites

• crowd.dev PR 2 (remove the v1 sink) is merged and deployed. This branch still contains health_score_sink.pipe, which reads health_score_copy_ds. Rebase on main after PR 2 merges, so the sink deletion comes with main.
• insights PR 2 is deployed. It stops the calls to the two signal pipes and the v1 overview route, and it reads every OSI tuple field from the end, so it works with either tuple layout.
• tinybird.pipe_stats_rt shows zero requests to health_score_security and health_score_contributions_outside_work_hours since insights PR 2 deployed. Any request means an unknown caller. Stop and ask.
• This branch is stacked on crowd.dev PR 1's branch. Retarget to main when PR 1 merges, then rebase.
• The sign-off gate above is cleared.

Steps

Run each step on staging first, then on prod. The commands below run on prod. Staging could not run project_insights_copy or the OSI pipes for crowd.dev PR 1, so prod is the only real test for those steps.

Run every command from services/libs/tinybird. tbpp and tbpd add the pipes/ and datasources/ prefix themselves, so pass bare file names.

Readers go before writers, and writers before datasources.

Before you change anything on prod, check for direct SQL readers of what this PR drops. Query API calls show up in tinybird.pipe_stats_rt under pipe_name = 'query_api'. Run this with tb sql or in the Tinybird UI:

SELECT token_name, count() AS requests, max(start_datetime) AS last_seen
FROM tinybird.pipe_stats_rt
WHERE start_datetime > now() - INTERVAL 7 DAY
  AND pipe_name = 'query_api'
  AND match(parameters['q'], '(?i)\\b(health_score_copy_ds|healthScore|contributorHealthScore|popularityHealthScore|developmentHealthScore|securityHealthScore)\\b')
GROUP BY token_name
ORDER BY requests DESC

Ignore rows from your own token. The word boundaries keep healthScoreV2 and repo_health_score_copy_ds out of the match. On 2026-10-13 only the ticket author's own token matched. Any other caller: stop and coordinate before the drops.

0. Push the three OSI pipes. Their tuples must lose the v1 slot before the populated column is dropped.

tbpp collections_oss_index.pipe
tbpp categories_oss_index.pipe
tbpp category_groups_oss_index.pipe

1. Push the readers.

tbpp project_insights.pipe
tbpp project_repo_insights.pipe
tbpp insightsProjects_filtered.pipe
tbpp projects_list.pipe

2. Remove two live pipes that read the v1 healthScore. Neither is in any repo, and nobody owns them. Both must go before the column drops in steps 3 and 4. If staging lacks either one, skip it there.
   • collections_oss_index__checker: an endpoint with 0 requests in the last 30 days.
   • collection_insights_aggregate_copy_pipe: an orphan copy pipe, created 2026-07-29, scheduled daily at 02:50 UTC. Its output is already junk, because test_copy_pipe overwrites it with one test row. Left in place, it fails every night after step 4 and pages insights-critical-alerts.

tb pipe rm collections_oss_index__checker --yes
tb pipe rm collection_insights_aggregate_copy_pipe --yes

3. Outside the 03:00 window, drop the five v1 columns from project_insights_copy_ds. tbpd prints the planned operations and any dependent pipes, then asks you to confirm (y/N). Expect five DROP COLUMN operations and no dependent pipes. Answer N and stop if you see anything else. (tb push --dry-run skips the ALTER planning, so it shows nothing useful here.) If the server rejects a drop, stop and ask. Do not recreate a datasource with about 13 dependents. Then push the copy pipe and run it. Poll until done.

tbpd project_insights_copy_ds.datasource
tbpp project_insights_copy.pipe
tbc project_insights_copy

4. Between two */15 runs, drop healthScore from insights_projects_populated_ds. The copy pipe from step 3 must already be live, because it no longer reads that column. Expect one DROP COLUMN at the tbpd prompt. Poll until done.

tbpd insights_projects_populated_ds.datasource
tbpp insights_projects_populated_copy.pipe
tbc insights_projects_populated_copy

5. Remove the v1 overview pipe, the v1 copy pipe and its datasource.

tb pipe rm health_score_overview --yes
tb pipe rm health_score_copy --yes
tb datasource rm health_score_copy_ds --yes

6. Check tinybird.pipe_stats_rt for health_score_security and health_score_contributions_outside_work_hours. Expect zero requests since insights PR 2 deployed. Any request means stop and ask.

7. Remove the two signal pipes.

tb pipe rm health_score_security --yes
tb pipe rm health_score_contributions_outside_work_hours --yes

Deleting the files does not remove live pipes, datasources or schedules. The commands above do.

After the deploy, prod /api/project responses can still carry the v1 healthScore field for up to a day. Insights caches them in Redis for one day, and only an insights deploy runs FLUSHALL. Insights PR 2 no longer reads that field, so the stale entries do no harm. No manual flush needed: they expire on their own.

Cross-repo order

crowd.dev PR 1 is already live, and insights PR 1 deploys after it.

1. lf-dbt PR 1
2. lf-dbt PR 2
3. Sign-off gate, then insights PR 2
4. lf-dbt PR 3
5. lf-dbt PR 4
6. crowd.dev PR 2
7. crowd-kube connector change (human)
8. Snowflake DROPs (human)
9. crowd.dev PR 3 (this PR)
10. insights PR 3 (transitional v1 compatibility removal, chore: remove transitional health score v1 compatibility insights#2376): deploys after this PR
11. crowd.dev PR 4

What breaks if out of order

• Before insights PR 2: /api/project/[slug]/overview/health-score-overview and /api/project?healthScore=true call deleted pipes and return 500.
• Before insights PR 2: the benchmark route fetches every signal pipe with Promise.all. If either of the two removed pipes is missing, /overview/health-score fails. Project pages lose their benchmark callouts, and the widget area shows "Error fetching benchmarks".
• Before insights PR 2 (tuple reads): removing the v1 slot moves every later field down one index. Insights PR 1 reads the v2 values from the tuple end, so the health pills keep working. Its fixed-position reads after the removed slot break. Collections get the wrong description, slug and status, so project links point at slugs like active. Categories and category groups get the wrong description and status. Insights PR 2 reads every field from the end and works with either layout.
• Before crowd.dev PR 2: health_score_copy_ds disappears while health_score_sink still reads it. Sink exports fail.
• A column drop before its readers are pushed: endpoints that still select the column return errors. The tbpd prompt lists dependent pipes. Answer N if any appear.
• A populated column drop before step 2: collection_insights_aggregate_copy_pipe fails at its next 02:50 UTC run and pages insights-critical-alerts.

Related PRs

• crowd.dev PR 1: add health score v2 to project lists and re-source dependency metrics (live)
• crowd.dev PR 2: remove the v1 health score Snowflake sink (gated)
• crowd.dev PR 4: remove the v1 repository health score pipes (gated)
• insights PR 1: show health score v2 in the open source index
• insights PR 2: remove health score v1 fields and routes (gated)
• insights PR 3: transitional v1 compatibility removal (gated, deploys after this PR, chore: remove transitional health score v1 compatibility insights#2376)
• lf-dbt PR 1: expose the v2 sink leg marker on gold project health metrics
• lf-dbt PR 2: source health score v2 columns from the v2 sink
• lf-dbt PR 3: drop v1 health score columns from gold, platinum and silver dim (gated)
• lf-dbt PR 4: remove the v1 health score sink source and silver leg (gated)
• lfx-self-serve PR (This is a test issue #3465): remove the v1 health band shim
• crowd-kube connector change: removing the sink topic from the Snowflake connector is a human step (crowd-kube connector runbook, gated). No PR exists yet; the human opens one during that runbook. crowd-kube Add links to dashboard sentiment widget #205 is docs only.

JIRA

IN-1222: Deprecate v1 health score sinks and tables
https://linuxfoundation.atlassian.net/browse/IN-1222

[Copilot]

🔵 Needs a closer look

The destructive cross-repository rollout remains gated on consumer sign-off, prerequisite deployments, usage checks, and manual production sequencing.

0 open findings

What changed in this PR

Removes legacy Health Score v1 Tinybird resources and fields while retaining Health Score v2.

Changes:

• Deletes v1 pipes and datasource.
• Removes v1 fields, joins, and outputs.
• Updates OSS index tuple layouts and documentation.

File	Description
pipes/​projects_list.pipe	Removes v1 sorting documentation.
pipes/​project_repo_insights.pipe	Removes repository v1 outputs.
pipes/​project_insights.pipe	Removes project v1 outputs.
pipes/​project_insights_copy.pipe	Removes v1 joins and fields.
pipes/​insightsProjects_filtered.pipe	Removes the v1 list field.
pipes/​insights_projects_populated_copy.pipe	Removes v1 population logic.
pipes/​health_score_security.pipe	Deletes the v1 security signal.
pipes/​health_score_overview.pipe	Deletes the v1 overview endpoint.
pipes/​health_score_copy.pipe	Deletes v1 computation and scheduling.
pipes/​health_score_contributions_outside_work_hours.pipe	Deletes the deprecated signal.
pipes/​collections_oss_index.pipe	Removes v1 tuple entries.
pipes/​category_groups_oss_index.pipe	Removes v1 tuple entries.
pipes/​categories_oss_index.pipe	Removes v1 tuple entries.
datasources/​project_insights_copy_ds.datasource	Drops five v1 columns.
datasources/​insights_projects_populated_ds.datasource	Drops the v1 score column.
datasources/​health_score_copy_ds.datasource	Deletes the v1 datasource definition.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

The remaining scheduled health_score_sink still reads the datasource deleted by this change, so the prerequisite sink deletion must be incorporated first.

0 open findings

🧠 Review effort: Balanced

[gaspergrom]

Closing for now: org-dashboard still reads the v1 health score (/api/project?healthScore=true) and will keep running until it is retired, expected early November 2026. Reopen and rebase this once org-dashboard is shut down. The branch is kept.