feat: hide affiliation-blocked organizations from org pages and dashboards (IN-1399)

[gaspergrom]

Hides affiliation-blocked organizations from org pages, org dashboards and leaderboard links.

• Adds isAffiliationBlocked (UInt8, default 0) to organizations and organizations_populated_slug; the populate pipe carries it through.
• The 6 org page pipes and organizations_filtered (search) exclude blocked orgs, so a blocked org page returns no rows and Insights shows a 404.
• The 5 id-keyed dashboard pipes (org_dash_metrics, org_dash_activities_count, org_dash_avg_merge_time, org_dash_contributors, org_dash_maintainers) exclude blocked orgs with organizationId NOT IN (SELECT id FROM organizations FINAL WHERE isAffiliationBlocked = 1). A blocked org returns no rows from org_dash_metrics and zero/null org columns from the other four, and the unfiltered org totals in org_dash_metrics no longer include blocked orgs (26 rows across 10 orgs).
• The leaderboard pipes keep blocked orgs listed but return slug '', so no link is rendered.

Deploy and rollout (already applied to prod Tinybird from this branch, before merge):

1. ALTER ... ADD COLUMN isAffiliationBlocked UInt8 DEFAULT 0 on both datasources, then the organizations_populate_slug copy, then the consumer pipes, then the five dashboard pipes.
2. Existing rows read the new column as 0, so Sequin had to re-emit the blocked rows: updatedAt was bumped in Postgres for isAffiliationBlocked = true rows only. Postgres and Tinybird now agree on the blocked set (27,011, identical id set). This is an in-place alter plus a targeted re-emit, not the delete-and-recreate procedure in the README; no other row needed a backfill because the flag defaults to false.
3. leaderboards_organizations was not re-run manually (it appends a daily snapshot); the 02:10 UTC copy picks up the blank slugs.

Known follow-ups: each dashboard call now scans organizations FINAL (about 1.5M rows, roughly +0.1s); 24,204 blocked orgs have an org page today (760 of them with at least 1,000 contributions; most are individuals or personal sites). Blocked foundation-like records include The Linux Foundation (the-linux-foundation, --2), The Apache Software Foundation (the-apache-software-foundation), etcd, TiKV Project, Node.js Foundation, OpenDaylight Project and the CNCF record cloud-native-computing-foundation-cncf; same-name records with other slugs keep their pages. They lose their org pages, so please confirm the current flag set is what should be hidden.

JIRA: https://linuxfoundation.atlassian.net/browse/IN-1399

[cursor]

PR Summary

Medium Risk
Changes which organizations appear in Insights dashboards and leaderboards and adds per-request scans of organizations FINAL (~1.5M rows); blocked foundations may lose public org pages, which needs product confirmation.

Overview
Introduces isAffiliationBlocked on Tinybird organizations / organizations_populated_slug (default 0) and threads it through organizations_populate_slug, so blocked orgs can be treated differently in Insights.

Org pages and search no longer resolve blocked orgs: slug lookups in the six org-page pipes require isAffiliationBlocked = 0, and organizations_filtered drops them from search—blocked slugs should 404 like missing Insights pages.

Org dashboards exclude blocked IDs from org-scoped series and from org_dash_metrics rollups via organizationId NOT IN (SELECT id FROM organizations FINAL WHERE isAffiliationBlocked = 1) on the five comparison pipes plus metrics filtering.

Leaderboards still rank blocked orgs but blank the slug (leaderboards_organizations, organizations_leaderboard) so UI can show the name without linking to an org page.

^{Reviewed by Cursor Bugbot for commit c6a9c67. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Copilot review overview

🟡 Changes recommended

Existing replicated organization rows require an explicit datasource recreation and backfill before the new filters behave correctly.

Review effort: Balanced
Findings: 1

Open (1)

• Require datasource recreation and backfill before enabling block filters · New

What changed in this PR

Adds affiliation-blocking awareness to Tinybird organization pages, search, and leaderboards.

Changes:

• Propagates isAffiliationBlocked through organization datasources.
• Excludes blocked organizations from search and organization-page endpoints.
• Preserves leaderboard entries while suppressing their links.

File	Description
datasources/​organizations.datasource	Adds the replicated block flag.
datasources/​organizations_populated_slug.datasource	Stores the propagated flag.
pipes/​organizations_populate_slug.pipe	Carries the flag into populated organizations.
pipes/​organizations_filtered.pipe	Excludes blocked organizations from search.
pipes/​organizations_leaderboard.pipe	Clears blocked organization slugs.
pipes/​leaderboards_organizations.pipe	Clears blocked organization links in snapshots.
pipes/​org_page_profile.pipe	Blocks profile lookup.
pipes/​org_page_projects.pipe	Blocks project lookup.
pipes/​org_page_kpis.pipe	Blocks KPI lookup.
pipes/​org_page_contributors.pipe	Blocks contributor lookup.
pipes/​org_page_contributors_timeseries.pipe	Blocks contributor timeseries lookup.
pipes/​org_page_activities_timeseries.pipe	Blocks activity timeseries lookup.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

The Datafiles check fails because several new dashboard predicates do not match Tinybird formatting.

Review effort: Balanced
Findings: 1 · 5

Open (6)

• Require datasource recreation and backfill before enabling block filters
• Split predicate to match tb fmt output · New
• Break overlong predicate for tb fmt compliance · New
• Break overlong predicate for tb fmt compliance · New
• Break overlong predicate for tb fmt compliance · New
• Split predicate to satisfy formatter line width · New

[themarolt]

lgtm

[Copilot]

Copilot review overview

🔵 Needs a closer look

The five dashboard pipes introduce repeated request-time full scans of the 1.5M-row organizations datasource.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Require datasource recreation and backfill before enabling block filters
• Split predicate to satisfy formatter line width

Resolved since last review (4)

• Break overlong predicate for tb fmt compliance
• Break overlong predicate for tb fmt compliance
• Break overlong predicate for tb fmt compliance
• Split predicate to match tb fmt output

Previously missed (5)

In code that hasn't changed since last review

Reuse blocked-ID lookup instead of repeated organizations scans

services/​libs/​tinybird/​pipes/​org_dash_activities_count.pipe:23

This adds another request-time FINAL scan of all organizations on a non-key flag. Because the dashboard calls several such pipes, use one compact blocked-ID lookup (or propagate the flag into the precomputed data) instead of scanning and deduplicating ~1.5M rows per endpoint.

Avoid full organizations scan on merge-time requests

services/​libs/​tinybird/​pipes/​org_dash_avg_merge_time.pipe:24

This causes each merge-time request to evaluate an unindexed organizations FINAL scan over the full organization datasource. Please query a compact precomputed blocked-ID lookup (or enrich the dashboard data with the flag) to avoid repeating this scan in every dashboard pipe.

Avoid repeated organizations FINAL scan on contributors requests

services/​libs/​tinybird/​pipes/​org_dash_contributors.pipe:23

This adds a full organizations FINAL scan on every contributors request, filtering on a column that is not in the sorting key. Please reuse a compact precomputed blocked-ID set (or carry the flag into the precomputed data) rather than repeating this ~1.5M-row scan across dashboard endpoints.

Avoid full organizations scan on maintainers requests

services/​libs/​tinybird/​pipes/​org_dash_maintainers.pipe:39

This condition makes every maintainers request run organizations FINAL and scan the entire organization datasource by a non-key flag. Use a shared compact blocked-ID lookup (or propagate the flag into the precomputed dashboard data) so each dashboard endpoint does not repeatedly deduplicate ~1.5M rows.

Avoid unindexed organizations FINAL scan on metrics requests

services/​libs/​tinybird/​pipes/​org_dash_metrics.pipe:21

This performs an unindexed FINAL scan of the ~1.5M-row organizations datasource on every metrics request; the datasource is sorted by id, not isAffiliationBlocked. Since the dashboard invokes several pipes with the same scan, this adds repeated full-table deduplication to a latency-sensitive path. Please filter against a compact precomputed blocked-ID lookup (or carry the flag into the dashboard copy datasource) instead.