[joanagmaia]

Summary

The Development tab's Pull requests widget shows 0 merged PRs for repositories that merge through a bot. istio/istio reported 1,305 opened and 0 merged over the past year (linuxfoundation/insights#2369). istio merges every PR with its Prow bot, istio-testing. A merge activity belongs to the account that clicked merge, and the activity buckets drop every bot member, so all 2,408 istio merges disappear before the widget counts them. The Development docs say these metrics include bots.

This PR adds count pipes that read the PR and issue lifecycle tables, which already include bots, and builds issues_analyzed with bot-opened and bot-closed issues. The insights PR that switches the widgets and the public API to these pipes depends on this one.

Changes

• pull_requests_count.pipe (new): counts PRs, MRs and Gerrit changesets by state (opened, merged or closed) from pull_requests_filtered, as a total or a time series. Each PR is counted once, and a PR a bot merged counts as merged. This is the same table the widget's average velocity already reads.
• issues_count.pipe (new): counts issues by state (opened or closed) from issues_analyzed, as a total or a time series.
• issue_analysis_copy_pipe.pipe: opened and closed events now come from the unfiltered activityRelations_enriched_deduplicated_bucket_union, so an issue a stale bot closes counts as closed. Those buckets refresh once a day, so the pipe also adds the hourly events in activityRelations_enrich_snapshot_MV_ds that the daily merge has not taken in yet, deduplicated the same way as the bucket mergers. Issues stay as fresh as before (hourly). Issues stay limited to enabled repositories, with the same repository filter as the cleaned buckets. The first non-author response still comes from the bot-filtered buckets, so a bot comment does not count as a response.
• issues_analyzed also feeds health scores (v1 and v2 issue resolution, lifecycle, maintainer), the issue-response leaderboard and the agentic AI copies, so those start counting bot-opened and bot-closed issues too. This matches pull_requests_analyzed, which already includes bots for PR merge health. The insights PR updates the health score docs to match.

istio/istio over the past year, checked against production:

	Bot-filtered (today)	With bots (this PR)
PRs opened	1,305	3,152
PRs merged	0	2,016
PRs closed	not checked	636
Issues opened	700	924
Issues closed	185	972

pull_requests_analyzed also had 2.07M duplicate rows, because bucket 5 of the append-mode pull_request_analysis_initial_snapshot had been loaded twice. That bucket was rebuilt in production on 2026-10-19, outside this PR, and now has 145 duplicate rows left, the same low level as the other buckets.

Deploy order: push these pipes to lfx_insights_stg, then to lfx_insights after merge, and do both before the insights PR deploys. Both new pipes declare TOKEN "insights-app-token" READ, so the insights app can read them once they are pushed.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

Related issue

Refs linuxfoundation/lfx-cdp-insights-ops#790

[cursor]

PR Summary

Medium Risk
Changes core issues_analyzed copy logic and downstream health/leaderboard consumers; new pipes are additive but metrics will shift materially for bot-heavy repos.

Overview
Fixes Development metrics that dropped bot-driven lifecycle events by sourcing issue open/close from unfiltered activity buckets (plus the realtime MV for hourly freshness) while keeping repo scoping and bot-filtered data for first human response.

Adds issues_count.pipe and pull_requests_count.pipe to return opened/closed (and merged for PRs) totals or time series from issues_analyzed and pull_requests_filtered, with project/collection, repo, platform, and date filters—aligned with docs that bot merges and stale-bot closes should count.

^{Reviewed by Cursor Bugbot for commit 9029806. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1c194d9. Configure here.}

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new blocking issues were found.

Summary

The PR adds bot-inclusive PR and issue counts. Issue analysis combines daily buckets with recent events while keeping bot comments out of first-response times.

• Issue analysis now includes bot-opened and bot-closed issues.
• Pull request counts cover opened, merged, and closed PRs.
• Issue counts cover opened and closed issues.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Buckets["Daily unfiltered buckets"] --> Opened["Opened events: prefer MV rows"]
  MV["Recent unfiltered MV"] --> Opened
  Buckets --> Closed["Earliest close per issue"]
  MV --> Closed
  Cleaned["Bot-filtered comments"] --> Response["First non-author response"]
  Opened --> Analysis["issues_analyzed"]
  Closed --> Analysis
  Response --> Analysis
  Analysis --> Issues["issues_count"]
  PRs["pull_requests_filtered"] --> Counts["pull_requests_count"]

Loading

_{Reviews (3) · Last reviewed commit: "fix: formatting" · Reviewed by Greptile}

[Copilot]

🟡 Changes recommended

Duplicate counting, incomplete default time-series bounds, and reduced issue freshness undermine metric accuracy.

3 open issues

• Use hourly source to preserve issue lifecycle freshness · New
• Deduplicate pull request rows before counting · New
• Derive series bounds from lifecycle event timestamps · New

What changed in this PR

This PR adds bot-inclusive lifecycle counts for Development widgets and updates issue analysis used by downstream health metrics.

Changes:

• Adds PR and issue count pipes supporting totals and time series.
• Includes bot-opened and bot-closed issues while keeping response metrics bot-filtered.

File	Description
services/​libs/​tinybird/​pipes/​pull_requests_count.pipe	Adds opened, merged, and closed PR counts.
services/​libs/​tinybird/​pipes/​issues_count.pipe	Adds opened and closed issue counts.
services/​libs/​tinybird/​pipes/​issue_analysis_copy_pipe.pipe	Uses unfiltered lifecycle events while preserving repository restrictions.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The snapshot/delta merge changes shared issue lifecycle data used by health metrics and needs staging SQL validation and human review.

No open issues 🎉

3 resolved since last review

• Derive series bounds from lifecycle event timestamps
• Deduplicate pull request rows before counting
• Use hourly source to preserve issue lifecycle freshness

_{Review effort: Balanced}

[gaspergrom]

LGTM. I compared the delta rule with the bucket mergers and the repo filter with the cleaned bucket copy, and both match. One thing to keep in mind: the cleaned buckets also join members_sorted, which drops team members and organization members, so issues from those accounts now count in issues_analyzed alongside bots.

[Copilot]

🔵 Needs a closer look

Inconsistent snapshot and delta deduplication keys can duplicate issues and inflate downstream metrics.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Use composite event key to prevent duplicate issue rows

services/​libs/​tinybird/​pipes/​issue_analysis_copy_pipe.pipe:43

Match bucket rows against the delta using (segmentId, timestamp, type, platform, channel, sourceId), as both issue_events_delta and activityRelations_snapshot_merger_copy_0.pipe:29–30 do. If the same source event appears under different activity IDs in the bucket and the delta, this predicate retains the bucket row and appends the delta row. Both then become issues_analyzed rows, inflating issue counts and downstream metrics. Use the same composite key for replacement and deduplication.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Unresolved snapshot/delta deduplication and shared downstream health metrics require correction and human validation in staging.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Match overlapping rows by deduplication key instead of activityId

services/​libs/​tinybird/​pipes/​issue_analysis_copy_pipe.pipe:42

Match overlapping rows by the source deduplication key, not activityId. If re-ingestion gives the same issue-opening event a new activity ID, this predicate retains the old bucket row while UNION ALL adds its replacement. Both reach issues_analyzed, inflating issue counts and downstream metrics until the daily merge. Use (segmentId, timestamp, type, platform, channel, sourceId), matching issue_events_delta at line 31 and activityRelations_snapshot_merger_copy_0.pipe:29–30.

_{Review effort: Balanced}