Add performance report generator pipeline

[siddardh-ra]

Performance Report Generator Pipeline

Adds an automated two-stage pipeline for generating KB-style performance comparison articles for OpenShift Pipelines releases.

What it does

• Stage 1 (generate_comparison.py): Connects to the Horreum PostgreSQL database (read-only), fetches the last N runs per version per deployment variant, applies statistical normalization, and outputs a structured JSON with comparison metrics.
• Stage 2 (generate_article.py): Takes the JSON output and a prompt template, sends it to an LLM (OpenAI API), and produces a publishable markdown KB article.

Supported modes

• Version comparison (e.g., 1.22 vs 1.23) — side-by-side analysis across all components and configurations
• Nightly comparison (e.g., 1.23 vs nightly) — track nightly build performance against a released baseline
• Single-version benchmark — performance profile for one version without comparison

Statistical approach — MAD-based outlier detection

Performance CI runs 3-4 times per version. Infrastructure flakiness (noisy neighbors, node pressure) can produce one bad run that skews averages. Standard approaches (Z-score, IQR) are unreliable at this sample size.

We use Median Absolute Deviation (MAD) — a robust statistical method that uses the median instead of the mean, so a single outlier cannot shift the reference point. If a run has >50% of its metrics deviating beyond 2.5x MAD, the entire run is excluded as an infrastructure anomaly. A safety guardrail ensures at least 2 runs always survive.

Data Correctness and Validation

Stage 1 saves intermediate artifacts alongside the final output:

• raw_data_*.json — unprocessed DB rows for cross-verification
• mad_analysis_*.json — outlier detection decisions (which runs survived, which were excluded and why)

This allows teams to trace any number in the final report back to the source data.

Security

• Database connection enforces read-only mode at the PostgreSQL session level
• All DB connection details (host, user, DB name, password) are configured via HORREUM_DB_* environment variables — no infrastructure details in the codebase

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: siddardh-ra

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [siddardh-ra]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[siddardh-ra]

Since this PR touches around 11 files, here's a quick summary to make the review easier:

reports/config.yaml — Defines all metric keys, Horreum test IDs, component/variant mappings, and statistical thresholds (MAD, significance levels).

reports/generate_comparison.py — Stage 1 CLI entry point. Connects to Horreum DB (read-only), fetches runs, applies MAD outlier detection, and outputs structured JSON. Supports comparison and benchmark modes.

reports/generate_article.py — Stage 2 script. Takes the JSON from Stage 1 + a prompt template, sends it to OpenAI API, and writes the final KB article as markdown.

reports/prompt_template.md — LLM prompt template for version comparison articles. Includes tone/framing rules to ensure customer-friendly, professional language.

reports/prompt_template_benchmark.md — LLM prompt template for single-version benchmark articles. Same tone guidelines, focused on characterizing performance rather than comparing.

reports/lib/db.py — PostgreSQL data fetcher. Enforces read-only mode at session level, builds queries using Horreum test IDs and version filters from config.

reports/lib/stats.py — MAD-based outlier detection and statistical computation. Handles both comparison (pct_change, verdict classification) and benchmark (mean/min/max) modes.

reports/lib/formatter.py — Structures raw statistical output into the final JSON format with metadata, component hierarchy, summaries, and data quality info.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[Copilot]

Pull request overview

Adds a new reports/ pipeline that pulls performance data from a read-only Horreum PostgreSQL database, applies MAD-based outlier filtering plus summary statistics, then optionally generates Red Hat KB-style markdown articles via the OpenAI API.

Changes:

• Introduces Stage 1 data generation (generate_comparison.py) with DB fetch, outlier exclusion, and structured JSON output (comparison + benchmark modes).
• Introduces Stage 2 article generation (generate_article.py) driven by markdown prompt templates for comparison and benchmark modes.
• Adds a large, centralized config.yaml for metric/test/variant definitions and thresholds, plus documentation and gitignore entries.

Reviewed changes

Copilot reviewed 9 out of 11 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
reports/README.md	Usage + operational documentation for the two-stage generator.
reports/generate_comparison.py	CLI entrypoint for DB→JSON generation and artifact writing.
reports/generate_article.py	CLI entrypoint for JSON→markdown KB article generation via OpenAI.
reports/lib/db.py	Horreum PostgreSQL query construction + run fetching (read-only).
reports/lib/stats.py	MAD outlier detection and benchmark/comparison statistics.
reports/lib/formatter.py	Shapes computed stats into LLM-friendly JSON, including top-N summaries.
reports/prompt_template.md	Comparison-mode KB article prompt template.
reports/prompt_template_benchmark.md	Benchmark-mode KB article prompt template.
reports/config.yaml	Metric, variant, test ID, and threshold configuration for the pipeline.
reports/lib/init.py	Marks reports/lib as a package.
.gitignore	Ignores generated report outputs and local artifacts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jhutar]

I suggest we take this out to not depend on these big (?) libraries. We can just generate the prompt and then user can feed it into AI agent of their choice. We do not need this to be fully automated. Also is OpenAI approved thingy in RedHat?

[siddardh-ra]

I had similar reservations while implementing this. My initial idea was to expose a report-generation API that managers or engineers could trigger to automatically generate the output, but that's more of a future enhancement. For now, I'm happy to keep this focused on prompt generation and let users use their preferred AI tool. If automation is needed later, we can revisit it using a Red Hat–approved LLM API.

[siddardh-ra]

Updated the same on the latest commit

[jhutar]

What is the difference between this file reports/prompt_template.md and reports/prompt_template_benchmark.md? Do we need both?

[siddardh-ra]

The templates have some shared formatting and tone, but they serve different purposes.

• prompt_template.md is for version comparisons ({{VERSION_A}} vs {{VERSION_B}}), focusing on regressions, improvements, trade-offs, and percentage changes.
• prompt_template_benchmark.md is for single-version benchmarking ({{VERSION}}), focusing on absolute performance, configuration comparisons, and deployment guidance.

So yes, I think both are needed. Comparison mode and benchmark mode produce different data, and each template is tailored to its respective output.

[jhutar]

I would rename this and remove all mentions of "Horreum" here as you are only talking to our PostgreSQL that contains mirror of Horreum data sets, not to actual Horreum PostgreSQL.

[siddardh-ra]

Sure. And I've updated the corrected var name in the latest commit