Skip to content

ci: track test flavor in per-test baseline and rename job to Track Test Duration - #4659

Draft
hsuan-lun-chiang wants to merge 9 commits into
mainfrom
track-test-duration-flavor
Draft

ci: track test flavor in per-test baseline and rename job to Track Test Duration#4659
hsuan-lun-chiang wants to merge 9 commits into
mainfrom
track-test-duration-flavor

Conversation

@hsuan-lun-chiang

@hsuan-lun-chiang hsuan-lun-chiang commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR addresses hardware-dependent test duration noise in the Per-Test Baseline comparison and updates workflow job naming.

Problem & Context

Previously, per_test_baseline.json tracked test durations using un-flavored test names. When a test ran on multiple hardware platforms with different execution times (e.g., QKClipMLATest.test_mla_dot_product_integration taking 0.94s on CPU vs 15.99s on GPU), baseline comparison falsely flagged regressions when comparing across hardware types.

Solution

  1. Track Hardware Test Flavor: Baseline keys in per_test_baseline.json are now formatted as flavor::test_name (e.g., gpu-unit::tests.unit.qk_clip_test...), isolating durations for each hardware flavor.
  2. Legacy Key Clean-up: Automatically purges un-flavored legacy baseline keys when initializing new_per_test_baseline.json.
  3. Rename Job & Workflow: Updated job name from Track Test Performance to Track Test Duration in track_performance.yml and ci_pipeline.yml.
  4. Unit Tests: Added tests/utils/process_test_results_test.py to verify per-flavor baseline isolation and regression checking.

Tests

Ran local unit tests and pre-commit hooks:

python3 tests/utils/process_test_results_test.py
pre-commit run --files tests/utils/process_test_results.py tests/utils/process_test_results_test.py .github/workflows/track_performance.yml .github/workflows/ci_pipeline.yml

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

…st Duration

Track hardware (test flavor) for each test duration in per_test_baseline.json
using key format `flavor::test_name` to prevent cross-hardware baseline comparison noise.
Also rename job and workflow from "Track Test Performance" to "Track Test Duration".
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant