Skip to content

Use the differences in source code metrics in the regressor model - #7596

Merged
marco-c merged 1 commit into
mozilla:masterfrom
marco-c:regressor-metrics-diff
Nov 19, 2026
Merged

marco-c merged 1 commit into
mozilla:masterfrom
marco-c:regressor-metrics-diff

Conversation

@marco-c

@marco-c marco-c commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Now that metrics_diff is computed against the files before each commit (#7563, commits DB v25), the SourceCodeMetricsDiff features carry real signal. Before the fix they were meaningless, so the earlier result that they didn't help no longer holds.

Evaluation: production RegressorModel pipeline (bug weighting, message TF-IDF, isotonic calibration), with and without the feature. Training on the 2 years before each cutoff date, testing on the following 3 months, scoring each bug by its highest commit risk, 3 seeds each:

Test window Bug AUC (before → after) Bug AP (before → after)
Dec 2025 – Mar 2026 0.761 → 0.763 0.304 → 0.311
Mar – Jun 2026 0.750 → 0.752 0.291 → 0.296
Jun – Sep 2026 0.758 → 0.760 0.281 → 0.285

The gain is small (about seed-to-seed noise for AUC on a single window) but it goes the same way in all three windows. Commit-level AUC is unchanged on average (−0.006 / +0.005 / −0.002, within seed variation).

No serving changes are needed: the commit classifier gets new commits through repository.download_commits, which computes metrics_diff too.

Now that the metrics differences are computed from the files before the
commits (mozilla#7563), they slightly but consistently improve the model.
@marco-c
marco-c marked this pull request as ready for review November 19, 2026 22:00
@marco-c
marco-c requested a review from a team as a code owner November 19, 2026 22:00
@marco-c
marco-c merged commit 808d229 into mozilla:master Nov 19, 2026
13 checks passed
@marco-c
marco-c deleted the regressor-metrics-diff branch November 19, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant