Repository navigation
[APMLP-598] Pin one-pipeline system tests on release branches - #10966
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits intoMar 26, 2026
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits into
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits into
Conversation
5 tasks
sarahchen6
added
tag: no release notes
sarahchen6
marked this pull request as ready for review
March 25, 2026 19:25
sarahchen6
requested review from
PerfectSlayer,
amarziali,
pawelchcki and
randomanderson
and removed request for
a team
March 25, 2026 19:25
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 62 metrics, 9 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~d28e883862, baseline=1.61.0-SNAPSHOT~517a89d1ec3
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.06 s) : 0, 1059905
Total [baseline] (11.039 s) : 0, 11039000
Agent [candidate] (1.06 s) : 0, 1059534
Total [candidate] (11.036 s) : 0, 11036290
section appsec
Agent [baseline] (1.246 s) : 0, 1246167
Total [baseline] (11.214 s) : 0, 11213564
Agent [candidate] (1.244 s) : 0, 1243849
Total [candidate] (11.187 s) : 0, 11186502
section iast
Agent [baseline] (1.227 s) : 0, 1226600
Total [baseline] (11.319 s) : 0, 11319174
Agent [candidate] (1.233 s) : 0, 1233004
Total [candidate] (11.323 s) : 0, 11323101
section profiling
Agent [baseline] (1.182 s) : 0, 1181887
Total [baseline] (10.999 s) : 0, 10998693
Agent [candidate] (1.189 s) : 0, 1189196
Total [candidate] (11.069 s) : 0, 11068519
gantt
title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~d28e883862, baseline=1.61.0-SNAPSHOT~517a89d1ec3
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.195 ms) : 0, 1195
crashtracking [candidate] (1.213 ms) : 0, 1213
BytebuddyAgent [baseline] (629.351 ms) : 0, 629351
BytebuddyAgent [candidate] (629.854 ms) : 0, 629854
AgentMeter [baseline] (29.439 ms) : 0, 29439
AgentMeter [candidate] (29.365 ms) : 0, 29365
GlobalTracer [baseline] (256.675 ms) : 0, 256675
GlobalTracer [candidate] (256.625 ms) : 0, 256625
AppSec [baseline] (31.769 ms) : 0, 31769
AppSec [candidate] (31.582 ms) : 0, 31582
Debugger [baseline] (60.4 ms) : 0, 60400
Debugger [candidate] (60.448 ms) : 0, 60448
Remote Config [baseline] (591.649 µs) : 0, 592
Remote Config [candidate] (597.797 µs) : 0, 598
Telemetry [baseline] (8.754 ms) : 0, 8754
Telemetry [candidate] (8.823 ms) : 0, 8823
Flare Poller [baseline] (5.706 ms) : 0, 5706
Flare Poller [candidate] (4.993 ms) : 0, 4993
section appsec
crashtracking [baseline] (1.184 ms) : 0, 1184
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (658.189 ms) : 0, 658189
BytebuddyAgent [candidate] (657.39 ms) : 0, 657390
AgentMeter [baseline] (12.191 ms) : 0, 12191
AgentMeter [candidate] (12.083 ms) : 0, 12083
GlobalTracer [baseline] (257.537 ms) : 0, 257537
GlobalTracer [candidate] (257.24 ms) : 0, 257240
IAST [baseline] (24.153 ms) : 0, 24153
IAST [candidate] (24.099 ms) : 0, 24099
AppSec [baseline] (177.936 ms) : 0, 177936
AppSec [candidate] (177.099 ms) : 0, 177099
Debugger [baseline] (66.387 ms) : 0, 66387
Debugger [candidate] (66.046 ms) : 0, 66046
Remote Config [baseline] (621.463 µs) : 0, 621
Remote Config [candidate] (621.637 µs) : 0, 622
Telemetry [baseline] (8.265 ms) : 0, 8265
Telemetry [candidate] (8.378 ms) : 0, 8378
Flare Poller [baseline] (3.514 ms) : 0, 3514
Flare Poller [candidate] (3.571 ms) : 0, 3571
section iast
crashtracking [baseline] (1.187 ms) : 0, 1187
crashtracking [candidate] (1.195 ms) : 0, 1195
BytebuddyAgent [baseline] (795.285 ms) : 0, 795285
BytebuddyAgent [candidate] (799.904 ms) : 0, 799904
AgentMeter [baseline] (11.412 ms) : 0, 11412
AgentMeter [candidate] (11.5 ms) : 0, 11500
GlobalTracer [baseline] (247.159 ms) : 0, 247159
GlobalTracer [candidate] (248.38 ms) : 0, 248380
IAST [baseline] (25.285 ms) : 0, 25285
IAST [candidate] (25.541 ms) : 0, 25541
AppSec [baseline] (26.479 ms) : 0, 26479
AppSec [candidate] (26.726 ms) : 0, 26726
Debugger [baseline] (70.601 ms) : 0, 70601
Debugger [candidate] (69.98 ms) : 0, 69980
Remote Config [baseline] (537.668 µs) : 0, 538
Remote Config [candidate] (527.934 µs) : 0, 528
Telemetry [baseline] (9.247 ms) : 0, 9247
Telemetry [candidate] (9.726 ms) : 0, 9726
Flare Poller [baseline] (3.451 ms) : 0, 3451
Flare Poller [candidate] (3.481 ms) : 0, 3481
section profiling
crashtracking [baseline] (1.163 ms) : 0, 1163
crashtracking [candidate] (1.177 ms) : 0, 1177
BytebuddyAgent [baseline] (683.009 ms) : 0, 683009
BytebuddyAgent [candidate] (685.275 ms) : 0, 685275
AgentMeter [baseline] (9.007 ms) : 0, 9007
AgentMeter [candidate] (9.068 ms) : 0, 9068
GlobalTracer [baseline] (214.947 ms) : 0, 214947
GlobalTracer [candidate] (216.312 ms) : 0, 216312
AppSec [baseline] (32.149 ms) : 0, 32149
AppSec [candidate] (32.571 ms) : 0, 32571
Debugger [baseline] (65.208 ms) : 0, 65208
Debugger [candidate] (65.882 ms) : 0, 65882
Remote Config [baseline] (584.583 µs) : 0, 585
Remote Config [candidate] (569.726 µs) : 0, 570
Telemetry [baseline] (8.445 ms) : 0, 8445
Telemetry [candidate] (7.823 ms) : 0, 7823
Flare Poller [baseline] (3.444 ms) : 0, 3444
Flare Poller [candidate] (4.303 ms) : 0, 4303
ProfilingAgent [baseline] (93.117 ms) : 0, 93117
ProfilingAgent [candidate] (95.232 ms) : 0, 95232
Profiling [baseline] (93.683 ms) : 0, 93683
Profiling [candidate] (95.805 ms) : 0, 95805
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~d28e883862, baseline=1.61.0-SNAPSHOT~517a89d1ec3
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1063852
Total [baseline] (8.852 s) : 0, 8852028
Agent [candidate] (1.063 s) : 0, 1063376
Total [candidate] (8.822 s) : 0, 8821992
section iast
Agent [baseline] (1.232 s) : 0, 1232036
Total [baseline] (9.538 s) : 0, 9538298
Agent [candidate] (1.228 s) : 0, 1228191
Total [candidate] (9.567 s) : 0, 9567390
gantt
title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~d28e883862, baseline=1.61.0-SNAPSHOT~517a89d1ec3
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.201 ms) : 0, 1201
crashtracking [candidate] (1.199 ms) : 0, 1199
BytebuddyAgent [baseline] (633.799 ms) : 0, 633799
BytebuddyAgent [candidate] (633.613 ms) : 0, 633613
AgentMeter [baseline] (29.731 ms) : 0, 29731
AgentMeter [candidate] (29.7 ms) : 0, 29700
GlobalTracer [baseline] (258.451 ms) : 0, 258451
GlobalTracer [candidate] (258.397 ms) : 0, 258397
AppSec [baseline] (32.075 ms) : 0, 32075
AppSec [candidate] (32.087 ms) : 0, 32087
Debugger [baseline] (60.174 ms) : 0, 60174
Debugger [candidate] (60.001 ms) : 0, 60001
Remote Config [baseline] (597.812 µs) : 0, 598
Remote Config [candidate] (581.971 µs) : 0, 582
Telemetry [baseline] (8.038 ms) : 0, 8038
Telemetry [candidate] (8.021 ms) : 0, 8021
Flare Poller [baseline] (3.553 ms) : 0, 3553
Flare Poller [candidate] (3.528 ms) : 0, 3528
section iast
crashtracking [baseline] (1.198 ms) : 0, 1198
crashtracking [candidate] (1.215 ms) : 0, 1215
BytebuddyAgent [baseline] (801.587 ms) : 0, 801587
BytebuddyAgent [candidate] (797.183 ms) : 0, 797183
AgentMeter [baseline] (11.652 ms) : 0, 11652
AgentMeter [candidate] (11.39 ms) : 0, 11390
GlobalTracer [baseline] (247.262 ms) : 0, 247262
GlobalTracer [candidate] (247.643 ms) : 0, 247643
AppSec [baseline] (26.416 ms) : 0, 26416
AppSec [candidate] (26.536 ms) : 0, 26536
Debugger [baseline] (69.426 ms) : 0, 69426
Debugger [candidate] (67.057 ms) : 0, 67057
Remote Config [baseline] (521.031 µs) : 0, 521
Remote Config [candidate] (524.871 µs) : 0, 525
Telemetry [baseline] (9.068 ms) : 0, 9068
Telemetry [candidate] (11.221 ms) : 0, 11221
Flare Poller [baseline] (3.456 ms) : 0, 3456
Flare Poller [candidate] (3.968 ms) : 0, 3968
IAST [baseline] (25.334 ms) : 0, 25334
IAST [candidate] (25.388 ms) : 0, 25388
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 2 performance regressions! Performance is the same for 17 metrics, 16 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~d28e883862, baseline=1.61.0-SNAPSHOT~517a89d1ec3
dateFormat X
axisFormat %s
section baseline
no_agent (1.189 ms) : 1178, 1201
. : milestone, 1189,
iast (3.024 ms) : 2982, 3066
. : milestone, 3024,
iast_FULL (6.011 ms) : 5950, 6072
. : milestone, 6011,
iast_GLOBAL (3.433 ms) : 3379, 3487
. : milestone, 3433,
profiling (2.18 ms) : 2160, 2200
. : milestone, 2180,
tracing (1.768 ms) : 1753, 1783
. : milestone, 1768,
section candidate
no_agent (1.2 ms) : 1188, 1211
. : milestone, 1200,
iast (3.214 ms) : 3165, 3263
. : milestone, 3214,
iast_FULL (6.059 ms) : 5996, 6122
. : milestone, 6059,
iast_GLOBAL (3.446 ms) : 3397, 3495
. : milestone, 3446,
profiling (2.126 ms) : 2107, 2145
. : milestone, 2126,
tracing (1.751 ms) : 1738, 1765
. : milestone, 1751,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~d28e883862, baseline=1.61.0-SNAPSHOT~517a89d1ec3
dateFormat X
axisFormat %s
section baseline
no_agent (19.019 ms) : 18821, 19217
. : milestone, 19019,
appsec (18.507 ms) : 18316, 18699
. : milestone, 18507,
code_origins (17.893 ms) : 17715, 18071
. : milestone, 17893,
iast (18.331 ms) : 18148, 18515
. : milestone, 18331,
profiling (18.617 ms) : 18431, 18804
. : milestone, 18617,
tracing (17.92 ms) : 17741, 18099
. : milestone, 17920,
section candidate
no_agent (17.21 ms) : 17035, 17385
. : milestone, 17210,
appsec (18.469 ms) : 18278, 18660
. : milestone, 18469,
code_origins (17.925 ms) : 17746, 18104
. : milestone, 17925,
iast (17.842 ms) : 17664, 18020
. : milestone, 17842,
profiling (18.655 ms) : 18472, 18838
. : milestone, 18655,
tracing (17.716 ms) : 17539, 17893
. : milestone, 17716,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~d28e883862, baseline=1.61.0-SNAPSHOT~517a89d1ec3
dateFormat X
axisFormat %s
section baseline
no_agent (15.511 s) : 15511000, 15511000
. : milestone, 15511000,
appsec (14.52 s) : 14520000, 14520000
. : milestone, 14520000,
iast (18.581 s) : 18581000, 18581000
. : milestone, 18581000,
iast_GLOBAL (18.017 s) : 18017000, 18017000
. : milestone, 18017000,
profiling (15.13 s) : 15130000, 15130000
. : milestone, 15130000,
tracing (15.303 s) : 15303000, 15303000
. : milestone, 15303000,
section candidate
no_agent (15.125 s) : 15125000, 15125000
. : milestone, 15125000,
appsec (14.573 s) : 14573000, 14573000
. : milestone, 14573000,
iast (18.144 s) : 18144000, 18144000
. : milestone, 18144000,
iast_GLOBAL (18.065 s) : 18065000, 18065000
. : milestone, 18065000,
profiling (14.979 s) : 14979000, 14979000
. : milestone, 14979000,
tracing (14.858 s) : 14858000, 14858000
. : milestone, 14858000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~d28e883862, baseline=1.61.0-SNAPSHOT~517a89d1ec3
dateFormat X
axisFormat %s
section baseline
no_agent (1.477 ms) : 1465, 1488
. : milestone, 1477,
appsec (3.801 ms) : 3581, 4022
. : milestone, 3801,
iast (2.256 ms) : 2187, 2326
. : milestone, 2256,
iast_GLOBAL (2.31 ms) : 2239, 2380
. : milestone, 2310,
profiling (2.103 ms) : 2046, 2160
. : milestone, 2103,
tracing (2.082 ms) : 2027, 2136
. : milestone, 2082,
section candidate
no_agent (1.475 ms) : 1463, 1486
. : milestone, 1475,
appsec (3.816 ms) : 3595, 4037
. : milestone, 3816,
iast (2.267 ms) : 2197, 2337
. : milestone, 2267,
iast_GLOBAL (2.31 ms) : 2239, 2380
. : milestone, 2310,
profiling (2.128 ms) : 2070, 2186
. : milestone, 2128,
tracing (2.083 ms) : 2029, 2138
. : milestone, 2083,
|
PerfectSlayer
approved these changes
Mar 26, 2026
PerfectSlayer
left a comment
PerfectSlayer
left a comment
Collaborator
There was a problem hiding this comment.
Looking good
Glad you made a dedicated script in the first place, it would have being harder to follow using inline script into Yaml 😓 (I will ask Codex review to be sure, Perl is not my strong suite)
Collaborator
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 259f4c2049
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| # Update the GitLab SYSTEM_TESTS_REF variable | ||
| echo "Updating 'SYSTEM_TESTS_REF' reference..." | ||
| perl -pe "s/$GITLAB_PATTERN/\${1}$REF\${3}/g" "$GITLAB_TARGET" > "$TEMP_FILE" |
There was a problem hiding this comment.
This replacement writes the raw commit SHA into .gitlab-ci.yml as an unquoted YAML scalar, which can be re-typed by YAML parsers instead of preserved verbatim. In particular, SHAs beginning with 0x (a valid SHA prefix) are parsed as hex integers by Ruby/Psych, so SYSTEM_TESTS_REF can become a decimal string in CI and no longer match a git ref, causing release-branch system-test pinning to fail intermittently. The inserted value should be quoted (and the pattern should preserve quotes) to force string semantics.
Useful? React with 👍 / 👎.
Collaborator
There was a problem hiding this comment.
Oh, interesting 🤔 That could be worst confirming the issue Sarah 🙏
Contributor
Author
There was a problem hiding this comment.
For previous runs (example), the fetched SHA doesn't contain a starting 0x and pins correctly for the Github ref here. Running the script locally applies the Gitlab ref like it does to the Github one, so the Gitlab SHA ref with no quotes should be fine as well.
I was thinking it wouldn't hurt to add "s to be safe, but it makes the patterns harder to read IMO (e.g. (\s*SYSTEM_TESTS_REF:\s*)(["'\'']?)([^"'\''\s#]+)(["'\'']?)(\s+# system tests.*) vs. (\s*SYSTEM_TESTS_REF:\s*)(\S+)(\s+# system tests.*)).
I think we should keep as-is for readability unless you have strong opinions otherwise!
Collaborator
There was a problem hiding this comment.
It makes it harder to read because it handles the quote as optional. But if we always use quoted value, it should be similar, right?
Contributor
Author
There was a problem hiding this comment.
Oh like (\s*SYSTEM_TESTS_REF:\s*")([^"]+)("\s+# system tests.*)... Much better!
randomanderson
approved these changes
Mar 26, 2026
Contributor
Author
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
sarahchen6/pin-one-pipeline-system-tests
branch
March 26, 2026 16:05
mikewoo-dd
changed the title
Contributor
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Pin
one-pipelinesystem tests for release branches. These currently default tomainin thelibdatadog-buildrepo here.Motivation
In the recent 1.60.2 patch release, Gitlab system tests were broken because they were still unpinned (i.e. using the tip of
main) whereas Github system tests were working because of existing tooling that had already pinned these system tests here in our Github Action workflow.Additional Notes
Testing was done by:
system-testsPR with an extra print line,BRANCH="sarahchen6/test-system-tests-pinning" tooling/update_system_test_reference.shlocally to pin Github and Gitlab system tests,compute_pipelineCI job exist (job).Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.