Skip to content

Improving smoke test capabilities - #12051

Open
PerfectSlayer wants to merge 8 commits into
masterfrom
bbujon/smoke-tests-tooling
Open

Improving smoke test capabilities#12051
PerfectSlayer wants to merge 8 commits into
masterfrom
bbujon/smoke-tests-tooling

Conversation

@PerfectSlayer

@PerfectSlayer PerfectSlayer commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

What Does This Do

This PR contains the follow up changes from all the other PRs of the stack.
In particular:

  • Add span link support to decoded spans and their assert rules
  • Remove test agent check customization (scope creep, not actionable in CI).

Motivation

Follow up changes from PR reviews listed in the additional notes.

Additional Notes

Related pull-requests:

Contributor Checklist

Jira ticket: APMLP-1247

@PerfectSlayer PerfectSlayer added type: feature Enhancements and improvements comp: testing Testing tag: do not merge Do not merge changes labels Jul 23, 2026
@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-tooling branch from 29804b8 to 817d668 Compare July 28, 2026 08:49
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
Patch Coverage: 34.48%
Overall Coverage: 57.39% (-1.62%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: adcb95a | Docs | View more details | Give us feedback!

@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-tooling branch from 817d668 to c659547 Compare July 28, 2026 09:11
@dd-octo-sts

dd-octo-sts Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.74 s 14.67 s [-0.5%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent 13.64 s 13.72 s [-1.2%; +0.1%] (no difference)
startup:petclinic:appsec:Agent 17.09 s 16.95 s [-0.0%; +1.7%] (no difference)
startup:petclinic:iast:Agent 16.93 s 17.03 s [-1.4%; +0.2%] (no difference)
startup:petclinic:profiling:Agent 16.75 s 16.88 s [-1.8%; +0.3%] (no difference)
startup:petclinic:sca:Agent 16.33 s 16.91 s [-8.8%; +1.9%] (unstable)
startup:petclinic:tracing:Agent 16.18 s 16.02 s [+0.1%; +1.9%] (maybe worse)

Commit: adcb95a8 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Bits found no code fix to apply

🟢 Investigated · ⚪ No code fix needed

main / End-to-end #3 / spring-boot-payara 3 fails because five Payara system tests cannot find the expected IAST span tag; the downstream Check system tests success gate only reports that failure. The failure is in an unchanged system-test path and is classified as a flaky test failure, while this PR changes smoke-test tooling and migrations, so no code change was made.


View in Datadog | Reviewed commit e39c84d · Any feedback? Reach out in #deveng-pr-agent

@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-tooling branch from c659547 to 089af14 Compare July 29, 2026 06:09
@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-tooling branch 6 times, most recently from f45d96b to 89f412f Compare August 7, 2026 08:13
@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-tooling branch 2 times, most recently from e39c84d to adca11c Compare August 13, 2026 12:33
@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-tooling branch 9 times, most recently from d28d501 to 18ce2e4 Compare August 27, 2026 09:13
@PerfectSlayer
PerfectSlayer changed the base branch from master to bbujon/smoke-tests-documentation August 27, 2026 09:14
@PerfectSlayer PerfectSlayer removed the tag: do not merge Do not merge changes label Aug 27, 2026
@PerfectSlayer
PerfectSlayer marked this pull request as ready for review August 27, 2026 09:32
@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-tooling branch from 18ce2e4 to 0cf64b4 Compare August 28, 2026 07:00
@PerfectSlayer

Copy link
Copy Markdown
Collaborator Author

I'll push the last changes from feedback as soon as the document PR is merged

@bric3 bric3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's still the point on PR title and description, otherwise it looks good to me. (pre-approving)

Base automatically changed from bbujon/smoke-tests-documentation to master August 28, 2026 08:55

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The v1 span-link decoder rejects valid unsigned span IDs with bit 63 set. This error stops decoding of the full trace payload.

Open Bits AI session

🤖 Datadog Autotest · Commit 0cf64b4 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@PerfectSlayer

Copy link
Copy Markdown
Collaborator Author

There's still the point on PR title and description, otherwise it looks good to me

Don't you think the final PR of the stack could sum up the whole work?
Would you rather have it limited to the last change only?

@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-tooling branch from 0cf64b4 to 3b675f0 Compare August 28, 2026 09:47
@PerfectSlayer

Copy link
Copy Markdown
Collaborator Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Aug 28, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-08-28 11:02:55 UTC ℹ️ Start processing command /merge


2026-08-28 11:03:04 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

  • Run /code blockers to see what is blocking it.
  • Run /remove to cancel it.

2026-08-28 15:55:18 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

@bric3

bric3 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Don't you think the final PR of the stack could sum up the whole work?
Would you rather have it limited to the last change only?

Having a summary of earlier work is OK, but the PR itself should precisely describe its change it tries to get in the repo not the previous ones. If the commit landing in git is too broad then it misleads. Gicen the title is used in git it should be tailored to the PR's changes.

In this case the commit will be "Improving smoke test capabilities" but it really is about span links harness tooling.

So yes I think the description should be focused on the changes. And possibly have a specific section that explain the complete set of work required to get there. Alternatively, I think it might be better to use an issue to track work.

@PerfectSlayer

PerfectSlayer commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

So yes I think the description should be focused on the changes.

At first, I thought about having another PR to stack for those changes, and keep this PR as the "preview full stack PR", still in draft. But as PR management / reviews become too painful, I did not want to go the clean way. I already baby sit PRs from 8am to 9pm so until we fix it, I'm not sure I will keep doing the "clean way". PRs are already way worst in the repo 😢

If the commit landing in git is too broad then it misleads

Commit log became significantly worse since merge queues… I don't use it anymore and not sure people still care about what is going in 😞 Anyway, I moved all the text behind HTLM comments so I can still read it but it won't be displayed.

@bric3

bric3 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

@PerfectSlayer Don't worry I get you 😢

@ygree

ygree commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-01T18:29:54.123005Z 3b675f0 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3b675f00e7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread dd-smoke-tests/src/main/java/datadog/smoketest/trace/SpanLinkMatcher.java Outdated
@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-tooling branch from acfa565 to 8559be0 Compare September 3, 2026 07:15
.message("Unexpected span link count")
.expected(this.linkMatchers.length)
.actual(links.size())
.buildAndThrow();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's throwing unconditionally anyway, why not just use the "throw" keyword?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m using the dedicated fluent API up to the throw. I added a manual throw / return in the other method to help with IntelliJ static checker but there is no other benefits. buildAndThrow() seems the right way to use it according the doc.

Comment on lines +13 to +14
/**
* Returns the span 64-bit trace identifier, dropping high-order bits if present.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's confusing, why getTraceId returns the span trace id?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That’s why I documented it. There is no model for trace, only span. So the trace ID is store within span.
Additionally, this trace ID is not the exact one but truncated… Another reason to document the behavior.
Note that I did not change it, I only documented the behavior.

@ygree ygree left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IDM mostly approves, except for a couple of places where I left notes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: testing Testing type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants