Skip to content

[APMLP-598] Pin one-pipeline system tests on release branches #10966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gh-worker-dd-mergequeue-cf854d merged 5 commits into from
Mar 26, 2026
Merged

[APMLP-598] Pin one-pipeline system tests on release branches #10966

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d8d886b
Pin one-pipeline system tests
sarahchen6 Mar 25, 2026
1065da5
Test pinning system-tests
sarahchen6 Mar 25, 2026
a169c03
Revert test pinning
sarahchen6 Mar 25, 2026
259f4c2
Simplify paths
sarahchen6 Mar 25, 2026
d28e883
Fix comments
sarahchen6 Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/create-release-branch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ jobs:
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git commit -m "chore: Pin system-tests for release branch" .github/workflows/run-system-tests.yaml
git commit -m "chore: Pin system-tests for release branch" .github/workflows/run-system-tests.yaml .gitlab-ci.yml
echo "commit=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT

- name: Push changes
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/run-system-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ jobs:
needs:
- build
# If you change the following comment, update the pattern in the update_system_test_reference.sh script to match.
uses: DataDog/system-tests/.github/workflows/system-tests.yml@main # system tests are pinned for releases only
uses: DataDog/system-tests/.github/workflows/system-tests.yml@main # system tests are pinned on release branches only
secrets:
TEST_OPTIMIZATION_API_KEY: ${{ secrets.DATADOG_API_KEY_PROD }}
permissions:
Expand All @@ -72,7 +72,7 @@ jobs:
with:
library: java
# If you change the following comment, update the pattern in the update_system_test_reference.sh script to match.
ref: main # system tests are pinned for releases only
ref: main # system tests are pinned on release branches only
binaries_artifact: binaries
desired_execution_time: 900 # 15 minutes
scenarios_groups: tracer-release
Expand Down
1 change: 1 addition & 0 deletions .gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1071,6 +1071,7 @@ publishing-gate:

configure_system_tests:
variables:
SYSTEM_TESTS_REF: main # system tests are pinned on release branches only
SYSTEM_TESTS_SCENARIOS_GROUPS: "simple_onboarding,simple_onboarding_profiling,simple_onboarding_appsec,docker-ssi,lib-injection"

create_key:
Expand Down
32 changes: 22 additions & 10 deletions tooling/update_system_test_reference.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
set -euo pipefail

# This script updates the system-tests reference in run-system-tests.yaml.
# This script updates the system-tests references used by GitHub Actions (run-system-tests.yaml) and GitLab CI (.gitlab-ci.yml).
# The reference will be updated with the latest commit SHA of the given branch (or `main` if not set) of https://github.com/DataDog/system-tests.
# Usage: BRANCH=<branch-name> tooling/update_system_test_reference.sh

Expand All @@ -11,9 +11,11 @@ if [ -z "${BRANCH:-}" ]; then
echo "BRANCH is not set. Defaulting to 'main'."
fi

TARGET=".github/workflows/run-system-tests.yaml" # target file to update
PATTERN_1='(\s*system-tests\.yml@)(\S+)(\s+# system tests.*)' # pattern to update the "system-tests.yml@" reference
PATTERN_2='(\s*ref: )(\S+)(\s+# system tests.*)' # pattern to update the "ref:" reference
GITHUB_TARGET=".github/workflows/run-system-tests.yaml"
GITLAB_TARGET=".gitlab-ci.yml"
GITHUB_PATTERN_1='(\s*system-tests\.yml@)(\S+)(\s+# system tests.*)' # pattern to update the "system-tests.yml@" reference
GITHUB_PATTERN_2='(\s*ref: )(\S+)(\s+# system tests.*)' # pattern to update the "ref:" reference
GITLAB_PATTERN='(\s*SYSTEM_TESTS_REF:\s*)(\S+)(\s+# system tests.*)' # pattern to update the GitLab SYSTEM_TESTS_REF variable

echo "Fetching latest commit SHA for system-tests branch: $BRANCH"
REF=$(git ls-remote https://github.com/DataDog/system-tests "refs/heads/$BRANCH" | cut -f 1)
Expand All @@ -23,8 +25,13 @@ if [ -z "$REF" ]; then
fi
echo "Fetched SHA: $REF"

if [ ! -f "$TARGET" ]; then
echo "Error: Target file $TARGET does not exist"
if [ ! -f "$GITHUB_TARGET" ]; then
echo "Error: Target file $GITHUB_TARGET does not exist"
exit 1
fi

if [ ! -f "$GITLAB_TARGET" ]; then
echo "Error: Target file $GITLAB_TARGET does not exist"
exit 1
fi

Expand All @@ -33,13 +40,18 @@ TEMP_FILE=$(mktemp)

# Update the "system-tests.yml@" reference
echo "Updating 'system-tests.yml@' reference..."
perl -pe "s/$PATTERN_1/\${1}$REF\${3}/g" "$TARGET" > "$TEMP_FILE"
cp "$TEMP_FILE" "$TARGET"
perl -pe "s/$GITHUB_PATTERN_1/\${1}$REF\${3}/g" "$GITHUB_TARGET" > "$TEMP_FILE"
cp "$TEMP_FILE" "$GITHUB_TARGET"

# Update the "ref:" reference
echo "Updating 'ref:' reference..."
perl -pe "s/$PATTERN_2/\${1}$REF\${3}/g" "$TARGET" > "$TEMP_FILE"
cp "$TEMP_FILE" "$TARGET"
perl -pe "s/$GITHUB_PATTERN_2/\${1}$REF\${3}/g" "$GITHUB_TARGET" > "$TEMP_FILE"
cp "$TEMP_FILE" "$GITHUB_TARGET"

# Update the GitLab SYSTEM_TESTS_REF variable
echo "Updating 'SYSTEM_TESTS_REF' reference..."
perl -pe "s/$GITLAB_PATTERN/\${1}$REF\${3}/g" "$GITLAB_TARGET" > "$TEMP_FILE"

@chatgpt-codex-connector chatgpt-codex-connector Bot Mar 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Quote GitLab ref value when writing pinned SHA

This replacement writes the raw commit SHA into .gitlab-ci.yml as an unquoted YAML scalar, which can be re-typed by YAML parsers instead of preserved verbatim. In particular, SHAs beginning with 0x (a valid SHA prefix) are parsed as hex integers by Ruby/Psych, so SYSTEM_TESTS_REF can become a decimal string in CI and no longer match a git ref, causing release-branch system-test pinning to fail intermittently. The inserted value should be quoted (and the pattern should preserve quotes) to force string semantics.

Useful? React with 👍 / 👎.

@PerfectSlayer PerfectSlayer Mar 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, interesting 🤔 That could be worst confirming the issue Sarah 🙏

@sarahchen6 sarahchen6 Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For previous runs (example), the fetched SHA doesn't contain a starting 0x and pins correctly for the Github ref here. Running the script locally applies the Gitlab ref like it does to the Github one, so the Gitlab SHA ref with no quotes should be fine as well.

I was thinking it wouldn't hurt to add "s to be safe, but it makes the patterns harder to read IMO (e.g. (\s*SYSTEM_TESTS_REF:\s*)(["'\'']?)([^"'\''\s#]+)(["'\'']?)(\s+# system tests.*) vs. (\s*SYSTEM_TESTS_REF:\s*)(\S+)(\s+# system tests.*)).

I think we should keep as-is for readability unless you have strong opinions otherwise!

@PerfectSlayer PerfectSlayer Mar 30, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes it harder to read because it handles the quote as optional. But if we always use quoted value, it should be similar, right?

@sarahchen6 sarahchen6 Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh like (\s*SYSTEM_TESTS_REF:\s*")([^"]+)("\s+# system tests.*)... Much better!

@sarahchen6 sarahchen6 Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in #10998

cp "$TEMP_FILE" "$GITLAB_TARGET"

# Clean up temporary file
rm -f "$TEMP_FILE"
Expand Down
Loading