Skip to content

smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu26) - #642

Merged
james-nesbitt merged 12 commits into
mainfrom
smoke/cutting-edge-test
Nov 11, 2026
Merged

james-nesbitt merged 12 commits into
mainfrom
smoke/cutting-edge-test

Conversation

@james-nesbitt

@james-nesbitt james-nesbitt commented Jun 8, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds TestCuttingEdgeCluster: rhel10, rocky10 and ubuntu26 as both managers and workers, on MCR stable-29.8.0 and MKE 3.9.7, replacing the single-node version of that test currently on main. Carries the kernel-modules-extra cloud-init payload the EL10 nodes need in order to start MCR at all.

Why

No smoke job covers RHEL 10, Rocky 10 or Ubuntu 26.04 in worker roles.

The EL10 failure, and what it actually was

Every previous run of this matrix failed on all four EL10 nodes at Install MCR. Earlier revisions of this PR attributed that to MCR's nftables backend lacking Swarm parity and concluded it was not a launchpad problem. That diagnosis was wrong. dockerd never got as far as choosing a firewall backend:

failed to register "bridge" driver: failed to add jump rules to ipv4 NAT table:
iptables --wait -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER:
Warning: Extension addrtype revision 0 not supported, missing kernel module?

xt_addrtype ships in kernel-modules-extra, which EL10 minimal and cloud images do not carry. iptables-1.8.11-15.el10 stopped recommending that package unconditionally, and docker-ee does not Require it — docker/docker-ce-packaging#1340.

The nftables backend was never an alternative here for an unrelated reason: Docker's docs state it "cannot be enabled when the Docker daemon is running in Swarm mode", which MKE requires.

Measured, not inferred

On EC2 in us-east-1, against the AMIs provision-aws v0.3.0 resolves for rhel_10 and rocky_10:

Host systemctl start docker docker info
control, no fix rc=1, is-active activating Cannot connect to the Docker daemon
rhel_10 + payload rc=0, active 29.8.0m1 overlayfs
rocky_10 + payload rc=0, active 29.8.0m1 overlayfs

The control is the informative half. Installing docker-ee there did pull kernel-modules-extra — for 6.12.0-211.74.1, onto a host running 6.12.0-211.16.1. The resolver picks the newest kernel's modules, which do nothing for the booted one, and docker still failed. Hence the payload pins kernel-modules-extra-$(uname -r) rather than leaving it to the resolver.

The payload needs a shebang

provision-aws passes user_data to the launch template verbatim — base64encode(var.user_data), no wrapping — so cloud-init executes only what it recognises.

Measured on the control instance, which carries the same bare sudo firewall-cmd … string every other Linux platform in test/platforms.go uses: cloud-init logged Unhandled non-multipart and /var/lib/cloud/instance/scripts/ was empty. On the fixed hosts: zero such log lines, and part-001 present. Those firewall rules have never executed, on any platform, in any smoke run. That is a property of main, not of this PR, and nothing here depends on changing it.

The new payload carries no firewall rules either: firewalld is not installed on either EL10 AMI, firewall-cmd is not on PATH, and node-to-node traffic is already carried by the security group's self rule in examples/terraform/aws-simple/launchpad.tf.

Racing cloud-init against launchpad's own dnf

cloud-init installs a package at boot while launchpad installs MCR over SSH, so they can contend for the RPM lock. Measured: a second dnf waits (Waiting for process with pid … to finish.) rather than failing. sshd becomes reachable 27s (rhel) to 59s (rocky) before cloud-init finishes, against roughly five minutes between instance creation and Install MCR in CI. No guard needed.

Other changes

  • MCRChannel stable-29.6.1 → stable-29.8.0: 29.6.1 does not exist for resolute
  • MKEVersion 3.9.6 → 3.9.7, the pairing with MCR 29.8.0 exercised live in PRODENG-4062
  • pkg/configurer/linux.go: EnableMCR now attaches journalctl -xeu docker output to a start failure. rig's StartService discards systemd's stderr, so the error above was otherwise invisible — this hunk is what made the diagnosis possible, and has standalone value
  • main's TestCuttingEdgeCluster (from [PRODENG-3593] Add Ubuntu 26.04 (Resolute) host configurer #661) is removed. It ran one ubuntu_26.04 manager against repos-internal.mirantis.com because resolute packages existed nowhere else on 2026-09-29; the public repo now carries them, and both ubuntu26 nodes installed MCR 29.8.0m1 cleanly in the 2026-11-06 run
  • The smoke-cutting-edge job is aligned with the six other suites in smoke-tests.yaml: arc-runner-set-mirantis-public, SHA-pinned checkout and setup-terraform, persist-credentials: false, terraform_wrapper: false, and the Set up Go step it was missing
  • Makefile and workflow comments claiming the job was merged red pending MCR packages for rhel/10 and rocky/10 are corrected: those packages exist (repos.mirantis.com/rhel/10 and /rocky/10 both return 200 for stable-29.8.0) and install cleanly

Accepted trade

This removes the only smoke coverage of repos-internal.mirantis.com: smoke_test.go is the sole setter of MCRRepoURL. The field and its plumbing stay, so reinstating coverage later is a one-line change on another suite. Accepted deliberately rather than left open.

Testing

  • EC2 validation above: three instances, control plus both EL10 platforms, torn down afterwards
  • go vet ./test/... and gofmt -l test/: clean
  • GHA structural check on smoke-tests.yaml: clean

Known risk

The EL10 nodes now get past Install MCR for the first time, so this run exercises MKE 3.9.7 on RHEL 10 and Rocky 10 — which nothing has done before. A second failure further down the phase list is possible.

Links

Checklist

  • Tests added or updated
  • Docs updated if user-visible behaviour changed (N/A)
  • No debug output or dead code left in

Written by AI: claude-opus-5

@james-nesbitt james-nesbitt added the smoke-cutting-edge Trigger cutting-edge smoke test label Jun 8, 2026
@james-nesbitt
james-nesbitt force-pushed the smoke/cutting-edge-test branch from 093c76b to d1a9f2d Compare July 22, 2026 07:24
james-nesbitt added a commit that referenced this pull request Sep 17, 2026
Registers ResoluteConfigurer for os.ID == "ubuntu" && os.Version ==
"26.04", matching the existing per-codename pattern (bionic, focal,
jammy, noble, xenial). No version-specific behavior needed: the shared
ubuntu.Configurer.InstallMCR already builds the APT repo URL generically
from VERSION_CODENAME.

MCR has not published Docker EE packages for Ubuntu 26.04 on either
repos.mirantis.com or repos-internal.mirantis.com as of 2026-09-02 (both
list only trusty/xenial/bionic/focal/jammy/noble). InstallMCR will fail
on this platform until MCR ships support - this configurer only makes
launchpad recognize the OS, it does not unblock installs. Same gap
already tracked by TestCuttingEdgeCluster in PR #642.

Verified: go build ./..., go vet, go test ./pkg/product/mke/config/...
./pkg/configurer/..., golangci-lint run (no new issues).
@james-nesbitt
james-nesbitt force-pushed the smoke/cutting-edge-test branch from 33a8927 to eac251c Compare September 17, 2026 15:01
james-nesbitt added a commit that referenced this pull request Sep 17, 2026
EnableMCR's error on a failed "systemctl start docker" was always a
bare "exit status 1" with no way to tell why: rig's StartService
issues "systemctl start docker 2> /dev/null", discarding systemd's own
diagnostic output.

Reproduced live on RHEL10/Rocky10 in TestCuttingEdgeCluster (PR #642)
with the latest available MCR (stable-29.6.1) and MKE (3.9.6): package
install succeeds, but docker.service still fails to start, with no
way from launchpad's own logs to see why -- exactly the gap this PR's
description flags as "root cause not yet isolated (stderr suppressed
by the install command)".

Best-effort fetch journalctl output for the docker unit on failure and
include it in the returned error, so the actual systemd failure reason
is visible without needing rig changes or manual SSH access. Ignore a
failure to fetch the diagnostic itself, since that must not mask the
original error.

Written by AI: claude-sonnet-5

Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>
@james-nesbitt james-nesbitt changed the title smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu24/sles15) smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu26/sles16) Sep 17, 2026
@james-nesbitt

Copy link
Copy Markdown
Collaborator Author

Current status: re-verified today with the latest available MCR (stable-29.6.1) and MKE (3.9.6). Still not passing, but root causes are now fully isolated for every platform in this test.

  • RHEL 10 / Rocky 10: package install succeeds; docker.service still fails to start. Added a diagnostic capture (journalctl -xeu docker) to pkg/configurer/linux.go, since rig's StartService was discarding systemd's own stderr. Isolated cause: RHEL 10/Rocky 10's iptables-nft backend does not support the addrtype match Docker's bridge driver requires ("Extension addrtype revision 0 not supported"). Matches MCR 29.6.1's own release notes, which list Swarm/nftables parity as still "planned for upcoming releases." Not a launchpad bug.
  • Ubuntu 26.04: was aborting the whole test at OS detection (launchpad didn't recognize the OS at all). Fixed by cherry-picking the configurer from [PRODENG-3593] Add Ubuntu 26.04 (Resolute) host configurer #661. Now correctly reaches Install MCR, where it fails because MCR has not published Ubuntu 26.04 packages yet (unchanged since July).
  • SLES 16: still no packages published at all (unchanged since July).

Confirmed identically across 2 local runs and 1 official CI run (35238411403) with no discrepancies. No orphaned AWS resources left behind.

Written by AI: claude-sonnet-5

james-nesbitt added a commit that referenced this pull request Sep 20, 2026
Registers ResoluteConfigurer for os.ID == "ubuntu" && os.Version ==
"26.04", matching the existing per-codename pattern (bionic, focal,
jammy, noble, xenial). No version-specific behavior needed: the shared
ubuntu.Configurer.InstallMCR already builds the APT repo URL generically
from VERSION_CODENAME.

MCR has not published Docker EE packages for Ubuntu 26.04 on either
repos.mirantis.com or repos-internal.mirantis.com as of 2026-09-02 (both
list only trusty/xenial/bionic/focal/jammy/noble). InstallMCR will fail
on this platform until MCR ships support - this configurer only makes
launchpad recognize the OS, it does not unblock installs. Same gap
already tracked by TestCuttingEdgeCluster in PR #642.

Verified: go build ./..., go vet, go test ./pkg/product/mke/config/...
./pkg/configurer/..., golangci-lint run (no new issues).
james-nesbitt added a commit that referenced this pull request Sep 20, 2026
EnableMCR's error on a failed "systemctl start docker" was always a
bare "exit status 1" with no way to tell why: rig's StartService
issues "systemctl start docker 2> /dev/null", discarding systemd's own
diagnostic output.

Reproduced live on RHEL10/Rocky10 in TestCuttingEdgeCluster (PR #642)
with the latest available MCR (stable-29.6.1) and MKE (3.9.6): package
install succeeds, but docker.service still fails to start, with no
way from launchpad's own logs to see why -- exactly the gap this PR's
description flags as "root cause not yet isolated (stderr suppressed
by the install command)".

Best-effort fetch journalctl output for the docker unit on failure and
include it in the returned error, so the actual systemd failure reason
is visible without needing rig changes or manual SSH access. Ignore a
failure to fetch the diagnostic itself, since that must not mask the
original error.

Written by AI: claude-sonnet-5

Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>
@james-nesbitt
james-nesbitt force-pushed the smoke/cutting-edge-test branch from 310d91c to a9592df Compare September 20, 2026 06:59
@james-nesbitt
james-nesbitt marked this pull request as draft September 21, 2026 07:25
james-nesbitt added a commit that referenced this pull request Sep 29, 2026
Registers ResoluteConfigurer for os.ID == "ubuntu" && os.Version ==
"26.04", matching the existing per-codename pattern (bionic, focal,
jammy, noble, xenial). No version-specific behavior needed: the shared
ubuntu.Configurer.InstallMCR already builds the APT repo URL generically
from VERSION_CODENAME.

MCR has not published Docker EE packages for Ubuntu 26.04 on either
repos.mirantis.com or repos-internal.mirantis.com as of 2026-09-02 (both
list only trusty/xenial/bionic/focal/jammy/noble). InstallMCR will fail
on this platform until MCR ships support - this configurer only makes
launchpad recognize the OS, it does not unblock installs. Same gap
already tracked by TestCuttingEdgeCluster in PR #642.

Verified: go build ./..., go vet, go test ./pkg/product/mke/config/...
./pkg/configurer/..., golangci-lint run (no new issues).
james-nesbitt added a commit that referenced this pull request Sep 29, 2026
Registers ResoluteConfigurer for os.ID == "ubuntu" && os.Version ==
"26.04", matching the existing per-codename pattern (bionic, focal,
jammy, noble, xenial). No version-specific behavior needed: the shared
ubuntu.Configurer.InstallMCR already builds the APT repo URL generically
from VERSION_CODENAME.

MCR has not published Docker EE packages for Ubuntu 26.04 on either
repos.mirantis.com or repos-internal.mirantis.com as of 2026-09-02 (both
list only trusty/xenial/bionic/focal/jammy/noble). InstallMCR will fail
on this platform until MCR ships support - this configurer only makes
launchpad recognize the OS, it does not unblock installs. Same gap
already tracked by TestCuttingEdgeCluster in PR #642.

Verified: go build ./..., go vet, go test ./pkg/product/mke/config/...
./pkg/configurer/..., golangci-lint run (no new issues).
james-nesbitt added a commit that referenced this pull request Sep 29, 2026
* [PRODENG-3593] Add Ubuntu 26.04 (Resolute) host configurer

Registers ResoluteConfigurer for os.ID == "ubuntu" && os.Version ==
"26.04", matching the existing per-codename pattern (bionic, focal,
jammy, noble, xenial). No version-specific behavior needed: the shared
ubuntu.Configurer.InstallMCR already builds the APT repo URL generically
from VERSION_CODENAME.

MCR has not published Docker EE packages for Ubuntu 26.04 on either
repos.mirantis.com or repos-internal.mirantis.com as of 2026-09-02 (both
list only trusty/xenial/bionic/focal/jammy/noble). InstallMCR will fail
on this platform until MCR ships support - this configurer only makes
launchpad recognize the OS, it does not unblock installs. Same gap
already tracked by TestCuttingEdgeCluster in PR #642.

Verified: go build ./..., go vet, go test ./pkg/product/mke/config/...
./pkg/configurer/..., golangci-lint run (no new issues).

* Fix resolute.go registration for rig v2 migration

PRODENG-3594 (#651) migrated OS module registration from
k0sproject/rig + rig/os/registry (func(rig.OSVersion) bool) to
pkg/configurer.RegisterOSModule + rigos.Release (func(*rigos.Release)
bool), landed on main after this branch was created. Update
ResoluteConfigurer to match the current pattern used by
NobleConfigurer; without this the branch fails to compile against
main.

Written by AI: claude-sonnet-5

Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>

* Add TestCuttingEdgeCluster smoke test for Ubuntu 26.04

- test/platforms.go: add Ubuntu26 platform entry (ubuntu_26.04)
- test/smoke/smoke_test.go: add MCRRepoURL to smokeConfig, wired into
  the launchpad tfvars map as mcr_repo_url when non-empty (defaults
  to the public repo otherwise, via PR #669's optional tfvar); add
  TestCuttingEdgeCluster exercising a single ubuntu_26.04 manager
  against repos-internal.mirantis.com channel test-29, the only
  place Ubuntu 26.04 MCR packages exist as of 2026-09-29

Verified: TestCuttingEdgeCluster passed against real AWS infra
(docker-testing-533267045383, us-east-1), MCR engine 29.8.0m1-rc1
installed and detected by MKE 3.9.2, cluster reported healthy,
Reset succeeded, all 28 AWS resources destroyed cleanly.

Written by AI: claude-sonnet-5

Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>

---------

Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>
james-nesbitt and others added 8 commits November 3, 2026 08:29
Adds a new smoke test targeting the latest gen-10 distributions:
- RHEL 10, Rocky 10, Ubuntu 24.04 as managers and workers
- SLES 15 as an additional worker

Also adds:
- Rhel10 and Rocky10 platform definitions in test/platforms.go
- smoke-cutting-edge Makefile target (50m timeout)
- smoke-cutting-edge CI job triggered by the smoke-cutting-edge label

NOTE: This test depends on terraform-mirantis-modules/terraform-mirantis-provision-aws
PR #22 being merged (adds rhel_10 and rocky_10 platform keys to the
provision-aws module). The test will fail until the module is updated and
the version bumped in examples/terraform/aws-simple/.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>
- shorten Name "cutting-edge" -> "cuttingedge" (11 chars): the 12-char
  value overflowed the 32-char AWS LB/target-group name limit and was
  the actual CI failure ("name" cannot be longer than 32 characters)
- pin examples/terraform/aws-simple to provision-aws >= 0.1.7, which
  ships the rhel_10/rocky_10 platform keys (provision-aws#22, released)
- refresh test doc comment now the upstream dependency is merged/released

gofmt also normalized two pre-existing violations in smoke_test.go.
rhel10/rocky10 AMIs and provision-aws platform keys (v0.1.7) are already
available, but launchpad's own smoke coverage previously failed with
'package manager could not install docker-ee' because repos.mirantis.com
had not yet published docker-ee for rhel/10 or rocky/10.

Checked repos.mirantis.com directly: docker-ee is now present in both
rhel/10/x86_64/stable-29.4 and rocky/10/x86_64/stable-29.4 -- that gap
looks closed, matching the AMI-availability timing pattern seen
elsewhere this cycle (Oracle Linux, Rocky 8).

Strategy going forward: keep TestCuttingEdgeCluster merged and
label-gated rather than blocking on it going green immediately, so it
picks up MCR support automatically as soon as it lands, with no further
code changes. Documented that intent on the Makefile target and the
GitHub Actions job.

Written by AI: claude-sonnet-5
Adds a Sles16 platform definition (sles_16) and switches
TestCuttingEdgeCluster's additional worker from sles15 to sles16.

NOTE: MCR has not published Docker EE packages for SLES 16 as of
2026-07-22 (repos.mirantis.com/sles/ only has 12/12.3/15). This node
is expected to fail at Install MCR until MCR ships SLES16 support -
same category of gap as RHEL10/Rocky10 (docker-ee installs but the
daemon won't start), tracked separately for that pair. Flagged in a
doc comment on the test so CI failures here are understood, not
mysterious.

gofmt also cleaned two pre-existing, unrelated violations in these
files (trailing blank line in platforms.go; WrkWin2025 alignment in
TestFIPSCluster).
Adds a Ubuntu26 platform definition (ubuntu_26.04, codename resolute)
and switches TestCuttingEdgeCluster's Ubuntu manager+worker from 24.04
to 26.04 - the current LTS as of 2026-07 (24.04 remains fully
supported, just no longer the newest release).

The ubuntu_26.04 key is already present in the pinned provision-aws
module (>= v0.1.7), AMI pattern confirmed against a live published AMI
(ubuntu-resolute-26.04-amd64-server-*, published as recently as
2026-07-22).

NOTE: MCR has not published Docker EE packages for Ubuntu 26.04 yet
(repos.mirantis.com/ubuntu/dists/ only has trusty/xenial/bionic/focal/
jammy/noble - no resolute). This node is expected to fail at Install
MCR alongside the existing SLES16 and RHEL10/Rocky10 gaps, until MCR
ships support. Flagged in the test's doc comment.
Re-checking whether the RHEL10/Rocky10 MCR-daemon-start gap and the
Ubuntu 26.04/SLES 16 missing-package gaps (documented in this PR's
description) have closed with the newest available releases. Was
stable-29.4/MKE 3.9.2 (docker-ee-29.4.1m1); repos.mirantis.com now also
publishes stable-29.6.1 (docker-ee-29.6.1m2) for rhel/10.
MSRVersion unchanged (3.1.18), already current elsewhere.

Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>
EnableMCR's error on a failed "systemctl start docker" was always a
bare "exit status 1" with no way to tell why: rig's StartService
issues "systemctl start docker 2> /dev/null", discarding systemd's own
diagnostic output.

Reproduced live on RHEL10/Rocky10 in TestCuttingEdgeCluster (PR #642)
with the latest available MCR (stable-29.6.1) and MKE (3.9.6): package
install succeeds, but docker.service still fails to start, with no
way from launchpad's own logs to see why -- exactly the gap this PR's
description flags as "root cause not yet isolated (stderr suppressed
by the install command)".

Best-effort fetch journalctl output for the docker unit on failure and
include it in the returned error, so the actual systemd failure reason
is visible without needing rig changes or manual SSH access. Ignore a
failure to fetch the diagnostic itself, since that must not mask the
original error.

Written by AI: claude-sonnet-5

Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>
…gle-node test

MCR package availability on repos.mirantis.com, checked 2026-11-03:

  rhel/10 (also serves rocky10)  stable-29.2 ... stable-29.8.0
  ubuntu resolute                stable-29.8.0 and test-29.8.0 only
  sles/16                        no directory; sles/ has 12, 12.3, 15

Changes following from that:

  - WrkSles16 removed from TestCuttingEdgeCluster and the Sles16 entry
    removed from test/platforms.go, its only consumer. A sles16 worker
    cannot install MCR at all, so it can only ever fail the run.
  - MCRChannel stable-29.6.1 -> stable-29.8.0. 29.6.1 does not exist for
    resolute, so the previous value could not install on ubuntu26.
  - MKEVersion 3.9.6 -> 3.9.7, the pairing with MCR 29.8.0 exercised live
    in PRODENG-4062.

main's TestCuttingEdgeCluster (added by #661) is removed and replaced by
this matrix. That test provisioned one ubuntu_26.04 manager against
repos-internal.mirantis.com test-29 because resolute packages existed
nowhere else on 2026-09-29. repos.mirantis.com now carries them, so the
public-repo matrix covers ubuntu26 in both manager and worker roles. The
cost is that no smoke test exercises the repos-internal path any more;
MCRRepoURL in smokeConfig is retained for the next test that needs it.

The provision-aws module constraint is the exact pin 0.1.8 from main
rather than this branch's >= 0.1.7: 0.2.0-0.2.2 fail at plan time.

Not run: the smoke job itself. The RHEL10/Rocky10 MCR-daemon-start gap
(missing xt_* kernel modules, nftables vs swarm) is unresolved and may
still fail this matrix.

Written by AI: claude-opus-5

Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>
@james-nesbitt
james-nesbitt force-pushed the smoke/cutting-edge-test branch from a9592df to 514d679 Compare November 3, 2026 06:44
@james-nesbitt james-nesbitt changed the title smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu26/sles16) smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu26) Nov 3, 2026
@james-nesbitt
james-nesbitt marked this pull request as ready for review November 5, 2026 12:44
TestCuttingEdgeCluster failed on all four EL10 nodes at Install MCR:
dockerd refused to start because its bridge driver could not append
"-m addrtype --dst-type LOCAL" to nat-PREROUTING. xt_addrtype lives in
kernel-modules-extra, which EL10 minimal/cloud images do not carry;
iptables 1.8.11-15.el10 stopped recommending it unconditionally and
docker-ee does not Require it (docker/docker-ce-packaging#1340).

Measured on EC2 in us-east-1, against the AMIs provision-aws v0.3.0
resolves for rhel_10 and rocky_10:

  control, no fix   start rc=1, is-active "activating",
                    "Cannot connect to the Docker daemon"
  rhel_10 + fix     active, docker info -> 29.8.0m1 overlayfs
  rocky_10 + fix    active, docker info -> 29.8.0m1 overlayfs

The control is the more interesting half: installing docker-ee there
pulled kernel-core, kernel-modules and kernel-modules-extra for
6.12.0-211.74.1 onto a box running 6.12.0-211.16.1, so the module was
still not loadable. That is why the install pins
kernel-modules-extra-$(uname -r) instead of leaving it to the resolver.

The nftables firewall backend is not an alternative: Docker 29 refuses
to enable it in Swarm mode, which MKE requires.

The payload needs a shebang. provision-aws passes user_data to the
launch template verbatim -- base64encode, no wrapping -- so cloud-init
runs only what it recognises. Measured on the control instance, which
carries the same bare "sudo firewall-cmd ..." string every other Linux
platform in this file uses: cloud-init logged "Unhandled non-multipart"
and /var/lib/cloud/instance/scripts/ was empty. Those firewall rules
have never executed, on any platform, in any smoke run.

No firewall rules in the new payload either: firewalld is not installed
on either EL10 AMI, firewall-cmd is not on PATH, and node-to-node
traffic is already carried by the security group's self rule.

On the race between cloud-init's dnf and launchpad's own: a second dnf
waits ("Waiting for process with pid ... to finish") rather than
failing, and sshd becomes reachable 27s (rhel) to 59s (rocky) before
cloud-init finishes, against roughly five minutes between instance
creation and Install MCR in CI.

Also aligns the smoke-cutting-edge job with the six other suites in
smoke-tests.yaml: arc-runner-set-mirantis-public, SHA-pinned checkout
and setup-terraform, persist-credentials off, terraform_wrapper off,
and the Set up Go step it lacked. Its Makefile and workflow comments
said the job was merged red pending MCR packages for rhel/10 and
rocky/10; those packages exist and install cleanly, so the comments are
corrected rather than kept.

Written by AI: claude-opus-5

Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>
The previous commit's user_data was a multi-line Go raw string. terratest
renders Options.Vars into `-var 'nodegroups={...}'` without escaping, so
terraform received a quoted HCL string split over several lines and
TestCuttingEdgeCluster died in 17s with nineteen parse errors --
"Invalid multi-line string" and "Invalid character" against
<value for var.nodegroups> -- before provisioning anything.

Written on one line with literal \n escapes instead. HCL unescapes them,
so the instance still gets a normal multi-line script. Verified by
rendering Platforms["Rhel10"].GetWorker() through TerraformOptions into
an offline fixture and reading the value back out of a terraform output:
shebang on its own line, one command per line.

The double quotes around kernel-modules-extra-$(uname -r) are gone for
the same reason: they would terminate the HCL string. The argument is a
single shell word, so the quoting was never load-bearing.

Written by AI: claude-opus-5

Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>
@james-nesbitt
james-nesbitt merged commit cb8ef09 into main Nov 11, 2026
16 checks passed
@james-nesbitt
james-nesbitt deleted the smoke/cutting-edge-test branch November 11, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

smoke-cutting-edge Trigger cutting-edge smoke test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant