[cxymds]

Direct Tier suite runs could enable notify without a running MQTT broker. The subscriber helper discarded stderr and reported success after a fixed sleep even when the subscriber had already exited, allowing setup to reach module switches persisted but runtime convergence failed: local notify without proving its dependencies were ready.

Move MQTT preparation into the suite so direct VM runs and CI share the same behavior. Reuse an existing broker, or create a uniquely named Docker broker only when the endpoint is local and has no listener. Require a publish/subscribe round trip and connectivity from the source and target nodes before destructive suite setup. External-only mode and a configurable MQTT port are documented.

Preserve subscriber, broker, and credential-filtered notification startup diagnostics. Cleanup targets only the owned subscriber and container; a persistent container receipt and the shared cleanup function allow CI to recover after cancellation or an earlier cleanup failure. Add 14 regression tests to the existing Tier contract runner.

Related to rustfs/backlog#3148. This addresses the missing MQTT prerequisite and misleading readiness result; the original VM runtime-convergence failure still needs a focused retest.

Validation

• Passed PYTHONDONTWRITEBYTECODE=1 bash test_rustfs_tier.sh, including all 14 new MQTT tests and the existing Tier contract tests.
• Passed bash test_validation_tier.sh and all 7 test_rustfs_tier_report.py tests.
• Passed Bash syntax checks, ShellCheck, workflow YAML parsing, syntax checks for all 21 workflow shell entries, and git diff --check.
• Real Mosquitto validation reproduced the baseline false success with no broker; the updated preflight rejected the missing broker, passed publish/subscribe and subsequent event delivery with a live broker, and preserved that external broker during cleanup.
• actionlint reports the pre-existing unrecognized smoke-testing runner label on both the base and modified workflow. No suppression or gate relaxation was added.
• Original VM scenario and full online Tier matrix were not rerun because another task held the shared lab mutation lock. Automatic Docker broker creation and recovery were covered by command-fixture tests; the live local test used an external Mosquitto process.

[Copilot]

🟡 Changes recommended

Recovery can target the wrong Docker daemon and leave the broker running, while non-atomic probe writes and a mismatched MQTT probe topic make readiness checks unreliable.

3 open issues

• Recovery should use the suite’s direct-first Docker selection · New
• Verify readiness on the configured MQTT event topic · New
• Write probe messages atomically to avoid partial JSON reads · New

What changed in this PR

Moves MQTT dependency preparation into the Tier suite so direct runs and CI check broker readiness before destructive node setup.

Changes:

• Adds broker reuse or managed Docker startup, publish/subscribe readiness checks, and node connectivity checks.
• Adds targeted cleanup, cancellation recovery, and retained startup diagnostics.
• Documents MQTT configuration and integrates 14 regression tests into the Tier contract runner.

File	Description
test_rustfs_tier.sh	Runs the new MQTT regression tests.
test_rustfs_tier_mqtt.py	Tests MQTT readiness, ownership, cleanup, and diagnostics using command fixtures.
rustfs-tier-test.sh	Implements MQTT preflight, managed broker lifecycle, and failure diagnostics.
README.md	Documents dependencies, configuration, and diagnostic artifacts.
.github/​workflows/​rustfs-tier-test.yml	Delegates broker setup to the suite and adds receipt-based recovery.

_{Review effort: Balanced}