[cxymds]

The legacy upgrade harness only checked the AWS CLI exit code while trimming seeds. Per-object Errors in an HTTP 200 response were ignored, so the final report only said trim left 182 objects (wanted exactly 2). Preparation failures also omitted combinations that never ran, while temporary Warp output and product file logs were lost during cleanup, making the actual deletion errors impossible to trace.

This change makes preparation failures traceable and records every selected case in the upgrade matrix.

Changes

• Write exactly two fixed 100 MiB seed objects, avoiding the generation of nearly 18 GiB of data before trimming. Keep the small-object Warp seed.
• Explicitly validate delete-payload generation, CLI exit status, and per-object errors, stopping at the first failed batch. For successful responses, record request keys, HTTP status, request IDs, error codes, and messages. Preserve the S3 error identity for top-level 503 responses.
• Read the seeds back before the upgrade and record the complete key list, byte counts, and SHA-256 digests. After the upgrade, compare the inventory and contents, while retaining the HEAD/GET ETag checks.
• Use the same set of 18 cases for every topology. When preparation fails or execution is interrupted, fill in NOT_RUN for the remaining cases and record PREPARE, UPGRADE, and VERIFY failures separately, along with diagnostic and cleanup outcomes. Clean up only after archiving is complete, and stop subsequent combinations when recovery fails.
• Preserve Markdown, structured JSON, per-case diagnostics, Warp output, node file logs, and journal logs. Keep signed requests, raw CLI debug output, credentials, and golden.tsv in a private temporary directory. Sanitize evidence before upload and delete the private directory on exit.
• Generate the Actions report from the synchronized events.log, count PASS and incomplete cases correctly, and upload the secure evidence directory. Do not automatically close historical Issues when results contain both PASS and NOT_RUN.
• Atomically save the selected matrix before dependency checks. Verify every group’s complete case list, terminal states, diagnostics, and cleanup evidence. Truncated logs and duplicate results must fail validation. Preserve preparation-failure and not-run events even when Python is unavailable.
• Use RUSTFS_NODES consistently for node isolation, the harness, and outer cleanup. Record PENDING before touching the environment. Release the shared environment only after harness-node and Vault cleanup verification, plus outer cleanup, all succeed. Redaction covers unknown quoted credentials and generic token fields. An interrupted diagnostic collection no longer changes the result to success.
• Make dry-run a pure preview and add dedicated Upgrade Script Tests CI and Chinese documentation.

All 54 assertions in the previous three local combinations passed. The first Vault group then failed while preparing seeds on the old version, so the remaining 54 assertions did not run. The run never reached the Vault upgrade and therefore cannot establish a new-version Vault compatibility regression. New tests verify that this scenario is reported as 54 PASS / 54 NOT_RUN and retains PREPARE_FAILED.

Validation

• python3 -B -m unittest -v test_rustfs_upgrade.py test_sensitive_diagnostics.py test_issue_manager.py: 29 tests passed.
• cd scripts && python3 -B -m unittest -v test_functional_dispatch.py test_prepare_functional_package.py: 20 tests passed.
• Real AWS CLI 1.45.19 with a loopback S3 XML fixture: quiet success, HTTP 200 responses with all or partial per-object errors, top-level 503, and jq-generation failure. 2,503 deletion keys were split into batches of 1,000 / 1,000 / 503, and later batches stopped on error.
• Covered 108 PASS, 54 PASS / 54 NOT_RUN, upgrade failure, assertion failure, TERM interruption, diagnostic failure, cleanup failure, keep-env, evidence-directory conflicts, and credential redaction.
• Shell syntax checks for all repository scripts, Python syntax checks, and git diff --check passed. actionlint also passed, ignoring the repository’s pre-existing smoke-testing custom-runner-label warning.

Validation used local protocol fixtures and simulated environment boundaries. No VM upgrade or live retest was performed. Product-environment validation must be scheduled after merge.

[Copilot]

🟡 Changes recommended

Unresolved critical issues can expose credentials, allow truncated reports to pass, and release a shared fleet after incomplete cleanup.

7 open issues

• Keep fleet quarantined until all touched resources are verified clean · New
• Require complete records for every planned matrix group · New
• Redact quoted credential keys and generic token fields · New
• Mark diagnostics complete only after successful collection · New
• Initialize the test plan and evidence before preflight checks · New
• Prevent dry-run log setup from truncating existing files · New
• Classify failures with no failing cases as harness failures · New

What changed in this PR

This PR strengthens the RustFS upgrade harness’s failure reporting and evidence retention, helping distinguish preparation failures from upgrade regressions.

Changes:

• Bounds large-object seeding, validates deletion responses, and compares seed inventories and content across upgrades.
• Records matrix execution, diagnostics, and cleanup outcomes in Markdown and JSON reports.
• Updates Actions reporting and issue classification, and adds regression tests and evidence documentation.

File	Description
test_rustfs_upgrade.py	Adds lifecycle, deletion-protocol, integrity, and redaction regression tests.
scripts/​upgrade_evidence.py	Adds evidence sanitization, deletion validation, and seed-integrity helpers.
scripts/​issue_manager.py	Recognizes NOT_RUN results and incomplete upgrade matrices.
scripts/​functional_case_report.py	Adds phase outcomes, cleanup details, and structured JSON reports.
rustfs-upgrade-test.sh	Adds bounded seeding, content verification, evidence collection, and matrix finalization.
docs/​upgrade-test-evidence.md	Documents evidence artifacts, statuses, and validation limits.
.github/​workflows/​upgrade-script-tests.yml	Adds dedicated upgrade-script regression CI.
.github/​workflows/​rustfs-upgrade-test.yml	Uses synchronous events, counts incomplete cases, and uploads evidence.

_{Review effort: Balanced}