Update e2e backup tests for backup-rework CLI changes - #1255
Open
RaunakJalan wants to merge 188 commits into
Open
Update e2e backup tests for backup-rework CLI changes#1255RaunakJalan wants to merge 188 commits into
RaunakJalan wants to merge 188 commits into
Conversation
RaunakJalan
commented
Aug 19, 2026
RaunakJalan
commented
Collaborator
- backup import: positional arg → --from-file flag (3 locations)
- backup restore: remove --cluster-id flag (no longer exists)
Lvol migration test fixes
… clone's parent The all-zeros DR fail-over, root-caused at the RPC level this time. Fail-over (replicate_lvol_on_target_cluster) left do_replicate=True and the pending FN_SNAPSHOT_REPLICATION tasks queued on the source. The "dead" source cluster auto-recovers within minutes, the tasks then complete, and each completion runs _prune_internal_snapshots for the source volume. Retention keeps only the newest replicated internal snapshot and deletes the TARGET copies of older ones — including the snapshot the fail-over volume was just cloned from. That delete reaches SPDK as bdev_lvol_delete(sync=False) (validated in the spdk_proxy logs: prune line to delete RPC in ~26ms, same thread), and sync=False frees the blocks immediately, so every DB-level guard downstream fires after the data is gone. Observed timeline (2026-08-11 lab): fail-over 21:22:24-37 with 4 replication tasks still `running`; prune + delete RPC 21:23:54; the controller's soft-delete guard fired the same second and the monitor guard at 21:24:09 — both too late. The volume reads zeros from ~90s after a successful fail-over: no filesystem, md5 mismatch, while every status field says online. Case-1-style migration cutover is unaffected because it retires the source volume through the normal path instead of leaving its replication running. Two-part fix, both ahead of the RPC: * replicate_lvol_on_target_cluster now calls replication_stop() before recording the relationship: do_replicate=False and the pending replication tasks cancelled. A failed-over volume no longer lives on the source, so there is nothing left to replicate; any later source delta is by definition past the RPO the fail-over accepted. * _prune_internal_snapshots skips (and logs) a target snapshot that a live volume is cloned from, keeping the paired source copy too — so no other caller can ever issue the fatal delete. An in_deletion clone deliberately does not pin the snapshot, so retention cannot deadlock behind a dying clone. The earlier hardening commits (fail-over point selection, clone-under-lock, monitor live-clone guard) remain valid but each acted after the sync=False delete had already freed the blocks; this addresses the mechanism itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to 7250fa0, hardening the dominant failure path of the 2026-08-11 soak: lvol_monitor's repair called add_ns for an already-bound namespace, SPDK rejected it with -32602 "Invalid parameters" (nsid taken), and add_lvol_thread returned on the error — before the listener loop — so the volume lost a PATH rather than just failing a redundant add, and the repair re-failed identically on every monitor cycle (20 hits across the two recovered nodes). The duplicate case is already recognised upstream since 7250fa0 (the add_ns idempotency probe matches by UUID), so reaching the error branch now means a real failure or a probe miss. Either way what matters for the client is whether the namespace is on the subsystem NOW: re-read it (bounded poll) and continue to listener setup when present; give up only when the namespace is genuinely absent. The empty-subsystem guard below stays fail-closed. Soak: SPDK verification is now a heal GATE rather than a fixed-window check. Redundant-path re-add after an outage runs on the health-check / reconcile cadence and legitimately takes minutes (measured 169s for full hublvol path convergence after a 30s all-node single-NIC outage; run 2 aborted spuriously because its 75s window undershot exactly that). The gate blocks the next iteration until every path, policy and listener has converged — a new outage on top of still-degraded redundancy would test an unplanned scenario — logs the healing time per phase as a measurement, and fails only on --path-heal-timeout (default 900s). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Root cause of the all-zeros DR fail-over, fourth and final layer. The fork addresses replicated cluster writes as (redirect_map_id << 48) | offset — the top 16 bits of the LBA carry the RECEIVING volume's map id so the target distrib routes the data into its map, and the end-of-transfer signal is only recognised as a control message when tagged (lib/lvol/lvol.c, R26.3 == fork-main). redirect_map_id comes from bdev_lvol_transfer's lvol_id parameter. snapshot_replication started every transfer WITHOUT lvol_id, defaulting to 0: writes went untagged, the target blob allocated clusters (used_size looked plausible) but no readable data ever landed in the map, and the completion signal was written as plain data at LBA 0. Every snapshot replicated this way is empty; a clone of it — the DR fail-over volume, or even a plain `snapshot clone` (verified live) — reads zeros with no filesystem while all metadata (chain, base_snapshot, allocated clusters) looks correct. The migration runner has always passed lvol_id=tgt_map_id to the same RPC, which is why planned cutover (case 1) worked while fail-over (case 2) failed 4/4. Fetch the receiving volume's map_id from the target node at transfer start and pass it as lvol_id; if it cannot be read, suspend and retry the task instead of launching a transfer that cannot land. The transfer→add_clone→convert wiring itself was verified correct — it just faithfully froze an empty volume. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add kubectl install to Dockerfile with multi-arch support * Fix kubectl not found by installing to /usr/local/bin * replace kubectl binary with kubernetes Python client in collect_logs.py and add missing task-runner services * fix collect_logs: use since_seconds instead of since_time and replace kubectl with kubernetes Python client
…ap_id" This reverts commit e5b1cf6.
…ap id Replication transfers were aimed at the receiving volume's own namespace with no lvol_id. Per the fork's transfer contract (confirmed by the data-plane design): bulk transfers must go over a HUBLVOL, and the receiving volume's map id must ride in each write's LBA (top 16 bits, lvol_map.lvol[offset >> 48]) — the demux only exists on a hublvol namespace. The migration runner has always done hub+map_id and works; replication did neither half. The previous attempt (e5b1cf6, reverted) added the map id but kept the volume-namespace gateway: tagged writes at an endpoint with no demux, which failed transfers outright. Attach the target node's transfer hublvol on the source (reusing ensure_hub_attached, shared+persistent across cycles like the migration hub), fetch the receiving volume's map_id from the target, and start the transfer hub+map_id. Suspend-and-retry the task if either is unavailable. The per-volume controller detaches at finish go away — there is no per-volume controller any more. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sters Reverts the leader-leg removal from ae46797 (its part 1, the inline non-leader sync legs, stays). That commit's premise — "the leader's async delete already removed the blob and unregistered the bdev" — is false: - bdev_lvol_delete sync=False ends at bs_delete_blob_finish_async -> blob_clear_clusters_async: data clusters cleared, in-memory clone-list entries stripped, blob metadata and bdev registration left in place. The delete-status "done" (deletion_status=2) means the unmap finished, nothing more. - The leader's sync=True delete (_vbdev_lvol_destroy is_sync=true) is the only operation that unregisters the bdev and deletes the blob md. SPDK admits it exactly once the async pass reports done. Evidence: upgrade run 20260812 (test_major_upgrade-20260812-170049). All 24 lvols of the teardown wave were "deleted" (records removed, subsystems gone, non-leader registrations gone) while EVERY leader kept every blob and bdev — end-of-run dumps show all 4 LVSes with their complete object sets. The follow-up snapshot delete failed EBUSY -16 ("Cannot remove snapshot because it is open", blobstore.c:11451) because the snapshot's two children were still alive on the leader (snapshot open_ref=3), and the soft-delete gate could not protect it: the children's DB records were already gone, so the snapshot looked clone-free to the control plane. Why the 20260807 evidence misled: - The 4361 "Clone entry not found" errors on the leader are BENIGN: blob_get_snapshot_and_clone_entries only logs when the async pass has already removed the in-memory entry; the sync delete proceeds and does the real work. Noise, not harm. - The "0 leftovers without the leader leg" check sampled the 162 create-rollback objects, which do not represent the regular delete path (failed creates often never registered a leader bdev, and the end dump postdates lvstore teardown). Changes: - lvol_monitor.process_lvol_delete_finish: leader sync delete restored (under the leader's lvstore lock), comment corrected. - snapshot_monitor.process_snap_delete_finish: leader sync delete restored (with special_delete passthrough), comment corrected. - snapshot_controller._rollback_snapshot_bdev: leader sync delete added after the bounded completion poll (this path NEVER had one — same leak for rolled-back snapshot bdevs), invariant docstring corrected. - lvol_controller._delete_lvol_from_all_nodes: comment corrected (the inline non-leader legs and fail-closed poll are unchanged). The "Clone entry not found" storm returns with the leader leg; it is understood and harmless. Proper long-term fix is SPDK-side: either the async delete carries through to metadata removal (then the leader leg and the noise both go away) or the post-async sync walk skips entries the async pass already stripped. Unit tier: test_async_delete_poll passes (9/9); ruff and mypy clean. pytest-style tests need Linux (SIGALRM conftest) and run in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… snapshot
On the target, every replicated snapshot is chained onto its predecessor
(add_clone at receive time): the newest snapshot reads THROUGH the older ones.
Retention ("keep only the last replicated internal snapshot") deleted those
predecessors outright, destroying the shared clusters the survivor depends on.
Observed on the 2026-08-13 lab with transfers finally landing real data
(hub+map_id): a clone of the newest target snapshot had a valid XFS superblock
(its own delta) but an empty tree — everything underneath was gone.
Before pruning a target snapshot, decouple every child SPDK reports
(bdev_lvol_decouple_parent copies the parent's allocated clusters into the
child) on the primary and its online secondary. If any decouple fails, keep
the snapshot pair and retry next pass rather than deleting a parent something
still reads through. New rpc_client wrapper for bdev_lvol_decouple_parent
(fork RPC, {"name": <child>}).
Note tests/unit/test_rollback_sync_delete_all_peers.py fails on current main
independently of this change (5f4315e restored the leader's sync delete;
the test still asserts the pre-restore behaviour).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#1245) * fix(alerting): use ratio query for cluster capacity alerts to preserve cluster label in fired alerts * fix(test): update rollback sync delete assertions for leader node --------- Co-authored-by: hamdykhader <hamdy.khader@gmail.com>
The target_node_id was used authoritatively, not as a hint. It silently overrode the intended cluster. Because the cluster-id is defined by the pool, the cluster-id flag is dropped completely.
The introduced API fixes a number of issues with the API v1 metrics, so
it's not compatible with it:
- Dropped size_util / size_prov_util: v1's were lossy (int(used/total*100)),
not wrong. Adequate for a threshold alert, coarse for graphs. Dividing the
byte gauges is exact.
- Status as {status="degraded"} label instead of numeric code. The label
just doesn't hardcode magic numbers and survives map changes.
- health_check absent instead of NaN: Same information, no float sentinel
for consumers to special-case.
- Dropped date, record_duration, record_start_time, record_end_time:
Metadata, the record_* trio was never populated by any collector.
- simplyblock_ namespace: Avoids collision with the node and foundationdb
jobs and allows coexistence with v1 metrics.
- Added pool_name to volume series: Preserves the human-readable $pool
dropdown that v1 got for free from its (otherwise buggy) label.
…a target snapshot" This reverts commit acc8e0c.
…the hub session
Two receive-side steps the migration runner has always performed and
replication never did, both mandatory per the fork's transfer contract:
* bdev_lvol_set_migration_flag on the receiving lvol BEFORE the transfer.
The flag (a) stamps the receiving blob's writes special_io=1, which the raid
layer encodes into the LBA and the distrib stack uses for receive-mode
placement (blobstore.c bs_batch_open_s/special_io), and (b) arms the hub
write handler's detection of the end-of-transfer signal — hublvol_write only
routes a 1-page write at page 0 into process_migration_write_request ("add
this lvol as clone ... mark migrate process completed") when migration_flag
is set (vbdev_lvol.c:1414). Without the flag the payload lands as ordinary
client IO and the completion signal is written ONTO LBA 0 as data: the
receive never finalises and every clone of the converted snapshot reads
zeros — reproduced with a 3-step repro (replicate one snapshot, clone the
target copy, read) with no failure injection at all. add_clone/convert clear
the flag, closing the lifecycle.
* Detach the transfer hub on the source when the cycle finishes (success and
abort paths). The connect -> transfer -> convert -> disconnect cycle is part
of the contract; the next cycle re-attaches via ensure_hub_attached.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sync c04cfcb adapted this test to the restored leader sync leg (5f4315e) with a bare call_count == 2, which would also pass if the leader received two async deletes and never its sync leg — the exact leak of upgrade run 20260812. Assert the full call list instead: phase-1 async first, then sync=True. Also correct the module docstring, which still stated the falsified "never on the leader" protocol. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ive path The flag drives the distrib-level special_io machinery of INTRA-cluster migration (copy-on-write context within one cluster's maps); it does not apply to a cross-cluster receive, where the source cluster's map/COW context does not exist on the target. It was added in 7ea16df by copying the intra-cluster migration recipe; reverting that half. The hub connect->transfer->disconnect session lifecycle from the same commit stays. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
replication_commit took one fire-and-forget internal snapshot, selected a cutover base that was 1-2 replication intervals old, and enqueued the final task — which froze immediately, WITHOUT waiting for anything to replicate. The writes between the last replicated snapshot and the pre-commit snapshot were covered neither by the final step's delta (top blob only = writes after the pre-commit snapshot) nor by anything on the target: every cutover silently lost up to ~2 intervals of data. Invisible to the harness because fio's sequential sweep rewrites its whole working set and verify_backlog only checks recent writes. The cutover now shrinks the delta iteratively before freezing: snapshot #1 (at commit) -> wait until replicated AND converted on the target -> IMMEDIATELY snapshot #2 (delta = just the wait window) -> wait again -> IMMEDIATELY build the target clone on that last replicated snapshot and run the freeze + ANA flip. replication_commit is now thin (validate, shrink snapshot #1, enqueue); the final-task runner owns the shrink state machine (bounded by REPL_CUTOVER_SHRINK_TIMEOUT_SEC, waiting does not burn task retries) and the clone/map-id/replication-record preparation, so the base is always the freshly replicated shrink snapshot. Old-style tasks with tgt_* params still run unchanged (skip shrink + prepare). Harness: case 1 now writes a baseline fio never touches and md5-verifies it through the cutover volume after a remount — the ONLY assertion that exercises the replicated snapshot history (fio's own data always arrives via the final step); target paths are connected inside the cutover wait loop as soon as the runner creates the target volume, keeping multipath ahead of the ANA flip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cutover enabled the target paths and only then set the source paths inaccessible. Two windows lose writes: (a) after the final delta is taken and the freeze lifts, the source is still ANA-optimized until the flip reaches it — a client write landing there is newer than the delta of record and silently lost; (b) during the flip itself both source and target are optimized — dual-writable. The intra-cluster migration runner already fences source replicas pre-freeze for exactly this reason; the cross-cluster cutover did not. New order: ALL source paths -> inaccessible (peers first, primary last), THEN freeze + final delta (nothing can land on the source by any means; the delta is definitively final), THEN target primary -> optimized, peers -> non_optimized. Client IO queues during the all-dark window (NVMe multipath semantics), bounded by freeze + residual delta — seconds, thanks to the delta-shrink rounds. If the freeze FAILS after the source was fenced, the source paths are restored (primary optimized, peers non_optimized): nothing moved, the source is still the authoritative copy, and a failed cutover attempt must not leave the volume dark. flip_ana_failover is split into fence_source_paths / enable_target_paths / restore_source_paths; ordering asserted in tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eive and convert
Two non-leader hazards in the fork's data plane:
* the transfer hub REJECTS receive IO on a non-leader ("receive io for hublvol
in nonleader mode") — a transfer started against a non-leader target fails
outright;
* bdev_lvol_convert on a non-leader DEGRADES SILENTLY: the non-leader branch
marks the blob CLEAN and replies success without persisting anything, so the
"snapshot" looks converted while its metadata never reached the journal —
return-value checks cannot catch it.
Gate both: replication verifies the receiving node holds LVS leadership before
starting a transfer and again before add_clone/convert (suspend-and-retry
otherwise); the lvol-migration and batch-migration runners verify leadership
before their converts (fail-and-retry). Shared probe: lvol_controller.
is_node_leader.
Also: setup_repl_test_2clusters ssh_exec gets the reconnect-retry the test
harness already had — a 10054 reset killed a second deployment at the finish
line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ive user
The hub is ONE session per target node. The per-cycle detach added with the
connect->transfer->disconnect contract ripped the shared qpair out from under
the other volumes' in-flight transfers: mass hub IO failures, LVS leadership
churn on the target ("receive io for hublvol in nonleader mode" storms,
observed live 2026-08-13), transfers landing nothing, converts silently
no-oping on flapped leadership. Detach now happens only when no other RUNNING
snapshot-replication task is transferring into the same target node — the
refcount discipline the migration runner's hub_manager embodies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n the remote cluster Replicated snapshots on the target were standalone blobs (clone=false, base=null in SPDK): the finish path only chained when snap_ref_id was set, but internal replication snapshots never populate it, and even then the lookup matched the remote node against the SOURCE snapshot's instances, which are source-cluster nodes. bdev_lvol_add_clone was never attempted (chain_attempts=0 across entire runs). A fail-over clone therefore read only the last delta and zeros elsewhere, and retention's delete could not swap-merge segments into a successor — the all-zeros DR fail-over. Resolve the predecessor by lvol + age (newest older snapshot with a completed remote copy, snap_ref_id still wins when set), chain to the remote copy's bdev, and fail-and-retry instead of silently finalizing an unchained snapshot when a predecessor exists but cannot be resolved. Harness: mark read-only DB/status pollers replayable so a mid-exec socket reset (WinError 10054) replays the query instead of aborting the case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…etry backoff
The monitor ran one serial pass per cycle: take internal snapshots, then
process EVERY in_deletion record with a full DB fetch and RPCs. Two
consequences observed on the 2026-08-14 lab (1298 records in in_deletion):
- a delete that cannot complete is retried every cycle forever, so the
in_deletion set only grows and each cycle costs more;
- internal-snapshot creation shares that pass, so it never came around
again — five replicated volumes went an hour with zero snapshots, which
looks like 'replication stopped' but is starvation.
Creation now runs for all clusters before any delete work. Deletes are
processed by a bounded pool keyed on the owning volume, so a chain
(clone -> snapshot -> parent) still advances in order on one worker while
different volumes proceed concurrently. Failing deletes get exponential
backoff (5s..5min) instead of a slot every cycle.
Concurrency contract: per-object create+register stays serialized by
object_mutation_lock; every synchronous single-node RPC is mutually
exclusive per node via lvstore_op_lock, which the phase-2 sync deletes now
take individually. That is the creators' key space ('<lvs>@<node8>') — a
whole-lvstore key would be a different key and exclude nothing. DB
finalize stays outside the lock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…discovery - Store checksums alongside each backup_id as (bk_id, checksums) tuple so restores verify against snapshot-time data, not FIO-modified source - Scope clone ns-rescan to source controller instead of all controllers to avoid picking up unrelated devices - Add nvme ns-rescan to _wait_for_new_ns_device polling loop so kernel discovers newly-created namespace children
…kups - Delete restore lvols after verification to prevent max_lvol exhaustion (was accumulating 800+ restore lvols, hitting 50/node subsystem limit) - Use clone nsid to determine connection strategy: nsid=1 does nvme connect (own subsystem), nsid>1 does ns-rescan on live controllers (shared parent subsystem) - Prefer older backup_ids for restores to avoid "Incomplete backups in chain" errors from freshly-created backups still uploading to S3 - Capture device baseline before add_clone, matching existing clone test patterns
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| if rst_mount: | ||
| self._unmount_and_disconnect( | ||
| self.fio_node, rst_mount, rst_id or "") | ||
| except Exception: |
| pass | ||
| try: | ||
| self._delete_lvol(rst_name, skip_error=True) | ||
| except Exception: |
Align MAX_ENTITY_COUNT and NUM_SUBSYSTEMS with the 6k class name (was misconfigured at 12000/60, now 6000/30).
Clone phase now runs all 10 clones concurrently in K8s mode using threads (Docker mode stays sequential due to device baseline races). Reduces clone phase from ~4h to ~25min on K8s. Fix utility pod SELinux relabel failure on OpenShift by setting securityContext.seLinuxOptions.type=spc_t on checksum pods. Without this, OpenShift auto-relabels volume mounts causing "bad message" errors on all clone restore verifications.
…/clone/namespace combos No deliberate concurrency anywhere in the test — many backups and restores are dispatched one after another to reach a high total operation count without engineering a race, to check whether ordinary bulk backup activity (not just an artificial race) can reproduce backup/restore failures.
Registers TestBackupHighVolumeCombosSequential under get_backup_stress_tests() instead of the routine get_backup_tests() suite - at ~6-7 hours in k8s mode it belongs with BackupStressComprehensive's budget, not the standard backup suite. Also attaches a versions=3 retention policy for the whole run so the service's merge logic (bdev_lvol_s3_merge) has to keep trimming each lvol's growing backup chain, instead of letting ~7 rounds of backups pile up unbounded. Logs resulting chain depth per lvol as a merge-at-scale sanity check.
With the versions=3 policy now attached, an absorbed backup doesn't get deleted - it transitions to status='merged' (Backup.STATUS_MERGED) and is no longer restorable. The restore loop was trusting its own locally-tracked backup_ids[-1], which can go stale once a merge absorbs it. Now re-resolves against a fresh, status-filtered live backup list each round and walks backward through the tracked ids to find the newest one still actually restorable, skipping gracefully if none survive.
Multiple GitHub Actions runners share one VM, and every "Start MinIO trace logging" step across 10 workflows was using a shared /tmp/mc binary path plus a pattern-matching pkill -f "mc admin trace" during setup and teardown. One job's install/cleanup could delete, overwrite, or kill another concurrently-running job's mc binary/trace process, causing spurious "installing mc" failures whenever two backup-related jobs landed on the same VM at once. Scopes every mc binary path, alias name, and log path to $GITHUB_RUN_ID, and replaces the global pkill/glob cleanup with PID- and path-scoped cleanup of only this job's own resources.
_connect_and_mount() is the wrong tool for a namespace child: it's not an independent NVMe target, it's a new namespace on the parent's already-connected subsystem. Two compounding bugs caused "No new block device after connecting" for the second child: 1. Reusing _connect_and_mount() issues a redundant nvme connect and does a naive before/after device diff, racing against how fast the kernel discovers a namespace added after the controller connected. 2. _unmount_and_disconnect(child_id) was a silent no-op: a namespace child has no NQN identity of its own (it lives under the parent's), so disconnect_lvol()'s `nvme list-subsys | grep <child_id>` finds nothing and the shared controller is never actually torn down - which is exactly why the next child's connect attempt found it still "already connected". Fixed by mirroring continuous_backup_stress.py's proven approach: connect once for the parent, keep the controller alive, and for each child snapshot the namespace-device list before add_lvol() and poll with `nvme ns-rescan` until the new device appears - no redundant connect calls, and the shared controller is disconnected exactly once at the end via the parent's own id. RCA: test_ns_child_connect_race_rca_20260902.md
…pgrade test One-off validation step for R26.3's `sbctl cluster switch-write-protection` (activates v2 distrib write protection cluster-wide after upgrading from a pre-v2 release). Runs after the rolling upgrade completes but before the test waits for FIO to finish, so FIO keeps running as I/O load through the switch and the second round of storage-node restarts. Marked TEMP STEP / END TEMP STEP in the code for easy removal after this validation run.
…tests storage-node configure's required flag (--max-lvol, then --max-subsys, then neither) has changed three times across sbcli versions with no reliable way to infer which era a base branch belongs to from its name. Rather than change bootstrap-cluster.sh's default behavior (shared by many parallel pipelines), this adds two opt-in checkboxes scoped to just the upgrade-test workflows: - BOOTSTRAP_CONFIGURE_MAX_LVOL / BOOTSTRAP_CONFIGURE_MAX_SUBSYS route the max-subsys value to storage-node configure (via --extra-sn-args) instead of cluster create, for old/mid-era base versions. Default (both unchecked) reproduces today's behavior exactly. - Also collapses the 5 separate BOOTSTRAP_MAX_SUBSYS/DATA_CHUNKS/ PARITY_CHUNKS/JOURNAL_PARTITION/HA_JM_COUNT inputs into one comma-separated BOOTSTRAP_PARAMS input, parsed back out by a new "Parse BOOTSTRAP_PARAMS" step. - Adds SIMPLYBLOCK_DEPLOY_BRANCH input so the simplyBlockDeploy bootstrap-cluster.sh version can be pinned per-run (default: main).
RaunakJalan
force-pushed
the
fix/backup-rework-e2e
branch
2 times, most recently
from
September 3, 2026 07:43
de01b96 to
faa8b7e
Compare
# Conflicts: # .github/workflows/e2e-bootstrap.yml # .github/workflows/k8s-native-cross-cluster-restore.yaml # .github/workflows/k8s-native-e2e-add-node.yaml # .github/workflows/k8s-native-e2e-node-migration.yaml # .github/workflows/k8s-native-e2e.yaml # .github/workflows/monitoring-suite-docker.yaml # .github/workflows/stress-run-bootstrap.yml # .github/workflows/upgrade-bootstrap-single.yml # .github/workflows/upgrade-bootstrap.yml # AGENTS.md # docs/replication-policies-design.md # e2e/e2e_tests/backup/test_backup_restore.py # e2e/e2e_tests/upgrade_tests/k8s_major_upgrade.py # e2e/stress_test/mass_create_delete_stress.py # scripts/aws_dual_node_outage_soak_multipath.py # scripts/deploy_gate_and_soak.py # scripts/hotfix_repl_lab.py # scripts/setup_perf_test.py # scripts/setup_perf_test1.py # scripts/setup_perf_test_failure_domain.py # scripts/setup_perf_test_multipath.py # scripts/setup_repl_test_2clusters.py # scripts/stage_and_run_repl_cases.py # scripts/start_soak_mp.sh # scripts/test_async_replication.py # simplyblock_cli/cli.py # simplyblock_cli/clibase.py # simplyblock_core/AGENTS.md # simplyblock_core/cluster_ops.py # simplyblock_core/constants.py # simplyblock_core/controllers/health_controller.py # simplyblock_core/controllers/lvol_controller.py # simplyblock_core/controllers/replication_policy_controller.py # simplyblock_core/models/cluster.py # simplyblock_core/models/replication.py # simplyblock_core/models/stats.py # simplyblock_core/rpc_client.py # simplyblock_core/services/capacity_and_stats_collector.py # simplyblock_core/services/health_check_service.py # simplyblock_core/services/lvol_monitor.py # simplyblock_core/services/main_distr_event_collector.py # simplyblock_core/services/snapshot_monitor.py # simplyblock_core/services/snapshot_replication.py # simplyblock_core/storage_node_ops.py # simplyblock_core/test/test_internal_subsystem_exemption.py # simplyblock_core/test/test_replication_backlog_reporting.py # simplyblock_core/test/test_replication_chain_completeness.py # simplyblock_core/test/test_snapshot_instance_handoff.py # simplyblock_core/test/test_snapshot_replication_retention.py # simplyblock_core/test/test_transfer_hub_heal.py # simplyblock_web/api/v2/cluster/__init__.py # simplyblock_web/api/v2/cluster/replication.py # simplyblock_web/api/v2/cluster/storage_pool/volume/__init__.py # simplyblock_web/api/v2/metrics.py # tests/unit/test_cluster_spdk_sizing.py # tests/unit/test_object_limit_per_lvstore.py # tests/unit/test_repair_gating.py # tests/unit/web/api/v2/test_cluster_endpoints.py
| self.fio_node, info["mount"]) | ||
| if current == info["checksums"]: | ||
| verify_ok += 1 | ||
| except Exception: |
| try: | ||
| self._unmount_and_disconnect( | ||
| self.fio_node, info["mount"], info["id"]) | ||
| except Exception: |
| try: | ||
| self._detach_policy( | ||
| info["policy_id"], "lvol", info["id"]) | ||
| except Exception: |
| for name, lid, mnt in lvols: | ||
| try: | ||
| self._unmount_and_disconnect(self.fio_node, mnt, lid) | ||
| except Exception: |
| try: | ||
| if r_mnt: | ||
| self._unmount_and_disconnect(self.fio_node, r_mnt, r_id) | ||
| except Exception: |
| pass | ||
| try: | ||
| self._delete_lvol(rname) | ||
| except Exception: |
Only exists on this branch (main deleted it in favor of the newer replication test files); keeping it desynced this branch's non-e2e test suite from main's.
- Annotate mutable class-level list attributes as ClassVar (COMBOS, _CAPACITY_PATTERNS, _REMOVE_API_SERVICES) - Drop f-prefix from strings with no placeholders in k8s_utils.py - Bind thread_results as a default arg instead of a loop closure in continuous_backup_stress.py to satisfy B023 - Rename ambiguous single-letter `l` loop variables to `lbl`/`ln`
_get_k8s_worker_nqns() (and the identical _get_host_nqn() helper in test_backup_restore.py) called k8s.k8s._exec_kubectl(), but _ensure_k8s_utils() already returns the K8sUtils instance directly. Broke every K8s DHCHAP test that sets up a pool+host at the first kubectl call, before any DHCHAP-specific logic ran.
Cross-checked every step against k8s_major_upgrade.py and a passing CI run (25.10.5 -> R26.3): added the missing prometheus-credential migration step, fixed the Step 7 CR YAML to the current schema, corrected the cert-manager install method, and documented open vs. fixed issues (ext4 FEATURE_C12, CRD lifecycle, SPDK image restart).
_ensure_pool_and_sc() and the DHCHAP host-registration step called add_storage_pool()/add_host_to_pool() on the low-level K8sUtils returned by _ensure_k8s_utils(), but those methods only exist on the outer K8sSbcliUtils (self.sbcli_utils). Same class of bug as the security test fix, found by grepping the pattern across both files.
…nnect The K8s operator enforces DHCHAP purely from StoragePool.spec.dhchap + allowedNodes: it derives each allowed node's NQN itself, labels those nodes, and restricts scheduling via the generated StorageClass/PV nodeAffinity. No host NQN is ever supplied by a client. The previous K8s branches of TestLvolCryptoWithDhchap/TestLvolDhchapBidirectional ported the docker-mode manual `nvme connect --host-nqn` flow as-is, which doesn't reflect how a real workload (or the operator) actually enforces the restriction, and never exercised the negative case since _setup_pool_and_host allowed ALL workers rather than a subset. - create_utility_pod()/create_fio_job() gain a node_name param to hard-pin a pod via spec.nodeName, bypassing the scheduler so a pod can be deliberately forced onto a disallowed node. - New get_pod_events() reads FailedMount events (a Pod event, not a container waiting-state reason get_pod_status_detail could see). - New _k8s_setup_dhchap_pool_subset()/_k8s_verify_pod_scheduling() helpers: pool's allowedNodes is a strict subset of workers, and a pod pinned to an allowed node must mount; pinned to the disallowed one must fail with FailedMount. - TestLvolCryptoWithDhchap, TestLvolDhchapBidirectional, and TestDhchapPodScheduling now use this for their K8s branch (docker branches unchanged — manual connect-string is the correct native approach there). TestDhchapPodScheduling also gains the missing negative case (pod on a disallowed node).
… limit
Every lvol creation in RandomMultiClientMultiFailoverAllNodesTest was
failing with HTTP 400 ("exceeds the hard limit of 50 namespaces per
subsystem"), including the retry, for all 40 lvols, from the first
attempt onward. The test ran its full multi-hour outage cycle against
zero actual volumes. Lowered to 30, comfortably under the limit.
add_storage_pool() blindly reused ANY existing sbcli-visible pool regardless of the caller's dhchap/allowed_nodes request. On a shared test cluster with leftover pools from unrelated tests (e.g. "encryption-pool"), a DHCHAP test asking for allowedNodes-restricted access got handed back a pool with no such restriction at all — so a pod pinned to a deliberately-disallowed node mounted it just fine, because the enforcement was never actually configured on the pool being used. Now: a dhchap/allowed_nodes request only reuses an existing pool if its StoragePool CRD already has that exact dhchap+allowedNodes config; otherwise it creates a dedicated, uniquely-named pool. Also scoped the CRD-existence and Terminating-CRD checks to that dedicated pool's own resource name (previously "any StoragePool CRD exists in the namespace" would skip creating ours and hand back whichever unrelated pool the operator listed first). Callers that don't pass dhchap/allowed_nodes keep the original, proven blind-reuse behavior unchanged.
DHCHAP enforcement reaches the volume only when the StorageClass carries dhchap_node_label — the CSI driver then writes a matching nodeAffinity onto the PV and a non-allowed node fails to mount. Without it the pool reports DHCHAP enabled and any node mounts the volume, which is why the disallowed-node assertion kept failing. Verified by hand on Talos and RHCOS: allowed node mounts, non-allowed node gets "MountVolume.NodeAffinity check failed". - create_storage_class() takes dhchap_node_label; security tests derive simplyblock.io/pool.<ns>.<cluster CR>.<pool> and pass it on both the plain and crypto classes. Paired with Immediate binding and no allowedTopologies, which needs no CSI driver restart. - Skip instead of fail when the host kernel ignores the DHCHAP connect options (no CONFIG_NVME_AUTH — Talos 6.18.24 does not have it, RHCOS 9.6 does), matching the RDMA test's skip pattern. - _k8s_verify_pod_scheduling cleans its pod in a finally and tracks it for teardown; teardown now deletes pods BEFORE PVCs and removes test-created StorageClasses. A leaked pod held pvc-protection on its claim and left three PVCs stuck Terminating for 3+ hours. Pipeline cleanup (cleanup_upgrade_test.sh) missed three things that let stale state survive into later runs: - CR_TYPES had pool./simplyblockpool. but not storagepools., stale since the Pool->StoragePool rename, so pools were never deleted and the next run reused one. - StorageClasses were filtered by name containing "simplyblock", which matched 1 of 53 leaked classes; now selected by provisioner. - Only two node labels were stripped; simplyblock.io/storage-node-uuid.* and simplyblock.io/pool.* embed an id in the key so each deploy adds a new one. Left behind, the CSI driver advertises topology for clusters that no longer exist. Now stripped by prefix.
The operator derives simplyblock.io/pool.<ns>.<cluster>.<pool> from the StoragePool CRD's metadata.name, while self.pool_name holds the backend pool name. Those have matched in every run observed so far, but nothing guarantees it — and a wrong key means the StorageClass silently carries no enforcement, which is the exact failure mode this change set exists to fix. Look the label up on an allowed node (value == "allowed", key suffix == pool name) and only fall back to the computed string, warning loudly when that happens.
Four deviations from the documented K8s DHCHAP flow now carry explicit HACK comments saying what they work around and when to delete them: 1. We bypass the operator-generated StorageClass and hand-roll one with only dhchap_node_label, no allowedTopologies, Immediate binding — because the operator's class provisions nothing until the CSI driver re-registers. 2. spec.nodeName pinning is only sound because of (1); against the documented WaitForFirstConsumer class it is a false negative. 3. DHCHAP pools get a timestamp-suffixed name so a shared leftover pool cannot shadow them, since blind pool reuse is load-bearing for every non-DHCHAP caller. 4. The older security tests still register a host NQN by hand in K8s mode and pass every worker as allowed, contradicting the doc and making a disallowed-node rejection untestable there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.