diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000000..654fa1db14 --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,119 @@ +# Task-runner rework — handoff + +Continuation notes for the `rework-task-runners` branch. The full design lives +in **`TASK_RUNNER_REWORK_PLAN.md`** (same dir) — read it first; this file is +just the current state + how to continue. + +## Where the work is + +Branch `rework-task-runners`, rebased onto `origin/main` (2026-08-05, commit +`4cb16a20b`). Pre-rebase safety backup: branch `backup/rework-pre-rebase3`. + +``` +fa26b7121 Migrate node add runner… (B3) +f954be2b9 Migrate backup and cluster expand… (B2) +d6cc94478 Migrate lvol sync runner… (B1d) +b436fa79d Migrate replication cutover runner… (B1c) +9c9118761 Migrate JC compression resume runner… (B1b) +d17c20064 Migrate FDB backup runner… (B1a) +3899c2452 Reset task result before each handler attempt +ad4768462 Introduce generic task runner (B0) + +``` + +## Status: the planned scope is complete + +All thirteen runners in scope are on the driver: `fdb_backup`, `jc_comp`, +`replication_final`, `sync_lvol_del`, `backup`, `cluster_expand`, `node_add`, +`restart`, `migration`, `new_dev_migration`, `failed_migration`, `node_removal`, +`port_allow`. + +Nothing outside a runner writes task lifecycle state any more: +`utils.handle_task_result` and `tasks_controller.defer_task_for_expansion` are +gone, and the three cancellation paths commit by CAS. + +**Deferred to a follow-up PR:** `lvol_migration` and `batch_migration` — under +active upstream rewrite (~17 commits in the last window, several still labelled +`TEMP:`). They keep their own loops meanwhile, which is why +`tasks_cluster_status.py` is not the only file left untouched. Migrating them is +the same exercise: their pre-run guard chains become `is_eligible`, their poll +loops `TaskProgress`, and their `_suspend_task(charge_retry=...)` helper maps +directly onto TaskDefer vs TaskRetry. + +## What the driver grew during the migrations + +Beyond what the plan describes, `task_runner_base` gained: + +- **`function_result` is cleared before each handler attempt**, so a task that + fails and later succeeds doesn't finish carrying the stale failure message. +- **`RunnerSpec.on_finish(task)`** — cleanup called after the task reaches + STATUS_DONE and is written, on *every* terminal path (success, `TaskAbort`, + cancel, retry ceiling). Needed because a handler never sees the terminal + paths the driver owns, yet resources it holds must still be released: + `sync_lvol_del` frees the primary's del-sync lock there, `backup` fails/ + un-merges the backup resource there. Both are written to be no-ops when the + handler already finished the resource. +- **All task writes are compare-and-set** (`_commit`), never full-object + `write_to_db`. This is not a refinement — the original driver reproduced the + lost update behind upstream's 2026-07-29 double-restart incident, and held the + stale copy for the whole handler duration. See the plan's "Upstream + reconciliation (2026-08)". +- **One dispatch path**: serialized execution submits to the pool and waits + rather than running inline, so the inflight registry is the single + mutual-exclusion authority. `RunnerSpec.serialize` is a per-task predicate, + because restart picks its mode from live cluster state. +- **`checkpoint(task, **params)`** — persist handler progress mid-handler, for a + destructive step that must not repeat after a crash. Doubles as the + cancellation probe before the next destructive step. +- **`RunnerSpec.on_cycle(cluster)`** — per-cluster upkeep attached to no task + (restart's orphaned-node watchdog). +- **`RunnerSpec.backoff(retry)`** — override the default curve where a runner + has a tuned one (restart's 1-minute lead-in). + +## Deployment entry point + +`simplyblock-task-runner ` (`simplyblock_core/services/task_runners.py`, +declared in `pyproject.toml`) starts a runner by its `RunnerSpec.name` instead of +by a source path inside the image. It resolves the name to a module, checks the +spec's own name matches, and calls `serve(spec)`; `lvol_migration` and +`batch_migration`, which still own their loops, fall back to their module's +`main()` and are listed in `_NOT_ON_DRIVER` until they migrate. + +Switched over: every task runner in `docker-compose-swarm.yml`, and the runner +services `cluster_ops.update_cluster_mgmt` creates on upgrade (which meant +`utils.create_docker_service` / `create_k8s_service` now take a `command` list +rather than a source path). Not switched: the non-runner services, the Helm +charts and the operator, which live outside this repository — so the path form +keeps working and `tests/unit/test_service_entrypoints.py` still pins it. + +## Gotchas + +- **`tests/unit/tasks/test_retry_ceiling.py` no longer hangs** — upstream fixed + it before this rebase. It parametrizes over runners *discovered from source* + by the presence of `.retry += 1`, so a runner migrating to the driver silently + drops out of it. Each migration therefore also moves the runner into that + file's `_DRIVER_MIGRATED` set, which is asserted to really have handed the + retry counter over (`test_migrated_runners_delegate_retry`). +- Per-runner behaviour tests for migrated runners live in + **`tests/unit/tasks/test_runner_specs.py`** (one section per runner: handler + outcome vocabulary + eligibility + `on_finish`). Extend it as you migrate. +- Several migrations fix latent bugs (a failure path that suspended without ever + incrementing retry, so a declared `max_retry` could never bind). Each is called + out in its commit message — keep doing that rather than folding them in + silently. + +## Verification + +```bash +tox run-parallel -e lint,types +tox run -e unit -- tests/unit/tasks/ tests/unit/test_lvol_sync_op_task.py \ + tests/unit/test_task_cancellation.py +tox run -e unit # full unit tier, ~45s, currently green +``` + +Do not run `tox run` (the integration tier is broken independently of this work). + +## Before opening the PR + +Delete these two handoff files (`HANDOFF.md`, `TASK_RUNNER_REWORK_PLAN.md`) — +they are transfer artifacts, not part of the change. diff --git a/TASK_RUNNER_REWORK_PLAN.md b/TASK_RUNNER_REWORK_PLAN.md new file mode 100644 index 0000000000..31ebc544fb --- /dev/null +++ b/TASK_RUNNER_REWORK_PLAN.md @@ -0,0 +1,359 @@ +# Rework task runners into a resilient, concurrent, deduplicated framework + +## Context + +`simplyblock_core/services/` contains ~15 `tasks_runner_*.py` files. Each is a +hand-written `main()` `while True` loop plus a `task_runner()` that share the +same skeleton but diverge in almost every detail. The divergence is +**accidental** — the result of copy-paste plus independent incident fixes, not +deliberate per-runner design — and it produces real correctness gaps: + +- **DB-error handling is broken in most runners.** The dominant idiom calls + `db.get_clusters()` once inside a `try` (discarding the result), then calls it + **again unguarded** and uses *that* — so the guard is dead code and any + `FDBError` crashes the process (`tasks_runner_backup/migration/restart/` + `port_allow/lvol_migration/jc_comp/new_dev_migration/failed_migration`). + `tasks_runner_fdb_backup` has no guard at all. Per-cluster `get_job_tasks()` + is unguarded in almost all of them — this is the 2026-07-16 incident that + killed several runners with no auto-restart. Only `node_add` and + `sync_lvol_del` handle DB errors coherently. +- **`max_retry` semantics differ** (`retry >= max_retry` vs + `0 <= max_retry <= retry` vs `max_retry > 0 and retry >= max_retry`). With the + model default `max_retry = -1` (meant to be "unbounded"), the first form + terminates a task *immediately* (`0 >= -1`). Pinned intent lives in + `tests/unit/tasks/test_max_retry_semantics.py` and `test_retry_ceiling.py`. +- **The task lease (`tasks_controller.claim_task`) is applied in only 6 of the + runners.** The rest can double-execute side-effecting tasks during rolling + deploys / dual-manager windows. +- **Concurrency exists in only two runners** (`node_add`, `restart`: + `ThreadPoolExecutor` + per-task/per-node inflight sets + capped backoff); + `cluster_expand` has inline backoff; the rest are strictly serial. +- Sleep intervals, `IN_ACTIVATION` skipping, and task re-fetch placement are all + ad-hoc. + +**Deployment context:** runners are Docker Swarm services +(`simplyblock_core/scripts/docker-compose-swarm.yml`), which restart a container +on exit by default. So the intended way to survive a wedged FDB client is to +**let a DB error surface as a process failure (`sys.exit(1)`) and rely on Swarm +to restart with a fresh FDB connection.** + +**Intended outcome:** one shared driver that owns the loop, DB-error-to-exit, +task lease, standardized retry/lifecycle, and opt-in concurrency — so each +runner is reduced to its domain-specific "advance one task" handler. Reached via +small single-purpose commits so the branch stays easy to rebase onto upstream. + +### Decisions (confirmed with user) + +- **DB-error model: immediate `sys.exit(1)`.** No consecutive-failure threshold. + This *simplifies* the current `sync_lvol_del` threshold counter down to the + common model. +- **Scope: convert all task runners** onto the shared driver, with two + exceptions: + - `tasks_cluster_status.py` — a shell-command runner, not `FN_*`-task based; + being replaced by a deployment-level solution. **Leave untouched.** + - `tasks_runner_backup_merge.py` — a periodic *policy/schedule evaluator* over + lvols with no `FN_*` tasks. **Reclassify as a service**: rename to follow the + `*_service.py` / `*_monitor.py` convention already used in the directory + (`snapshot_monitor.py`, `lvol_monitor.py`, `health_check_service.py`, …). +- **Execution model: serial by default, concurrency opt-in** (per-task and + per-key exclusion + capped backoff), generalizing `node_add`/`restart`. +- **Exceptions are the failure-signaling mechanism.** Specific task functions + return `None` (void) and never touch task state; they *raise* to signal + failure, using ordinary Python function semantics. The driver owns **all** + `task.status` / `task.retry` / `write_to_db` mutation and translates the + handler's return-or-raise into the task's next state. Retryable vs + non-retryable is disambiguated by exception type (see the vocabulary in B0). + - **Placement pushback (accepted friction):** this cannot land in Phase A. A + handler can only stop mutating task state once a driver exists to own that + state — before then, a raised exception has nothing to catch it and would + crash the runner. So the contract is *defined* in **B0** and *realized* + per-runner during each **B1…Bn** migration. Phase A therefore leaves handler + internals alone on purpose (Phase B rewrites them), and A2/A4 only touch the + loop-level and terminate-decision code, not the handlers' error plumbing. + +### Constraints to preserve + +- Each runner keeps an **importable, unit-testable per-task entry point** + (`tests/unit/tasks/` imports the module and calls `task_runner(...)` directly; + this works because `main()` sits behind `if __name__ == "__main__"`). The + handler function must stay module-level and callable in isolation. +- Reuse existing helpers, do not reinvent: `tasks_controller.claim_task`, + `get_active_node_tasks`, `get_active_node_mig_task`, `is_auto_restart_paused`; + `constants.TASK_EXEC_INTERVAL_SEC`, `RESTART_TASK_EXEC_INTERVAL_MAX_SEC`, + `NODE_RESTART_MAX_PARALLEL_SUSPENDED`; `storage_node_ops.fd_dead_recovery_allowed`; + `snapshot_controller.lvstore_op_lock`. +- Keep documented deliberate exceptions (e.g. `port_allow` intentionally does + **not** skip `IN_ACTIVATION` clusters — preserve with its comment). + +--- + +## Phase A — Reconcile inconsistencies (no shared abstraction yet) + +Each item is one focused commit spanning the affected runners. These land first +so behavior is uniform *before* it is centralized, which also makes Phase B a +mechanical extraction. + +**A1 — DB errors become runner failures (immediate exit).** +In every converted runner's `main()`: delete the discard-then-recall +`get_clusters()` double-call; call it once and let exceptions propagate. Wrap the +loop body so any DB access failure (`get_clusters`, `get_job_tasks`, +`get_cluster_by_id`, …) logs and `sys.exit(1)`. Remove `sync_lvol_del`'s +`_DB_FAILURE_RESTART_THRESHOLD` counter (now redundant). Task-*handler* errors +stay caught and non-fatal — only infra/DB errors exit. +Files: all `tasks_runner_*` in scope. + +**A2 — Standardize `max_retry` semantics.** +One canonical rule: `max_retry < 0` ⇒ unbounded; otherwise terminate when +`retry >= max_retry`. Fix the runners using the immediate-terminate form. Verify +against `tests/unit/tasks/test_max_retry_semantics.py` and `test_retry_ceiling.py`. +Files: `node_add`, `jc_comp`, `fdb_backup`, `replication_final`, `backup`, +`cluster_expand`, migration trio, `restart`, `port_allow`, `lvol_migration`, +`sync_lvol_del`. + +**A3 — Apply the task lease uniformly.** +Add `tasks_controller.claim_task(task)` (skip-if-owned) to every side-effecting +runner that lacks it: `backup`, `fdb_backup`, `jc_comp`, `new_dev_migration`, +`failed_migration`, `replication_final`, `sync_lvol_del`. + +**A4 — Loop hygiene → folded into Phase B (no standalone commit).** +Investigation showed the three "hygiene" concerns are premature to reconcile in +Phase A and are owned correctly by the B0 driver + `RunnerSpec`: +- **Interval** — the literals deliberately differ (`3`/`5`/`10`/`60`s; + `TASK_EXEC_INTERVAL_SEC` is `10`), so unifying onto one constant changes polling + cadence. Each runner keeps its value via `spec.interval`. +- **`IN_ACTIVATION` skip** — task-dependent (some runners run during activation; + `port_allow` opts out by design). Folded into the **eligibility predicate** as + the condition `cluster.status != IN_ACTIVATION`, dropping the separate + `skip_in_activation` flag. +- **Re-fetch** — every runner *already* re-fetches before running; only the + *location* differs (main vs handler), which is harmless. The driver performs a + single consistent re-fetch before the handler; handlers stop re-fetching. + +**A5 — Reclassify `backup_merge` as a service.** +Rename `tasks_runner_backup_merge.py` → `backup_merge_service.py` (fix its own +dead double `get_clusters()` guard while moving it). Update the `command:` in +`docker-compose-swarm.yml` and any other references. It stays a plain periodic +service and does **not** move onto the driver in Phase B. + +--- + +## Phase B — Shared driver + per-runner migration + +**B0 — Introduce `simplyblock_core/services/task_runner_base.py`.** One commit, +new module + unit tests, no runner changed yet. Provides: + +- `serve(spec)` — the entrypoint wrapper: runs the loop; on any uncaught + DB/infra exception, logs and `sys.exit(1)` (the A1 model, centralized). +- The loop: `get_clusters()` → per-cluster `get_job_tasks()` → filter by the + spec's `function_names` and `status != DONE` → dispatch. All DB calls unguarded + (failures exit). (Activation-skipping is not a loop concern — it is an + eligibility condition, below.) +- Per-task lifecycle wrapper applied around the runner's handler, in order: + **single re-fetch** (`db.get_task_by_id`, the one authoritative fresh read — + handlers no longer re-fetch) → canceled→DONE → `max_retry` terminate → **pre-run + skip-gates** (below) → set `RUNNING` → **call the void handler** → map its + return-or-raise to the next task state and `write_to_db`. The wrapper is the + *only* place task state is mutated. Handler exceptions are caught here → task + updated per the vocabulary below, **loop survives**. +- **Pre-run skip-gates.** One category of gate, evaluated before the handler, + sharing one outcome: *skip this cycle, do not consume a retry, do not mutate task + lifecycle state*. Two members, applied in this order: + 1. **Eligibility** — `spec.is_eligible(task, cluster) -> bool`, **default + `True`**, a side-effect-free "can I run right now?" read. The default keeps + simple runners trivial; complex tasks supply a predicate over + cluster/node/sibling-task state. It is the abstraction for what the migration + family does ad-hoc today: `lvol_migration`'s pre-run chain (`src`/`tgt` node + online, `cluster.status in (ACTIVE, DEGRADED)`, no open + `get_active_cluster_expand_task`), the `get_active_node_mig_task` / + same-node-sibling exclusion in the migration trio, and `restart`'s + `is_auto_restart_paused`. **`IN_ACTIVATION` skipping is just an eligibility + condition** (`cluster.status != IN_ACTIVATION`): runners that pause during + activation include it in their predicate; `port_allow` (documented opt-out) + omits it — replacing the former separate `skip_in_activation` flag. Modeling + these as a *pre-run* gate rather than `TaskDefer` (an *in-handler* raise) + keeps the "just wait, never consume a retry" family unbounded exactly as + `test_retry_ceiling.py`'s `INTENTIONALLY_UNBOUNDED` set documents. + 2. **Lease** — the built-in `tasks_controller.claim_task(task)` gate, **always + applied** (no opt-out: it is universal double-execution protection). + Conceptually the lease *is* an eligibility question ("is this host eligible to + run this?") and shares the skip-without-retry outcome, so it lives in the same + category — but it stays a **distinct built-in gate, not folded into + `is_eligible`**, for three reasons: (a) a + successful claim has a **side-effect** (writes `owner`/`updated_at` to + acquire/refresh ownership) whereas `is_eligible` is a pure read; (b) it is + **cross-cutting** double-execution protection that a custom predicate must + never be able to silently drop; (c) it runs **after** the cheap pure + eligibility so an ineligible task short-circuits *before* the driver churns + the lease's `updated_at` on a task it will skip anyway. +- **Exception vocabulary** (defined here; handlers raise these, driver + interprets): + - *returns normally* → success → `STATUS_DONE`. + - `TaskRetry` (and any other/unexpected `Exception`, the safe default) → + retryable failure → `STATUS_SUSPENDED`, `retry += 1`, capped backoff. + - `TaskDefer(reason)` → not a failure, just "can't proceed yet" (node not + online, peer restart in flight, sibling task on same node) → + `STATUS_SUSPENDED`, **retry NOT consumed**, short re-poll. Replaces today's + scattered "suspend but don't increment retry" branches. + - `TaskAbort(reason)` → permanent / non-retryable (missing param, object not + found, "not needed") → `STATUS_DONE` with a failure/short-circuit result. + Replaces the handlers that currently mark DONE mid-body. +- **Task writes are compare-and-set, never full-object writes** (revised after + the 2026-08 rebase — see "Upstream reconciliation" below). Every transition + runs as a mutator against the row as it stands (`db.atomic_update`), refusing + a row another actor has finished — and, for non-terminal transitions, one it + has canceled — and reporting whether it won. Only the two handler-owned + fields (`function_result`, `function_params`) are carried over from the + driver's copy. `on_finish` runs only for the winner of the terminal + transition. +- Execution: **one dispatch path**. Every execution is submitted to the pool and + registered in the per-task inflight set (plus the optional + `exclusion_key(task)` per-key set); *serialized* execution submits and waits + on the future rather than running inline. Capped exponential backoff on + failure. Generalizes `node_add`/`restart`. +- A `RunnerSpec` describing: `function_names`, `handler` (a void callable), + `is_eligible` (default `lambda task, cluster: True`), `interval` (each runner + keeps its current cadence), `concurrency`, `exclusion_key`, `on_finish`, + `serialize` (per-task/per-cycle predicate; defaults to `concurrency == 1`). + +**B1…Bn — Migrate one runner per commit**, simplest → hardest, each preserving +behavior and keeping the module-level handler importable for the existing tests: + +1. ✅ `fdb_backup`, `jc_comp`, `replication_final`, `sync_lvol_del` (trivial serial) +2. ✅ `backup`, `cluster_expand` +3. ✅ `node_add` (concurrency opt-in, `node_addr` exclusion key) +4. `restart` — **moved up** (was 6th). Concurrency + per-node exclusion; its + parallel-vs-serialized choice is the spec's `serialize` predicate + (`suspend_drain_complete` / `fd_dead_recovery_allowed`), and + `is_auto_restart_paused` becomes part of `is_eligible`. The driver's + `exclusion_key` + backoff subsumes `_node_inflight`/`_restart_next_attempt`, + and its `_task_finish`/`_task_update` CAS helpers are subsumed by the + driver's. Migrated first of the remainder because it is the runner that + defines the driver's hard requirements — if the design holds here it holds + everywhere. +5. migration trio — `migration`, `new_dev_migration`, `failed_migration` + (serial; `get_active_node_mig_task` / same-node-sibling gating becomes the + spec's `is_eligible` predicate, not a concurrency `exclusion_key`) +6. `port_allow` (large; its `is_eligible` omits the `IN_ACTIVATION` check — the + documented opt-out — and keeps the recovery logic) +7. `node_removal` (`FN_NODE_REMOVAL`; already leased, module-level loop, + unbounded — a straightforward serial migration) + +**Deferred to a follow-up PR** (see rebase-friendliness below — both are under +active upstream development, and neither blocks the rest): + +- `lvol_migration` (largest; migrate the loop/lease/retry shell only, leave the + domain snapshot-copy state machine intact. Its pre-run guard chain — node + status, `cluster.status in (ACTIVE, DEGRADED)`, open cluster-expand task — + becomes the spec's `is_eligible` predicate; unbounded `max_retry=-1` is kept) +- `batch_migration` (`FN_LVOL_BATCH_MIG`; migrate the loop/lease shell, leave + its group orchestration intact) + +**Upstream reconciliation (2026-08 rebase onto origin/main `4cb16a20b`).** +Upstream landed `a61b00ad4` — *"fix(restart): stop double execution of a +node-restart task (2026-07-29 incident)"* — which invalidates two of B0's +original choices, both now corrected above: + +- **Full-object task writes are unsafe.** A `task.write_to_db()` of a copy read + before a long handler ran reverts whatever other actors committed meanwhile: + it un-canceled a task that `cancel_pending_node_restart_tasks` canceled when + the node came back ONLINE, and reclaimed a lease another host had taken. The + driver held that stale copy for the *entire* handler duration, i.e. the widest + possible window, and centralizing it would have propagated the defect to every + runner. All transitions are CAS now; the migrated runners needed no changes, + since none of them touch task state. +- **A split dispatch path re-enters running tasks.** Restart chooses parallel vs + serialized per task per cycle from live cluster state (`suspend_drain_complete`, + `fd_dead_recovery_allowed`), and a mode flip mid-restart re-entered a task + still running on the pool because the inline branch consulted no inflight map. + Hence the single dispatch path and the `serialize` predicate: `concurrency` + cannot be a static number for restart. + +Also revised: B0 originally said handlers stop re-fetching. They stop +re-fetching for *lifecycle* decisions, but a handler must re-read the task +immediately before a destructive step — upstream added exactly that before +restart's shutdown. + +**Rebase-friendliness (2026-08).** `lvol_migration` and `batch_migration` took +~17 upstream commits in this window (cleanup state machine rewritten, multipath, +retry ceilings removed), several still labelled `TEMP:`. B7/B9 are therefore +deferred to a follow-up PR rather than rebased repeatedly against a moving +target; `restart` moves up to be migrated first, since it is the runner that +defines the driver's hard requirements. + +**Earlier upstream reconciliation (2026-07 rebase).** +Upstream independently reworked the runners: renamed `task_runner`→`process_task` +and added `tasks_controller.task_lease_heartbeat` (a lease-keepalive thread) to +the six long-blocking runners. This is now **folded into the B0 driver** — the +driver wraps every handler call in `task_lease_heartbeat`, so each migrated runner +gets keepalive for free and the per-runner boilerplate is deleted on migration. +The branch's original "Align task runner structure" commit was dropped as +superseded; A1/A2/A3 were re-applied on top of upstream (and A1/A3 extended to the +two new runners). Note: `tests/unit/tasks/test_retry_ceiling.py` **hangs on +origin/main itself** — upstream restructured the runners it drives without updating +it. It is stale and will be superseded by the driver's own tests as runners migrate +(each `main()` becomes `serve(SPEC)`); it should be skipped/removed rather than +propped up. + +Each `main()` collapses to `serve(SPEC)`. Each `task_runner(task)` becomes the +spec's **void handler**: it keeps only the domain work, returns `None` on +success, and raises `TaskRetry` / `TaskDefer` / `TaskAbort` (per B0) instead of +setting `task.status`/`retry` or calling `write_to_db`. Existing per-runner +mappings to preserve when translating: `jc_comp`'s "compression not needed" → +`TaskAbort`; "node not online" / "task on same node" → `TaskDefer`; RPC failure +→ `TaskRetry`. `restart`'s defer-vs-fail distinction (currently inferred by +comparing `retry` before/after) becomes explicit `TaskDefer` vs `TaskRetry`, +removing that bookkeeping. The module-level handler stays importable so +`tests/unit/tasks/` keeps calling it directly (tests assert on raised exceptions ++ driver-applied state rather than in-handler writes). + +--- + +## Files + +- **New:** `simplyblock_core/services/task_runner_base.py` (+ `tests/unit/tasks/test_task_runner_base.py`) +- **Renamed:** `tasks_runner_backup_merge.py` → `backup_merge_service.py` +- **Modified (all in `simplyblock_core/services/`):** `tasks_runner_fdb_backup`, + `_jc_comp`, `_replication_final`, `_sync_lvol_del`, `_backup`, `_cluster_expand`, + `_node_add`, `_migration`, `_new_dev_migration`, `_failed_migration`, + `_port_allow`, `_restart`, `_lvol_migration` +- **Modified:** `simplyblock_core/scripts/docker-compose-swarm.yml` (backup_merge command; no other command changes — filenames of converted runners stay the same) +- **Untouched:** `tasks_cluster_status.py` +- **Reused (no change):** `simplyblock_core/controllers/tasks_controller.py`, + `constants.py`, `storage_node_ops.py`, `controllers/snapshot_controller.py` + +## Verification + +Per the `tox-verify` skill — after each commit, targeted; full suite before finishing: + +1. `tox run-parallel -e lint,types` — must be green. +2. Targeted unit tests while iterating: + `tox run -e unit -- tests/unit/tasks/ tests/unit/test_task_lease.py \ + tests/unit/test_port_allow_recovery_refactor.py \ + tests/unit/test_drain_replaced_with_fixed_sleep.py \ + tests/unit/test_lvol_sync_op_task.py \ + tests/unit/tasks/test_task_runner_base.py` + The existing retry/lease tests must pass unchanged (they pin the standardized + semantics); extend them where new behavior warrants. New `test_task_runner_base.py` + asserts the exception vocabulary: a void return → DONE, `TaskRetry`/unexpected + `Exception` → SUSPENDED+retry+backoff, `TaskDefer` → SUSPENDED without + consuming a retry, `TaskAbort` → DONE; that an `is_eligible → False` task is + deferred without entering its handler or consuming a retry (default-`True` + spec always runs); and that a handler exception never escapes the loop while a + DB error does exit. +3. Import-smoke every converted runner (`python -c "import simplyblock_core.services.tasks_runner_X"`) + to confirm module-level definitions still load under the stubbed-fdb unit env. +4. `tox run -e unit` full unit tier; then the integration test that patches the + lvol-migration runner (`tests/integration/test_dual_fault_tolerance.py`) if + Docker + `libfdb_c` are available. +5. **DB-failure behavior (manual, infra-dependent):** run one converted runner + against a dev-compose FDB, stop FDB, and confirm the process exits non-zero + (immediate `sys.exit(1)`) rather than hanging — i.e. it surfaces as a Swarm + restart. Confirm a task-handler exception instead only suspends that task and + the loop keeps polling. + +### Rebase-friendliness note + +Commits are single-purpose and mostly additive (new module) or line-local +(loop-body edits), so conflicts against ongoing upstream runner fixes stay small +and localized — mirroring the resolution already done for +`tasks_runner_sync_lvol_del.py` on this branch. diff --git a/docker/AGENTS.md b/docker/AGENTS.md index 323a54d2df..b9115b106e 100644 --- a/docker/AGENTS.md +++ b/docker/AGENTS.md @@ -28,6 +28,7 @@ docker run --rm IMG python3 -c 'import sys; print(sys.version, sys.prefix, sys._ docker run --rm IMG sudo -E python3 -c 'import sys; print(sys.prefix)' # must be /opt/venv docker run --rm IMG sbctl --version docker run --rm IMG simplyblock-service --help +docker run --rm IMG simplyblock-task-runner --help ``` ## Stages and caching @@ -98,9 +99,10 @@ let the scan go green while the published image still carries the vulnerability. This is compatibility scaffolding with a defined exit. The packaging half is already done — `uv sync --no-editable` installs cleanly and every data file (`env_var`, `scripts/**`, `templates/**`, dashboards) resolves from `site-packages`, so nothing in the wheel blocks the - switch. The only thing left is the consumers: once they all use the `simplyblock-service` / - `sbctl` entry points, the change is `uv sync --locked --no-editable` in the builder plus - dropping `COPY . /app` from the runtime stage. Remaining offenders: + switch. The only thing left is the consumers: once they all use the `simplyblock-task-runner` / + `simplyblock-service` / `sbctl` entry points, the change is `uv sync --locked --no-editable` in + the builder plus dropping `COPY . /app` from the runtime stage. The compose file's task runners + and `cluster_ops.py`'s runner services are already off the paths. Remaining offenders: ```bash # from the sbcli repository root, not from docker/ diff --git a/docker/Dockerfile b/docker/Dockerfile index be68ae355e..dfb335206c 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -168,9 +168,9 @@ COPY . . # compatibility scaffolding, not the intended end state: the Helm charts, the # Swarm compose file and the operator still name source files # (`python3 simplyblock_core/services/.py`), which only resolve because -# /app is the package. Once those consumers use the `simplyblock-service` and -# `sbctl` entry points, this becomes `--no-editable` and the `COPY . /app` in -# the runtime stage goes away. See AGENTS.md. +# /app is the package. Once those consumers use the `simplyblock-task-runner`, +# `simplyblock-service` and `sbctl` entry points, this becomes `--no-editable` +# and the `COPY . /app` in the runtime stage goes away. See AGENTS.md. # # uv creates the environment without pip, setuptools or wheel, so there is # nothing to uninstall afterwards and no vendored pip bundle for the image scan diff --git a/pyproject.toml b/pyproject.toml index e8e2cb8497..759c8aa2b6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -49,6 +49,7 @@ Homepage = "https://www.simplyblock.io/" sbctl = "simplyblock_cli.cli:main" sbcli-dev = "simplyblock_cli.cli:main" simplyblock-service = "simplyblock_core.services.__main__:main" +simplyblock-task-runner = "simplyblock_core.services.task_runners:main" [dependency-groups] test = [ diff --git a/scripts/collect_logs.py b/scripts/collect_logs.py index b8d7d3e11a..f2ec1606a5 100755 --- a/scripts/collect_logs.py +++ b/scripts/collect_logs.py @@ -101,7 +101,7 @@ "TasksRunnerJCCompResume", "TasksRunnerLVolSyncDelete", "TasksRunnerBackup", - "TasksRunnerBackupMerge", + "BackupMergeService", # Async cross-cluster replication: these run on the CP but were missing from # this list, so a replication incident collected no replication logs at all. "SnapshotReplication", diff --git a/simplyblock_core/cluster_ops.py b/simplyblock_core/cluster_ops.py index 76e88b77fe..f0b4a4d585 100644 --- a/simplyblock_core/cluster_ops.py +++ b/simplyblock_core/cluster_ops.py @@ -2495,28 +2495,28 @@ def update_cluster(cluster_id, mgmt_only=False, restart=False, spdk_image=None, utils.create_docker_service( cluster_docker=cluster_docker, service_name="app_SnapshotMonitor", - service_file="python3 simplyblock_core/services/snapshot_monitor.py", + command=["python3", "simplyblock_core/services/snapshot_monitor.py"], service_image=service_image) if "app_TasksRunnerLVolSyncDelete" not in service_names: utils.create_docker_service( cluster_docker=cluster_docker, service_name="app_TasksRunnerLVolSyncDelete", - service_file="python3 simplyblock_core/services/tasks_runner_sync_lvol_del.py", + command=["simplyblock-task-runner", "tasks-runner-sync-lvol-del"], service_image=service_image) if "app_TasksRunnerJCCompResume" not in service_names: utils.create_docker_service( cluster_docker=cluster_docker, service_name="app_TasksRunnerJCCompResume", - service_file="python3 simplyblock_core/services/tasks_runner_jc_comp.py", + command=["simplyblock-task-runner", "tasks-runner-jc-comp"], service_image=service_image) if "app_BackupService" not in service_names: utils.create_docker_service( cluster_docker=cluster_docker, service_name="app_BackupService", - service_file="python3 simplyblock_core/services/tasks_runner_fdb_backup.py", + command=["simplyblock-task-runner", "tasks-runner-fdb-backup"], service_image=service_image) logger.info("Done updating mgmt cluster") @@ -2554,7 +2554,7 @@ def update_cluster(cluster_id, mgmt_only=False, restart=False, spdk_image=None, namespace=namespace, deployment_name="simplyblock-tasks-runner-sync-lvol-del", container_name="tasks-runner-sync-lvol-del", - service_file="simplyblock_core/services/tasks_runner_sync_lvol_del.py", + command=["simplyblock-task-runner", "tasks-runner-sync-lvol-del"], container_image=service_image) if "simplyblock-snapshot-monitor" not in deployment_names: @@ -2562,7 +2562,7 @@ def update_cluster(cluster_id, mgmt_only=False, restart=False, spdk_image=None, namespace=namespace, deployment_name="simplyblock-snapshot-monitor", container_name="snapshot-monitor", - service_file="simplyblock_core/services/snapshot_monitor.py", + command=["python", "simplyblock_core/services/snapshot_monitor.py"], container_image=service_image) # Update DaemonSets diff --git a/simplyblock_core/controllers/cluster_expansion/preconditions.py b/simplyblock_core/controllers/cluster_expansion/preconditions.py index 5188657ddf..479e92375e 100644 --- a/simplyblock_core/controllers/cluster_expansion/preconditions.py +++ b/simplyblock_core/controllers/cluster_expansion/preconditions.py @@ -25,7 +25,7 @@ runner defers its tasks while a cluster-expand task is open (they may be QUEUED — e.g. by an unexpected node outage mid-expansion — but never run before the expansion completes; see -``tasks_controller.defer_task_for_expansion``), ``shutdown_storage_node`` +``migration_task_common.require_active_cluster``), ``shutdown_storage_node`` refuses shutdowns during IN_EXPANSION, the executor holds a restart-phase gate on each donor (queueing create/delete/resize for the affected LVS), and the donors' outbound hublvol connections are dropped up-front (see @@ -78,7 +78,7 @@ #: Subset of the blocking families that DEFER on an open cluster-expand task #: (their runners suspend while the expansion is in progress — see -#: ``tasks_controller.defer_task_for_expansion``). A RESUME of an in-progress +#: ``migration_task_common.require_active_cluster``). A RESUME of an in-progress #: plan tolerates open tasks from these families: they are typically the #: recovery migrations queued by an unexpected node outage mid-expansion, #: and they wait for us, not the other way around (required order: expansion diff --git a/simplyblock_core/controllers/tasks_controller.py b/simplyblock_core/controllers/tasks_controller.py index 14eeb79a23..2af9701b83 100644 --- a/simplyblock_core/controllers/tasks_controller.py +++ b/simplyblock_core/controllers/tasks_controller.py @@ -6,10 +6,10 @@ import threading import time import uuid +from typing import Callable, Optional from simplyblock_core import db_controller, constants, utils -from simplyblock_core.controllers import ( - tasks_events, device_controller, events_controller) +from simplyblock_core.controllers import tasks_events, device_controller from simplyblock_core.models.cluster import Cluster from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.storage_node import StorageNode @@ -95,6 +95,62 @@ def _mutate(t): return refreshed["ok"] +def _cancel_atomically( + task: JobSchedule, + mutate: Callable[[JobSchedule], bool], +) -> Optional[JobSchedule]: + """Apply a cancellation to the task row as it currently stands. + + NOT ``task.write_to_db()``: the copy a canceller holds was read before it + decided to cancel — off a bulk ``get_job_tasks`` scan, in the case below — + and the runner driving that task writes the same row meanwhile, claiming + its lease, moving it to running, advancing retry and recording handler + progress in function_params. A full-object write puts all of that back. The + damaging one is the owner lease: clearing it hands the task to the next + runner host that polls, which executes it a second time. That is the lost + update behind the 2026-07-29 double restart, arriving from the other side. + + ``mutate`` receives the fresh row and returns False to decline (a guard + that no longer holds). It may be replayed on transaction conflict, so it + must do nothing but mutate the object it is given. + + Returns the committed task if this call performed the cancellation, or None + if the row is gone or another actor got there first. + """ + now = str(datetime.datetime.now(datetime.timezone.utc)) + performed = {"ok": False} + + def _mutate(fresh): + if mutate(fresh) is False: + return False + fresh.updated_at = now + performed["ok"] = True + return True + + committed = db.atomic_update(task, _mutate) + return committed if performed["ok"] else None + + +def _flag_canceled(fresh: JobSchedule) -> bool: + """Set only the canceled flag. Where the task has got to — status, retry, + owner, progress — stays as the runner left it; the runner reads the flag on + its next pass and finishes the task itself.""" + if fresh.canceled: + return False + fresh.canceled = True + return True + + +def _flag_canceled_if_pending(fresh: JobSchedule) -> bool: + """As _flag_canceled, but declines a task that has already finished. An + opportunistic bulk cancellation has nothing left to cancel there, and + flagging it would misreport a completed task as canceled. (An operator's + explicit cancel_task is deliberately not this strict.)""" + if fresh.status == JobSchedule.STATUS_DONE: + return False + return _flag_canceled(fresh) + + @contextlib.contextmanager def task_lease_heartbeat(task, owner=None): """Refresh this host's lease on `task` every TASK_LEASE_HEARTBEAT_SEC for @@ -531,22 +587,54 @@ def cancel_pending_node_restart_tasks(cluster_id, node_id): # via the dedup guard in `_validate_new_task_node_restart` until the # task runner happens to pick it up — observed as a 5-minute window # of failing manual restarts after the node was already back online. + def _cancel_if_still_pending(fresh: JobSchedule) -> bool: + # Re-checked on the fresh row: a task that reached its own outcome + # between the scan and this write has nothing left to cancel, and its + # result must not be overwritten with ours. + if fresh.canceled or fresh.status == JobSchedule.STATUS_DONE: + return False + fresh.canceled = True + fresh.status = JobSchedule.STATUS_DONE + fresh.function_result = "canceled: node back online" + return True + canceled = 0 for task in db.get_job_tasks(cluster_id): if (task.function_name == JobSchedule.FN_NODE_RESTART and task.node_id == node_id and task.status != JobSchedule.STATUS_DONE and not task.canceled): - task.canceled = True - task.status = JobSchedule.STATUS_DONE - task.function_result = "canceled: node back online" - task.write_to_db(db.kv_store) + if _cancel_atomically(task, _cancel_if_still_pending) is None: + continue canceled += 1 logger.info( f"Canceled obsolete node_restart task {task.get_id()} (node {node_id} back online)") return canceled +def cancel_node_tasks(cluster_id, node_id, function_names): + """Flag the node's unfinished tasks of the given kinds canceled. + + Used when a node is going away and the work queued against it is moot + (shutdown cancelling its migration tasks). Like every canceller this reads + in bulk and writes one row at a time, so each write goes through the CAS in + _cancel_atomically rather than putting the scan's copy back. + + Returns how many tasks this call canceled. + """ + canceled = 0 + for task in db.get_job_tasks(cluster_id): + if task.node_id != node_id or task.function_name not in function_names: + continue + if task.status == JobSchedule.STATUS_DONE or task.canceled: + continue + if _cancel_atomically(task, _flag_canceled_if_pending) is None: + continue + canceled += 1 + logger.info(f"Canceled task {task.get_id()} ({task.function_name}) on node {node_id}") + return canceled + + def list_tasks(cluster_id, is_json=False, limit=50, **kwargs): try: db.get_cluster_by_id(cluster_id) @@ -619,9 +707,9 @@ def cancel_task(task_id): if task.device_id: device_controller.device_set_retries_exhausted(task.device_id, True) - task.canceled = True - task.write_to_db(db.kv_store) - tasks_events.task_canceled(task) + committed = _cancel_atomically(task, _flag_canceled) + if committed is not None: + tasks_events.task_canceled(committed) return True @@ -782,27 +870,6 @@ def get_active_cluster_expand_task(cluster_id): return False -def defer_task_for_expansion(task): - """Suspend ``task`` if a cluster expansion is in progress. Returns True - when deferred. - - Migration-family runners call this right after their cluster-status - gate. The status gate alone is not enough: a node outage mid-expansion - suspends the cluster-expand task and restores the cluster status to - ACTIVE between its retries — without this gate the outage's recovery - migrations would start running in that window and then block the - expansion resume, inverting the required order (expansion completes - FIRST, then outage device migration, then expansion migration). - Deliberately does not consume a retry: this is a deferral, not a - failure.""" - if not get_active_cluster_expand_task(task.cluster_id): - return False - task.function_result = "cluster expansion in progress, deferring" - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return True - - def get_active_node_tasks(cluster_id, node_id): tasks = db.get_job_tasks(cluster_id) out = [] @@ -948,19 +1015,6 @@ def add_batch_mig_task(group): ) -def _is_already_gone(err): - """True when an RPC error means "the thing is not there" (-19/ENODEV). - - For a delete that is success; for a register it is still a failure, but - it is not an actionable one to alert on repeatedly -- the object the - caller wanted registered has gone away underneath it. - """ - if err is None: - return False - if isinstance(err, dict): - return err.get("code") == -19 - return "-19" in str(err) - def add_lvol_sync_del_task(cluster_id, node_id, lvol_bdev_name, primary_node): """Deferred per-node sync delete. Retried until it succeeds. @@ -1013,148 +1067,6 @@ def add_lvol_sync_op_task(cluster_id, node_id, lvol_id, op, secondary_index=0): max_retry=-1) -def run_lvol_sync_op_task(task): - """Execute one FN_LVOL_SYNC_OP task (called by the sync runner's loop; - lives here so it is importable without triggering the runner's - module-level loop). Idempotent; never raises.""" - from simplyblock_core import storage_node_ops - from simplyblock_core.models.lvol_model import LVol - - def _finish(result): - task.function_result = result - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - - def _defer(result): - task.function_result = result - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - - def _defer_failure(result, err=None): - """Defer after a GENUINE failure, and escalate a persistent one. - - Distinct from _defer, which also covers benign waiting (node not - ONLINE yet, LVS owned by a restart, lvol not yet settled) -- those - are not failures and must not raise an alert. These tasks retry - forever by design, so without this a permanently failing - registration is invisible except as a replica that is silently - missing and every later snapshot on that node failing. - """ - previous = task.function_result - if _is_already_gone(err): - # -19 / ENODEV: nothing to act on, do not count it. - _defer(result) - return - task.retry = (task.retry or 0) + 1 - threshold = constants.TASK_FAILURE_ALERT_THRESHOLD - should_alert = task.retry > threshold and ( - task.retry == threshold + 1 or previous != result) - _defer(result) - if not should_alert: - return - try: - events_controller.log_event_cluster( - cluster_id=task.cluster_id, - domain=events_controller.DOMAIN_STORAGE, - event="LVOL_TASK_FAILING_REPEATEDLY", - db_object=task, - caused_by=events_controller.CAUSED_BY_MONITOR, - message=(f"Deferred lvol {task.function_params.get('op')} " - f"for {task.function_params.get('lvol_id')} has " - f"failed {task.retry} times on node {task.node_id}; " - f"it will keep retrying, but this is not a " - f"transient condition and needs attention. " - f"Last error: {result}"), - node_id=task.node_id, - event_level="Critical") - except Exception as event_error: - logger.warning("Could not log repeated-failure event: %s", event_error) - - try: - if task.canceled: - _finish("canceled") - return - - lvol_id = task.function_params.get("lvol_id") - op = task.function_params.get("op") - try: - lvol = db.get_lvol_by_id(lvol_id) - except KeyError: - # A missing record is normally obsolescence (the lvol was - # deleted) and completes the task. But add_lvol_ha queues these - # register tasks in its pre-check, BEFORE it writes the lvol - # record at the end of the create -- so this runner (3s poll) - # can see the task while the create is still running and - # permanently DROP the registration, leaving the replica - # missing and every later snapshot on that node failing - # (the LVOL_109 class). Inside the grace window, defer instead: - # once the record appears, the status check below takes over. - try: - age = time.time() - float(task.date or 0) - except (TypeError, ValueError): - # Unknown age: fall back to the previous behaviour and treat - # the record as genuinely gone, rather than deferring forever. - age = float("inf") - if age < constants.LVOL_SYNC_OP_RECORD_GRACE_SEC: - _defer("lvol record not written yet, retrying") - else: - _finish("lvol no longer exists") - return - if lvol.status == LVol.STATUS_IN_DELETION: - _finish("lvol is being deleted") - return - if lvol.status != LVol.STATUS_ONLINE: - _defer(f"lvol status is {lvol.status}, retrying") - return - - try: - node = db.get_storage_node_by_id(task.node_id) - except KeyError: - _finish("node no longer exists") - return - if node.get_id() not in lvol.nodes: - _finish("node no longer hosts this lvol (topology moved)") - return - if node.status != StorageNode.STATUS_ONLINE: - _defer(f"node is {node.status}, retrying") - return - if storage_node_ops.get_restart_phase(task.node_id, lvol.lvs_name): - # The owning flow (restart/activation/expansion) re-registers - # lvols itself; re-check once it has released the LVS. - _defer("LVS owned by a restart/activation/expansion, retrying") - return - - if task.status != JobSchedule.STATUS_RUNNING: - task.status = JobSchedule.STATUS_RUNNING - task.write_to_db(db.kv_store) - - if op == "register": - ok, err = storage_node_ops.repair_lvol_registration_on_non_leader( - lvol, node, task.function_params.get("secondary_index", 0)) - if ok: - _finish(f"registered lvol {lvol_id} on {task.node_id}") - else: - _defer_failure(f"registration failed: {err}", err) - elif op == "resize": - # Converge to the CURRENT DB size — resize_lvol persists the new - # size after the fan-out, so this always applies the latest - # target even if the lvol was resized again meanwhile. - size_in_mib = utils.convert_size(lvol.size, 'MiB') - ret = node.rpc_client(timeout=10, retry=2).bdev_lvol_resize( - f"{lvol.lvs_name}/{lvol.lvol_bdev}", size_in_mib) - if ret: - _finish(f"resized lvol {lvol_id} on {task.node_id} to {size_in_mib} MiB") - else: - _defer("resize RPC failed, retrying") - else: - _finish(f"unknown op {op!r}") - except Exception as e: - logger.error(f"lvol sync-op task {task.uuid} failed: {e}") - try: - _defer(f"error: {e}") - except Exception as defer_e: - logger.debug(f"lvol sync-op task {task.uuid}: _defer fallback also failed: {defer_e}") - def get_lvol_sync_del_task(cluster_id, node_id, lvol_bdev_name=None): tasks = db.get_job_tasks(cluster_id) for task in tasks: diff --git a/simplyblock_core/scripts/collect_logs.py b/simplyblock_core/scripts/collect_logs.py index 7149e60667..f120d8a11e 100755 --- a/simplyblock_core/scripts/collect_logs.py +++ b/simplyblock_core/scripts/collect_logs.py @@ -113,7 +113,7 @@ "TasksRunnerJCCompResume", "TasksRunnerLVolSyncDelete", "TasksRunnerBackup", - "TasksRunnerBackupMerge", + "BackupMergeService", "HAProxy", ] diff --git a/simplyblock_core/scripts/docker-compose-swarm.yml b/simplyblock_core/scripts/docker-compose-swarm.yml index ac31fd5262..ff59dfac73 100644 --- a/simplyblock_core/scripts/docker-compose-swarm.yml +++ b/simplyblock_core/scripts/docker-compose-swarm.yml @@ -313,7 +313,7 @@ services: TasksRunnerRestart: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_restart.py" + command: "simplyblock-task-runner tasks-runner-restart" deploy: placement: constraints: [node.role == manager] @@ -327,7 +327,7 @@ services: TasksRunnerMigration: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_migration.py" + command: "simplyblock-task-runner tasks-runner-migration" deploy: placement: constraints: [node.role == manager] @@ -341,7 +341,7 @@ services: TasksRunnerLVolMigration: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_lvol_migration.py" + command: "simplyblock-task-runner tasks-runner-lvol-migration" deploy: placement: constraints: [node.role == manager] @@ -355,7 +355,7 @@ services: TasksRunnerBatchMigration: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_batch_migration.py" + command: "simplyblock-task-runner tasks-runner-batch-migration" deploy: placement: constraints: [node.role == manager] @@ -369,7 +369,7 @@ services: TasksRunnerFailedMigration: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_failed_migration.py" + command: "simplyblock-task-runner tasks-runner-failed-migration" deploy: placement: constraints: [node.role == manager] @@ -397,7 +397,7 @@ services: TasksRunnerNewDeviceMigration: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_new_dev_migration.py" + command: "simplyblock-task-runner tasks-runner-new-dev-migration" deploy: placement: constraints: [node.role == manager] @@ -411,7 +411,7 @@ services: TasksNodeAddRunner: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_node_add.py" + command: "simplyblock-task-runner tasks-runner-node-add" deploy: placement: constraints: [node.role == manager] @@ -425,7 +425,7 @@ services: TasksNodeRemovalRunner: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_node_removal.py" + command: "simplyblock-task-runner tasks-runner-node-removal" deploy: placement: constraints: [node.role == manager] @@ -439,7 +439,7 @@ services: TasksRunnerClusterExpand: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_cluster_expand.py" + command: "simplyblock-task-runner tasks-runner-cluster-expand" deploy: placement: constraints: [node.role == manager] @@ -453,7 +453,7 @@ services: TasksRunnerPortAllow: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_port_allow.py" + command: "simplyblock-task-runner tasks-runner-port-allow" deploy: placement: constraints: [node.role == manager] @@ -467,7 +467,7 @@ services: TasksRunnerJCCompResume: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_jc_comp.py" + command: "simplyblock-task-runner tasks-runner-jc-comp" deploy: placement: constraints: [node.role == manager] @@ -481,7 +481,7 @@ services: TasksRunnerLVolSyncDelete: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_sync_lvol_del.py" + command: "simplyblock-task-runner tasks-runner-sync-lvol-del" deploy: placement: constraints: [node.role == manager] @@ -495,7 +495,7 @@ services: TasksRunnerBackup: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_backup.py" + command: "simplyblock-task-runner tasks-runner-backup" deploy: placement: constraints: [node.role == manager] @@ -506,10 +506,10 @@ services: environment: SIMPLYBLOCK_LOG_LEVEL: "$LOG_LEVEL" - TasksRunnerBackupMerge: + BackupMergeService: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_backup_merge.py" + command: "python3 simplyblock_core/services/backup_merge_service.py" deploy: placement: constraints: [node.role == manager] @@ -537,7 +537,7 @@ services: TasksRunnerReplicationFinal: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_replication_final.py" + command: "simplyblock-task-runner tasks-runner-replication-final" deploy: placement: constraints: [node.role == manager] @@ -551,7 +551,7 @@ services: BackupService: <<: *service-base image: $SIMPLYBLOCK_DOCKER_IMAGE - command: "python3 simplyblock_core/services/tasks_runner_fdb_backup.py" + command: "simplyblock-task-runner tasks-runner-fdb-backup" deploy: placement: constraints: [node.role == manager] diff --git a/simplyblock_core/services/__main__.py b/simplyblock_core/services/__main__.py index 40ea976b6a..0f62dfb53f 100644 --- a/simplyblock_core/services/__main__.py +++ b/simplyblock_core/services/__main__.py @@ -18,14 +18,25 @@ import sys -# Modules that live here but are libraries imported by the services, not -# services themselves -- they have no entry point and nothing invokes them as a -# command. Listing them would advertise names that do nothing when run. -_NOT_SERVICES = frozenset({ +# Libraries imported by the services, not services themselves -- they have no +# entry point and nothing invokes them as a command. Listing them would +# advertise names that do nothing when run. +_LIBRARIES = frozenset({ "hub_controller_manager", + "migration_task_common", "replication_final_step", + "task_runner_base", }) +# Modules that do have an entry point of their own, but are not a service: +# ``task_runners`` is the ``simplyblock-task-runner`` console script, a sibling +# dispatcher over the task runners' specs. +_DISPATCHERS = frozenset({ + "task_runners", +}) + +_NOT_SERVICES = _LIBRARIES | _DISPATCHERS + def _service_names(): return sorted( diff --git a/simplyblock_core/services/tasks_runner_backup_merge.py b/simplyblock_core/services/backup_merge_service.py similarity index 83% rename from simplyblock_core/services/tasks_runner_backup_merge.py rename to simplyblock_core/services/backup_merge_service.py index d28cfdc9a8..74b8976671 100644 --- a/simplyblock_core/services/tasks_runner_backup_merge.py +++ b/simplyblock_core/services/backup_merge_service.py @@ -1,6 +1,6 @@ # coding=utf-8 """ -tasks_runner_backup_merge.py - periodic service that evaluates backup policies +backup_merge_service.py - periodic service that evaluates backup policies and triggers merges when retention limits are exceeded. """ import time @@ -16,12 +16,6 @@ def main(): logger.info("Starting backup merge service...") while True: - try: - db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue clusters = db.get_clusters() for cl in clusters: if cl.status == Cluster.STATUS_IN_ACTIVATION: diff --git a/simplyblock_core/services/migration_task_common.py b/simplyblock_core/services/migration_task_common.py new file mode 100644 index 0000000000..90d0f7393f --- /dev/null +++ b/simplyblock_core/services/migration_task_common.py @@ -0,0 +1,228 @@ +# coding=utf-8 +"""Shared pieces of the three device-migration runners. + +``tasks_runner_migration`` (FN_DEV_MIG), ``tasks_runner_new_dev_migration`` +(FN_NEW_DEV_MIG) and ``tasks_runner_failed_migration`` (FN_FAILED_DEV_MIG) all +drive the same data-plane operation — start a distr migration, then poll +``distr_migration_status`` until it settles — and differ only in what they +migrate and when they are allowed to start. +""" +from datetime import datetime, timezone + +from simplyblock_core import db_controller, utils +from simplyblock_core.controllers import tasks_controller +from simplyblock_core.models.nvme_device import NVMeDevice +from simplyblock_core.models.storage_node import StorageNode +from simplyblock_core.services.task_runner_base import ( + TaskAbort, + TaskDefer, + TaskProgress, + TaskRetry, + checkpoint, +) + +logger = utils.get_logger(__name__) + +db = db_controller.DBController() + +MIGRATION_WAIT_UNAVAILABLE_KEY = "wait_unavailable_before_retry" + +# A migration may only start once the cluster has been serving for a moment: +# a node that just came back is still settling its lvstore. +NODE_SETTLE_SEC = 60 + + +def migration_started(task): + """Whether this task has already issued its data-plane migration. + + The runners used to read this off ``task.status`` being RUNNING. That no + longer distinguishes anything — the driver moves a task to RUNNING before + calling the handler — so the marker the start step writes is used directly, + which is what the status was standing in for anyway. + """ + return "migration" in task.function_params + + +def require_active_cluster(task): + # IN_SHRINK is operable: draining data off the departing node is the node + # removal's own work, and removal blocks until every data device reaches + # FAILED_AND_MIGRATED — which only this family sets. Refusing here would + # deadlock the removal against the status it set itself. + cluster = db.get_cluster_by_id(task.cluster_id) + if not cluster.allows_operation(): + raise TaskRetry("cluster is not active, retrying") + + # Expansion-first ordering: no data migration runs while a cluster + # expansion is open — even between the expand task's retries, when the + # cluster status is momentarily ACTIVE. Without this, an outage's recovery + # migrations would start in that window and then block the expansion + # resume, inverting the required order (expansion completes FIRST, then + # outage device migration, then expansion migration). A deferral, not a + # failure: no retry consumed. + if tasks_controller.get_active_cluster_expand_task(task.cluster_id): + raise TaskDefer("cluster expansion in progress, deferring") + + +def require_node(task): + try: + return db.get_storage_node_by_id(task.node_id) + except KeyError: + raise TaskAbort(f"Node not found: {task.node_id}") + + +def nodes_settled(cluster_id, primaries_only=False): + """False while any node has been back for less than NODE_SETTLE_SEC. + + The three runners disagree on whether waiting for this costs a retry, so + the verdict is returned rather than raised. + """ + for node in db.get_storage_nodes_by_cluster_id(cluster_id): + if primaries_only and node.is_secondary_node: + continue + if not node.online_since: + continue + try: + settled_for = datetime.now(timezone.utc) - datetime.fromisoformat(node.online_since) + except Exception as e: + logger.error(f"Failed to get online since: {e}") + continue + if settled_for.total_seconds() < NODE_SETTLE_SEC: + return False + return True + + +def cluster_unavailable_state(cluster_id): + """Nodes and devices that are neither serving nor written off, as stable + ids — the set a migration waits on before (re)starting.""" + unavailable = [] + for node in db.get_storage_nodes_by_cluster_id(cluster_id): + if node.status in [StorageNode.STATUS_IN_CREATION, StorageNode.STATUS_REMOVED]: + continue + if node.status not in [StorageNode.STATUS_ONLINE, StorageNode.STATUS_SUSPENDED]: + unavailable.append(f"node:{node.get_id()}") + for dev in node.nvme_devices: + if dev.status in [NVMeDevice.STATUS_REMOVED, NVMeDevice.STATUS_FAILED_AND_MIGRATED]: + continue + if dev.status != NVMeDevice.STATUS_ONLINE: + unavailable.append(f"dev:{dev.get_id()}") + return sorted(unavailable) + + +def require_recovery_progress(task, unavailable): + """Hold a migration back while the cluster is degraded, releasing it only + when something actually recovers. + + Retrying against an unchanged set of unavailable nodes/devices just burns + the budget, so the set is recorded on the task and compared: an unchanged + (or grown) set defers, while any member coming back is the recovery event + that lets the migration proceed. + """ + previous = sorted(task.function_params.get(MIGRATION_WAIT_UNAVAILABLE_KEY, [])) + + if not unavailable: + if previous: + task.function_params.pop(MIGRATION_WAIT_UNAVAILABLE_KEY, None) + return + + recovered = set(previous) - set(unavailable) + task.function_params[MIGRATION_WAIT_UNAVAILABLE_KEY] = unavailable + if previous and recovered: + logger.info("Migration retry allowed after recovery event for task %s: %s", + task.uuid, sorted(recovered)) + return + + raise TaskDefer("waiting for unavailable nodes/devices to recover before " + f"restarting migration: {unavailable}") + + +def start_migration(task, start): + """Issue the data-plane migration and record that it was issued. + + ``start`` is the runner's own RPC call, returning falsy on failure. The + marker is checkpointed immediately: a crash between the RPC and the end of + the handler would otherwise lose it and start a second migration. + """ + try: + started = start() + except Exception as e: + logger.error(e) + started = False + if not started: + raise TaskRetry("Failed to start device migration task, retry later") + + return checkpoint(task, migration={"name": task.function_params["distr_name"]}) + + +def report_migration_status(task, res, allow_all_errors=False, allowed_error_codes=None): + """Translate a ``distr_migration_status`` poll into the task's outcome.""" + if not res: + raise TaskRetry("Failed to get mig status") + + allowed_error_codes = allowed_error_codes or [0] + res_data = res[0] + migration_status = res_data.get("status") + error_code = res_data.get("error", -1) + progress = res_data.get("progress", -1) + + if migration_status == "completed": + if error_code == 0: + task.function_result = "Done" + return + if error_code in allowed_error_codes or allow_all_errors: + task.function_result = f"mig completed with status: {error_code}" + return + # Drop the marker so the next attempt starts a fresh migration rather + # than polling the one that just errored. + del task.function_params['migration'] + raise TaskRetry(f"mig error: {error_code}, retrying") + + if migration_status == "failed": + raise TaskAbort(migration_status) + + if migration_status == "none": + del task.function_params['migration'] + raise TaskRetry("mig retry after restart") + + raise TaskProgress(f"Status: {migration_status}, progress:{progress}") + + +def no_sibling_migration(task): + """Eligibility: one migration at a time per node. + + Only gates a task that has not started yet — once its own migration is + running it is itself the sibling everything else waits on. + """ + if migration_started(task): + return True + return not tasks_controller.get_active_node_mig_task( + task.cluster_id, task.node_id, task.function_params.get("distr_name")) + + +def allow_all_migration_errors(cluster_id, statuses): + for node in db.get_storage_nodes_by_cluster_id(cluster_id): + for dev in node.nvme_devices: + if dev.status in statuses: + return True + return False + + +def poll_migration(task, rpc_client, allow_all_errors=False): + try: + res = rpc_client.distr_migration_status(**task.function_params["migration"]) + except (TaskAbort, TaskDefer, TaskProgress, TaskRetry): + raise + except Exception as e: + logger.error("Failed to get migration task status") + logger.exception(e) + raise TaskRetry("Failed to get migration status") + + report_migration_status(task, res, allow_all_errors=allow_all_errors) + + +def qos_high_priority(cluster_id): + return db.get_cluster_by_id(cluster_id).is_qos_set() + + +def sibling_eligibility(task, cluster): + """Spec-shaped wrapper of :func:`no_sibling_migration`.""" + return no_sibling_migration(task) diff --git a/simplyblock_core/services/task_runner_base.py b/simplyblock_core/services/task_runner_base.py new file mode 100644 index 0000000000..30c23e6049 --- /dev/null +++ b/simplyblock_core/services/task_runner_base.py @@ -0,0 +1,506 @@ +# coding=utf-8 +"""Shared driver for the task runners. + +A task runner is a long-lived service that polls FoundationDB for `JobSchedule` +tasks of one or more function names and advances each one. Historically every +runner hand-rolled its own ``while True`` loop, lease handling, retry ceiling and +error plumbing, which drifted apart. This module centralizes that skeleton so a +runner is reduced to a :class:`RunnerSpec` — most importantly a *handler* that +does only the domain work. + +Handler contract +---------------- +The handler is a callable ``handler(task) -> None``. It performs its domain work +(including mutating its own domain models — Backup, LVol, migration, … — and +writing those) but it MUST NOT touch task lifecycle state (``status`` / +``retry``) or call ``task.write_to_db`` for the task: the driver owns all of +that. The handler signals its outcome purely through ordinary Python control +flow: + +- **return** (``None``) — the task is terminally complete → ``STATUS_DONE``. +- **raise** :class:`TaskDefer` — cannot proceed yet, blocked on external state. + Suspend and re-poll next cycle; **no retry consumed**; no backoff. +- **raise** :class:`TaskProgress` — the work is under way and this poll found it + unfinished. As TaskDefer, but the task stays RUNNING; runners that gate + mutual exclusion on a RUNNING sibling depend on that. +- **raise** :class:`TaskRetry` (or any other, unexpected ``Exception``) — a + retryable failure. Suspend, **consume a retry**, and back off before the next + attempt. A failure whose message differs from the previous attempt's also + fires the spec's ``on_failure`` alert. +- **raise** :class:`TaskAbort` — a permanent, non-retryable stop (missing param, + object gone, "not needed"). Finish the task (``STATUS_DONE``) with the reason. + +Two task fields ARE the handler's to set: ``function_result`` (the message the +outcome is recorded with) and ``function_params`` (where a multi-cycle handler +records progress — ``recovery_started``, ``merge_started``, ``fail_count``). +Both are carried onto the row when the driver commits the outcome. + +A handler that does something destructive should re-read the task immediately +before doing it. The driver's pre-run re-fetch is authoritative for the +*lifecycle* decisions it makes, but it happens before the handler starts, and by +the time a long handler reaches its point of no return the task may have been +canceled. + +DB errors are deliberately NOT caught: an unhandled ``get_clusters`` / +``get_job_tasks`` failure propagates out of :func:`serve`, exits the process +non-zero, and lets the orchestrator restart it with a fresh FDB connection. + +Task writes are compare-and-set, never full-object writes: see +:meth:`TaskRunner._cas`. +""" +import datetime +import threading +import time +from concurrent.futures import ThreadPoolExecutor +from dataclasses import dataclass +from typing import Any, Callable, Optional, Sequence + +from simplyblock_core import constants, db_controller, utils +from simplyblock_core.controllers import tasks_controller +from simplyblock_core.models.job_schedule import JobSchedule + +logger = utils.get_logger(__name__) + +db = db_controller.DBController() + +# Cap the per-task exponential backoff so a permanently-failing task can't grow +# its retry delay without bound. +_BACKOFF_CAP_SEC = constants.RESTART_TASK_EXEC_INTERVAL_MAX_SEC + + +class TaskDefer(Exception): + """Handler signal: the task cannot proceed yet — blocked on external state. + Suspend and re-poll next cycle without consuming a retry.""" + + +class TaskProgress(Exception): + """Handler signal: the work is under way and this poll found it unfinished. + + Like :class:`TaskDefer` it consumes no retry, but the task stays RUNNING + rather than being suspended. That distinction is load-bearing: the + migration family gates mutual exclusion on a sibling task being RUNNING + (tasks_controller.get_active_node_mig_task), so suspending a migration + between polls would let a second one start on the same node.""" + + +class TaskRetry(Exception): + """Handler signal: a retryable failure. Suspend, consume a retry, back off. + + Any other unexpected ``Exception`` from a handler is treated identically.""" + + +class TaskAbort(Exception): + """Handler signal: a permanent, non-retryable stop. Finish the task.""" + + +def _default_eligible(task: JobSchedule, cluster: Any) -> bool: + return True + + +def _commit(task: JobSchedule, apply: Callable[[JobSchedule], None], + terminal: bool = False, context: str = "task-runner") -> Optional[JobSchedule]: + """Commit a change onto the task row as it exists NOW. + + Never a full-object ``write_to_db`` of ``task``: that copy was read before + the handler ran, and a handler runs for minutes (node add, restart, + migration). Writing it back reinstates every field another actor changed + meanwhile — it un-cancels a task that ``cancel_pending_node_restart_tasks`` + canceled when the node came back ONLINE, and reclaims a lease another host + has since taken. That pair of lost updates is what re-ran a restart against + an already-recovered node (2026-07-29 double restart). + + Only the two fields a handler owns are carried over from ``task``: + ``function_result`` and ``function_params`` (where handlers record progress + like ``recovery_started`` / ``merge_started``, which must survive so the + next attempt does not repeat the step). + + A terminal commit may finish a canceled task — that IS the cancellation + being carried out; a non-terminal one would be reviving it. + + Returns the committed task, or None if another actor already owns the + outcome — the caller must stop driving it. + """ + result = task.function_result + params = task.function_params + now = str(datetime.datetime.now(datetime.timezone.utc)) + won = {"ok": False} + + def _mutate(fresh: JobSchedule): + if fresh.status == JobSchedule.STATUS_DONE: + return False + if not terminal and fresh.canceled: + return False + fresh.function_result = result + fresh.function_params = params + apply(fresh) + fresh.updated_at = now + won["ok"] = True + return True + + committed = db.atomic_update(task, _mutate) + if committed is None or not won["ok"]: + logger.info(f"{context}: task {task.uuid} was finished or canceled " + f"concurrently; another actor owns the outcome") + return None + return committed + + +def checkpoint(task: JobSchedule, **params) -> Optional[JobSchedule]: + """Record handler progress on the task, mid-handler. + + For a long handler with a step that must not be repeated — a cleanup + shutdown, an issued transfer — mark it done the moment it succeeds rather + than when the handler returns, where a crash in between would lose the fact + and repeat the step on the next attempt. + + Doubles as the cancellation probe such a handler needs anyway: returns the + fresh task to carry on with, or None if the task was canceled or finished + underneath it, in which case the handler must stop rather than proceed to + the next destructive step. + """ + def _apply(fresh: JobSchedule) -> None: + fresh.function_params = dict(fresh.function_params, **params) + + return _commit(task, _apply) + + +@dataclass +class RunnerSpec: + """Describes one task runner. ``function_names`` and ``handler`` are the only + required fields; the rest default to a simple serial runner.""" + + function_names: Sequence[str] + handler: Callable[[JobSchedule], None] + name: str = "task-runner" + # Pure, side-effect-free "can I run this task right now?" predicate. The + # default always-eligible keeps simple runners trivial. A task judged + # ineligible is skipped this cycle without a lease claim or a write, exactly + # like the ad-hoc IN_ACTIVATION / same-node-sibling gates it replaces. + is_eligible: Callable[[JobSchedule, Any], bool] = _default_eligible + interval: float = constants.TASK_EXEC_INTERVAL_SEC + # Serial by default. > 1 runs tasks on a thread pool of this size. + concurrency: int = 1 + # Optional per-key mutual exclusion for concurrent mode: two tasks whose + # exclusion_key() is equal never run at the same time (e.g. one restart per + # node). Ignored when concurrency == 1. + exclusion_key: Optional[Callable[[JobSchedule], Any]] = None + # Optional cleanup, called once the task has reached STATUS_DONE and been + # written — whichever way it got there (handler success, TaskAbort, cancel + # or retry ceiling). For releasing state the task held, which would + # otherwise leak on the terminal paths the handler never sees. Runs only + # for the caller that won the terminal transition. + on_finish: Optional[Callable[[JobSchedule], None]] = None + # Optional alert, called with (task, reason) when a task fails with a + # message DIFFERENT from the one its previous attempt recorded. For a + # failure an operator must see in the cluster event log rather than only in + # `sbctl task list` — a repeat of the same message is not re-reported, so a + # cause that persists for hours writes one event, not one per attempt. + # Never called for a TaskDefer: waiting on external state is not failing. + on_failure: Optional[Callable[[JobSchedule, str], None]] = None + # Optional per-task, per-cycle "must this one run to completion before the + # loop moves on?". Defaults to serializing exactly when the pool has a + # single worker. A runner whose mode depends on live cluster state (node + # restart fans out only for a drained suspension or a fully-dead failure + # domain) supplies a predicate instead of a fixed concurrency. + serialize: Optional[Callable[[JobSchedule, Any], bool]] = None + # Optional per-cluster work, run once each cycle after that cluster's tasks + # are dispatched. For upkeep a runner owns that is not attached to any task + # — the restart runner's watchdog for nodes left in a transitional state + # with no task owning them. Failures are logged, never fatal. + on_cycle: Optional[Callable[[Any], None]] = None + # Optional delay-before-next-attempt for a task that consumed a retry, + # given the new retry count. Defaults to interval * 2**(retry-1), capped. + # A runner whose recovery curve is tuned to its own workload (node restart + # holds a steady lead-in cadence before backing off) supplies its own. + backoff: Optional[Callable[[int], float]] = None + + def __post_init__(self) -> None: + if self.concurrency < 1: + raise ValueError("concurrency must be >= 1") + + +class TaskRunner: + """Drives the tasks matched by a :class:`RunnerSpec`. See module docstring + for the handler contract.""" + + def __init__(self, spec: RunnerSpec): + self.spec = spec + self._executor = ThreadPoolExecutor(max_workers=spec.concurrency, + thread_name_prefix=spec.name) + self._lock = threading.Lock() + # task uuid -> Future-in-flight guard (this host), so the dispatch loop + # never hands the same task to two workers. Cross-host duplicate + # execution is prevented separately by the per-task lease. + self._inflight: set = set() + self._inflight_keys: dict = {} # exclusion key -> task uuid + self._next_attempt: dict = {} # task uuid -> earliest retry timestamp + + # -- public entrypoint -------------------------------------------------- + + def run(self) -> None: + logger.info(f"Starting {self.spec.name}...") + while True: + # DB errors are intentionally uncaught: they propagate out, exit the + # process, and the orchestrator restarts us with a fresh FDB client. + clusters = db.get_clusters() + if not clusters: + logger.error("No clusters found!") + else: + for cl in clusters: + for task in db.get_job_tasks(cl.get_id(), reverse=False): + if task.function_name not in self.spec.function_names: + continue + if task.status == JobSchedule.STATUS_DONE: + self._forget(task.uuid) + continue + self._dispatch(task, cl) + self._run_cycle_hook(cl) + time.sleep(self.spec.interval) + + def _run_cycle_hook(self, cluster: Any) -> None: + if self.spec.on_cycle is None: + return + # Upkeep failing must not stop the loop from serving tasks — but a DB + # error still propagates, since that means the process should exit. + try: + self.spec.on_cycle(cluster) + except Exception as e: # noqa: BLE001 - upkeep failure is not fatal + logger.error(f"{self.spec.name}: cycle hook failed for " + f"cluster {cluster.get_id()}: {e}") + logger.exception(e) + + # -- dispatch ----------------------------------------------------------- + + def _dispatch(self, task: JobSchedule, cluster: Any) -> None: + uuid = task.uuid + # Backoff gate: a task not yet due is skipped so a waiting task does not + # block the others behind it (the loop revisits every task each cycle). + if time.time() < self._next_attempt.get(uuid, 0): + return + + with self._lock: + if uuid in self._inflight: + return + key = self.spec.exclusion_key(task) if self.spec.exclusion_key else None + if key is not None and key in self._inflight_keys: + return + self._inflight.add(uuid) + if key is not None: + self._inflight_keys[key] = uuid + + # Single dispatch path: serialized execution submits to the pool and + # waits, rather than running inline. A split — one branch registering + # in-flight and another not — is what let a dispatch-mode flip + # mid-restart re-enter a task that was still running, and force-shut an + # already-recovered node (2026-07-29 double restart). Going through the + # registry either way makes a flip harmless in both directions. + future = self._executor.submit(self._process_worker, task, cluster) + if self._serialized(task, cluster): + future.result() + + def _serialized(self, task: JobSchedule, cluster: Any) -> bool: + if self.spec.serialize is not None: + return self.spec.serialize(task, cluster) + return self.spec.concurrency == 1 + + def _process_worker(self, task: JobSchedule, cluster: Any) -> None: + # A worker crash must be contained to this task, never kill the service + # loop or leave the task wedged in the in-flight set. + try: + self._process(task, cluster) + except Exception as e: # noqa: BLE001 - contain crash to this worker + logger.error(f"{self.spec.name}: task {task.uuid} crashed in worker: {e}") + logger.exception(e) + finally: + self._release_inflight(task.uuid) + + # -- per-task lifecycle ------------------------------------------------- + + def _process(self, task: JobSchedule, cluster: Any) -> None: + uuid = task.uuid + + # Pre-run skip-gate 1 — eligibility (pure, no write): not ready yet. + if not self.spec.is_eligible(task, cluster): + return + + # Pre-run skip-gate 2 — lease: another live host owns this task. + if not tasks_controller.claim_task(task): + logger.info(f"{self.spec.name}: task {uuid} owned by another runner host; skipping") + return + + # Authoritative re-fetch AFTER the claim: claim_task mutated the DB row + # (owner / updated_at) but not this local object, and the lifecycle + # decisions below — canceled, retry ceiling — must be made on the row as + # it stands, not on whatever the dispatch loop happened to read. + task = db.get_task_by_id(uuid) + if task is None or task.status == JobSchedule.STATUS_DONE: + self._forget(uuid) + return + + if task.canceled: + self._finish(task, "canceled") + return + if 0 <= task.max_retry <= task.retry: + self._finish(task, "max retry reached") + return + + if task.status != JobSchedule.STATUS_RUNNING: + running = self._cas(task, self._to(JobSchedule.STATUS_RUNNING)) + if running is None: + self._forget(uuid) + return + task = running + + # Drop the previous attempt's result so a task that fails and later + # succeeds does not finish carrying the stale failure message. Handlers + # that set a success message overwrite this; the rest get "completed". + # Kept first: the handler cannot see what the last attempt reported, so + # recognizing a repeated failure is the driver's job (see _fail). + previous_result = task.function_result + task.function_result = "" + + try: + # Heartbeat the lease for the duration of the handler: TASK_LEASE_TTL + # is far shorter than a node-add / restart / migration, so a lease + # refreshed only on task writes would go stale mid-handler and let a + # second host claim and double-drive the task. + with tasks_controller.task_lease_heartbeat(task): + self.spec.handler(task) + except TaskProgress as e: + self._progress(task, str(e)) + except TaskDefer as e: + self._defer(task, str(e)) + except TaskAbort as e: + self._finish(task, str(e) or "aborted") + except TaskRetry as e: + self._fail(task, str(e) or "retry", previous_result) + except Exception as e: # noqa: BLE001 - unexpected == retryable failure + logger.error(f"{self.spec.name}: task {uuid} handler raised: {e}") + logger.exception(e) + self._fail(task, f"unhandled error: {e}", previous_result) + else: + self._succeed(task) + + # -- outcome transitions (the only places task state is mutated) -------- + + def _cas(self, task: JobSchedule, apply: Callable[[JobSchedule], None], + terminal: bool = False) -> Optional[JobSchedule]: + return _commit(task, apply, terminal=terminal, context=self.spec.name) + + @staticmethod + def _to(status: str) -> Callable[[JobSchedule], None]: + def _apply(task: JobSchedule) -> None: + task.status = status + return _apply + + def _succeed(self, task: JobSchedule) -> None: + if not task.function_result: + task.function_result = "completed" + self._write_terminal(task) + + def _finish(self, task: JobSchedule, result: str) -> None: + """Terminal DONE for a non-handler-success reason (canceled, max retry, + abort).""" + task.function_result = result + self._write_terminal(task) + + def _write_terminal(self, task: JobSchedule) -> None: + committed = self._cas(task, self._to(JobSchedule.STATUS_DONE), terminal=True) + self._forget(task.uuid) + if committed is None or self.spec.on_finish is None: + # Losing the transition means someone else finished the task and + # owns its cleanup too; running it here would release the resource + # twice. + return + # Cleanup runs after the terminal write, so a hook that inspects the + # task's own state (a lock held until no active task remains) sees it + # as finished. A failing hook must not take the loop down with it. + try: + self.spec.on_finish(committed) + except Exception as e: # noqa: BLE001 - cleanup failure is not fatal + logger.error(f"{self.spec.name}: task {task.uuid} on_finish failed: {e}") + logger.exception(e) + + def _progress(self, task: JobSchedule, reason: str) -> None: + if reason: + task.function_result = reason + # Status stays RUNNING — see TaskProgress. The commit still happens, to + # record the progress message and refresh the lease. + if self._cas(task, self._to(JobSchedule.STATUS_RUNNING)) is None: + self._forget(task.uuid) + return + self._clear_backoff(task.uuid) + + def _alert(self, task: JobSchedule, reason: str) -> None: + if self.spec.on_failure is None: + return + # As on_finish: a hook that cannot record its alert must not turn a + # retryable task failure into a dead runner. + try: + self.spec.on_failure(task, reason) + except Exception as e: # noqa: BLE001 - alerting failure is not fatal + logger.error(f"{self.spec.name}: task {task.uuid} on_failure failed: {e}") + logger.exception(e) + + def _defer(self, task: JobSchedule, reason: str) -> None: + if reason: + task.function_result = reason + if self._cas(task, self._to(JobSchedule.STATUS_SUSPENDED)) is None: + self._forget(task.uuid) + return + self._clear_backoff(task.uuid) + + def _fail(self, task: JobSchedule, reason: str, previous_result: str = "") -> None: + logger.error(f"{self.spec.name}: task {task.uuid} failed: {reason}") + task.function_result = reason + + def _apply(fresh: JobSchedule) -> None: + fresh.retry += 1 + fresh.status = JobSchedule.STATUS_SUSPENDED + + committed = self._cas(task, _apply) + if committed is None: + self._forget(task.uuid) + return + # As on_finish: alert only for the caller that recorded the outcome, and + # only once per distinct message — a cause that persists for hours must + # not write one event per attempt. + if reason != previous_result: + self._alert(committed, reason) + # Back off on the committed retry count, not the stale local one. + with self._lock: + self._next_attempt[task.uuid] = time.time() + self._backoff_delay(committed.retry) + + # -- bookkeeping -------------------------------------------------------- + + def _backoff_delay(self, retry: int) -> float: + if retry <= 0: + return 0.0 + if self.spec.backoff is not None: + return self.spec.backoff(retry) + exp = min(retry - 1, 16) # guard the shift against absurd retry counts + return min(self.spec.interval * (2 ** exp), _BACKOFF_CAP_SEC) + + def _clear_backoff(self, uuid: str) -> None: + with self._lock: + self._next_attempt.pop(uuid, None) + + def _release_inflight(self, uuid: str) -> None: + with self._lock: + self._inflight.discard(uuid) + for key, owner_uuid in list(self._inflight_keys.items()): + if owner_uuid == uuid: + del self._inflight_keys[key] + + def _forget(self, uuid: str) -> None: + with self._lock: + self._next_attempt.pop(uuid, None) + self._inflight.discard(uuid) + for key, owner_uuid in list(self._inflight_keys.items()): + if owner_uuid == uuid: + del self._inflight_keys[key] + + +def serve(spec: RunnerSpec) -> None: + """Instantiate and run the driver for ``spec`` (a runner's ``main``).""" + TaskRunner(spec).run() diff --git a/simplyblock_core/services/task_runners.py b/simplyblock_core/services/task_runners.py new file mode 100644 index 0000000000..a4f0debcf5 --- /dev/null +++ b/simplyblock_core/services/task_runners.py @@ -0,0 +1,104 @@ +# coding=utf-8 +"""Entry point for the task runners (``simplyblock-task-runner``). + +After the rework a task runner *is* its :class:`~simplyblock_core.services. +task_runner_base.RunnerSpec`: the module exports one as ``SPEC`` and running it +is ``serve(SPEC)``. That makes a single shared entry point possible, rather than +one console script per runner:: + + simplyblock-task-runner tasks-runner-restart + +The name is the runner's own — ``RunnerSpec.name``, the same string its log +lines carry — not a file inside the image. Consumers that use it stop depending +on the source layout, which is what the historical form does:: + + python3 simplyblock_core/services/tasks_runner_restart.py + +That form keeps working, and has to: see ``tests/unit/test_service_entrypoints`` +for the artifacts pinned to it. So does the generic ``simplyblock-service +`` dispatcher, which reaches the runners as ordinary services. This is the +form new consumers should use. + +Runner names are derived from module names rather than by importing every runner +and reading its ``SPEC.name``: a runner module pulls in the controllers, +``storage_node_ops`` and an FDB client at import time, and listing the runners +must not cost that. Only the runner being served is imported, and its spec name +is checked against the name it was asked for — the two drifting apart would make +the deployment name a lie. +""" +import argparse +import importlib +import pkgutil +from typing import List, Optional + +from simplyblock_core.services.task_runner_base import RunnerSpec, serve + + +_PACKAGE = "simplyblock_core.services" +_MODULE_PREFIX = "tasks_runner_" + +# Runners that still drive their own poll loop instead of exporting a spec, and +# are therefore started through their module's ``main()``. Both are under active +# upstream rewrite; they join the rest when they migrate onto the driver. +# Explicit rather than "no SPEC attribute means legacy", so a migrated runner +# that fails to export its spec is an error instead of a silent fallback. +_NOT_ON_DRIVER = frozenset({ + "tasks-runner-batch-migration", + "tasks-runner-lvol-migration", +}) + + +def _module_name(name: str) -> str: + return name.replace("-", "_") + + +def runner_names() -> List[str]: + """The names of every task runner in this package.""" + package = importlib.import_module(_PACKAGE) + return sorted( + name.replace("_", "-") + for _, name, ispkg in pkgutil.iter_modules(package.__path__) + if not ispkg and name.startswith(_MODULE_PREFIX) + ) + + +def load_spec(name: str) -> Optional[RunnerSpec]: + """The spec ``name`` denotes, or None for a runner not on the driver yet. + + Importing the runner is the point where the name is validated, so a runner + renamed on one side of the deployment contract fails loudly at start-up + rather than serving the wrong tasks. + """ + if name in _NOT_ON_DRIVER: + return None + + module = importlib.import_module(f"{_PACKAGE}.{_module_name(name)}") + spec = getattr(module, "SPEC", None) + if spec is None: + raise RuntimeError(f"task runner {name} exports no SPEC") + if spec.name != name: + raise RuntimeError( + f"task runner {name} exports a spec named {spec.name!r}; " + "the deployment name and the spec name must agree") + return spec + + +def main(): + names = runner_names() + parser = argparse.ArgumentParser( + prog="simplyblock-task-runner", + description="Run a simplyblock task runner.", + ) + parser.add_argument("runner", choices=names, metavar="RUNNER", + help="one of: " + ", ".join(names)) + args = parser.parse_args() + + spec = load_spec(args.runner) + if spec is None: + importlib.import_module(f"{_PACKAGE}.{_module_name(args.runner)}").main() + return + serve(spec) + + +if __name__ == "__main__": + main() diff --git a/simplyblock_core/services/tasks_runner_backup.py b/simplyblock_core/services/tasks_runner_backup.py index e3e9c6de6e..a001e9af45 100644 --- a/simplyblock_core/services/tasks_runner_backup.py +++ b/simplyblock_core/services/tasks_runner_backup.py @@ -6,74 +6,79 @@ - FN_BACKUP: perform an S3 backup from a snapshot - FN_BACKUP_RESTORE: restore a backup chain into a new lvol - FN_BACKUP_MERGE: merge two backups to shorten the chain + +All three are multi-cycle: a task issues its RPC, defers, and polls the data +plane's transfer state on later cycles until it reaches a terminal state. """ import time -from simplyblock_core import constants, db_controller, utils +from simplyblock_core import db_controller, utils from simplyblock_core.controllers import backup_events from simplyblock_core.models.backup import Backup from simplyblock_core.models.cluster import Cluster from simplyblock_core.models.job_schedule import JobSchedule +from simplyblock_core.models.lvol_model import LVol from simplyblock_core.models.storage_node import StorageNode from simplyblock_core.rpc_client import RPCException +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskAbort, + TaskDefer, + TaskRetry, + serve, +) logger = utils.get_logger(__name__) db = db_controller.DBController() +# Time-based backstop for a task that is stuck but not erroring. +_DEFAULT_BACKUP_TIMEOUT_SEC = 14400 + + +def _online_node(node_id): + """The node the task's RPCs go to, or a signal to stop/wait.""" + try: + snode = db.get_storage_node_by_id(node_id) + except KeyError: + raise TaskAbort(f"Node {node_id} not found") + + if snode.status != StorageNode.STATUS_ONLINE: + raise TaskRetry(f"Node {snode.status}, retrying") + return snode + + +def _transfer_state(rpc_client, bdev_name): + try: + stat = rpc_client.bdev_lvol_transfer_stat(bdev_name) + except RPCException: + raise TaskRetry("transfer stat RPC failed, retrying") -def _fail_backup(backup, task, message): - backup.status = Backup.STATUS_FAILED - backup.error_message = message - backup.write_to_db() - backup_events.backup_failed(backup.cluster_id, backup.node_id, backup) - task.function_result = message - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) + if not stat or not isinstance(stat, dict): + raise TaskRetry("unexpected transfer stat response, retrying") + return stat.get("transfer_state", "") def _run_backup(task): backup_id = task.function_params.get("backup_id") if not backup_id: - task.function_result = "Missing backup_id" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return + raise TaskAbort("Missing backup_id") try: backup = db.get_backup_by_id(backup_id) except KeyError: - task.function_result = f"Backup {backup_id} not found" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return + raise TaskAbort(f"Backup {backup_id} not found") if backup.status not in (Backup.STATUS_PENDING, Backup.STATUS_IN_PROGRESS): - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return - - try: - snode = db.get_storage_node_by_id(backup.node_id) - except KeyError: - _fail_backup(backup, task, f"Node {backup.node_id} not found") - return - - if snode.status != StorageNode.STATUS_ONLINE: - task.retry += 1 - task.function_result = f"Node {snode.status}, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskAbort(f"Backup is already {backup.status}") + snode = _online_node(backup.node_id) rpc_client = snode.rpc_client(timeout=30) - # Resolve snapshot bdev name (needed for both kick-off and polling) try: snapshot = db.get_snapshot_by_id(backup.snapshot_id) except KeyError: - _fail_backup(backup, task, f"Snapshot {backup.snapshot_id} not found") - return + raise TaskAbort(f"Snapshot {backup.snapshot_id} not found") snap_bdev_name = snapshot.snap_bdev if not snap_bdev_name: @@ -82,69 +87,43 @@ def _run_backup(task): if backup.status == Backup.STATUS_PENDING: try: ret = rpc_client.bdev_lvol_s3_backup(backup.s3_id, [snap_bdev_name], cluster_batch=16) - if not ret: - _fail_backup(backup, task, "bdev_lvol_s3_backup RPC failed") - return except RPCException as e: - _fail_backup(backup, task, f"RPC error: {e}") - return + raise TaskAbort(f"RPC error: {e}") + if not ret: + raise TaskAbort("bdev_lvol_s3_backup RPC failed") backup.status = Backup.STATUS_IN_PROGRESS backup.write_to_db() # Give the data plane time to start the transfer before polling - task.status = JobSchedule.STATUS_SUSPENDED - task.function_result = "Backup in progress" - task.write_to_db(db.kv_store) - return + raise TaskDefer("Backup in progress") - # Poll via bdev_lvol_transfer_stat on the snapshot bdev - try: - stat = rpc_client.bdev_lvol_transfer_stat(snap_bdev_name) - except RPCException: - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) + state = _transfer_state(rpc_client, snap_bdev_name) + if state == "Done": + backup.status = Backup.STATUS_COMPLETED + backup.completed_at = int(time.time()) + backup.write_to_db() + backup_events.backup_completed(backup.cluster_id, backup.node_id, backup) + task.function_result = "Backup completed" return - if stat and isinstance(stat, dict): - state = stat.get("transfer_state", "") - if state == "Done": - backup.status = Backup.STATUS_COMPLETED - backup.completed_at = int(time.time()) - backup.write_to_db() - backup_events.backup_completed(backup.cluster_id, backup.node_id, backup) - task.function_result = "Backup completed" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - elif state == "Failed": - _fail_backup(backup, task, "Backup transfer failed on data plane") - elif state == "No process" and backup.status == Backup.STATUS_IN_PROGRESS: - # "No process" means no transfer is running for this bdev — the - # backup died (e.g. an SPDK crash wiped the in-flight transfer). - # Re-issue by resetting to PENDING, but COUNT it as a retry so the - # max_retry ceiling in process_task() can stop a backup that keeps - # failing. Without the increment this branch loops forever, and - # re-issuing an RPC that crashes the data plane just re-crashes it. - # NOTE: this treats "No process" as a failure. It relies on a - # healthy in-progress backup NOT sitting in "No process"; if the - # data plane ever reports "No process" for a running backup, this - # would fail it prematurely and completion needs another signal. - task.retry += 1 - backup.status = Backup.STATUS_PENDING - backup.write_to_db() - task.function_result = "No process, retrying backup start" - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - else: - # "In progress" — still running, retry later - task.status = JobSchedule.STATUS_SUSPENDED - task.function_result = "Backup in progress" - task.write_to_db(db.kv_store) - else: - # Unexpected response — retry - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) + if state == "Failed": + raise TaskAbort("Backup transfer failed on data plane") + + if state == "No process" and backup.status == Backup.STATUS_IN_PROGRESS: + # "No process" means no transfer is running for this bdev — the backup + # died (e.g. an SPDK crash wiped the in-flight transfer). Re-issue by + # resetting to PENDING, but COUNT it as a retry so the max_retry ceiling + # can stop a backup that keeps failing. Without that, re-issuing an RPC + # that crashes the data plane just re-crashes it, forever. + # NOTE: this treats "No process" as a failure. It relies on a healthy + # in-progress backup NOT sitting in "No process"; if the data plane ever + # reports "No process" for a running backup, this would fail it + # prematurely and completion needs another signal. + backup.status = Backup.STATUS_PENDING + backup.write_to_db() + raise TaskRetry("No process, retrying backup start") + + raise TaskDefer("Backup in progress") def _set_lvol_online(task): @@ -153,7 +132,6 @@ def _set_lvol_online(task): if not lvol_id: return try: - from simplyblock_core.models.lvol_model import LVol lvol = db.get_lvol_by_id(lvol_id) if lvol.status == LVol.STATUS_RESTORING: lvol.status = LVol.STATUS_ONLINE @@ -169,7 +147,6 @@ def _set_lvol_restore_failed(task, reason): if not lvol_id: return try: - from simplyblock_core.models.lvol_model import LVol lvol = db.get_lvol_by_id(lvol_id) if lvol.status == LVol.STATUS_RESTORING: lvol.status = LVol.STATUS_RESTORE_FAILED @@ -184,171 +161,96 @@ def _run_restore(task): lvol_name = task.function_params.get("lvol_name") chain_ids = task.function_params.get("chain_ids", []) node_id = task.node_id - recovery_started = task.function_params.get("recovery_started", False) - - try: - snode = db.get_storage_node_by_id(node_id) - except KeyError: - task.function_result = f"Node {node_id} not found" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return - - if snode.status != StorageNode.STATUS_ONLINE: - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + snode = _online_node(node_id) rpc_client = snode.rpc_client(timeout=30) # Check that the target lvol still exists in DB before doing any RPC work lvol_id = task.function_params.get("lvol_id") if lvol_id: try: - from simplyblock_core.models.lvol_model import LVol - lvol = db.get_lvol_by_id(lvol_id) - if lvol.status == LVol.STATUS_IN_DELETION: - task.function_result = f"Restore target {lvol_id} has been deleted" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return + if db.get_lvol_by_id(lvol_id).status == LVol.STATUS_IN_DELETION: + raise TaskAbort(f"Restore target {lvol_id} has been deleted") except KeyError: - task.function_result = f"Restore target {lvol_id} no longer exists" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return + raise TaskAbort(f"Restore target {lvol_id} no longer exists") - if not recovery_started: + if not task.function_params.get("recovery_started", False): try: ret = rpc_client.bdev_lvol_s3_recovery(lvol_name, chain_ids, cluster_batch=16) - if not ret: - task.function_result = "bdev_lvol_s3_recovery RPC failed" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return except RPCException as e: - task.function_result = f"RPC error: {e}" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskRetry(f"RPC error: {e}") + if not ret: + raise TaskRetry("bdev_lvol_s3_recovery RPC failed") - # Mark recovery as started so we don't re-issue the RPC on subsequent polls + # Don't re-issue the RPC on subsequent polls, and give the data plane + # time to start the transfer before the first one. task.function_params["recovery_started"] = True - # Give the data plane time to start the transfer before polling - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer("Restore started") - # Poll via bdev_lvol_transfer_stat on the target lvol - try: - stat = rpc_client.bdev_lvol_transfer_stat(lvol_name) - except RPCException: - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) + state = _transfer_state(rpc_client, lvol_name) + if state == "Done": + _set_lvol_online(task) + try: + backup = db.get_backup_by_id(backup_id) + backup_events.backup_restore_completed( + task.cluster_id, node_id, backup, lvol_name) + except KeyError: + logger.warning( + f"Backup {backup_id} no longer exists, " + f"skipping restore-completed event for {lvol_name}") + task.function_result = f"Restore completed: {lvol_name}" return - if stat and isinstance(stat, dict): - state = stat.get("transfer_state", "") - if state == "Done": - _set_lvol_online(task) - try: - backup = db.get_backup_by_id(backup_id) - backup_events.backup_restore_completed( - task.cluster_id, node_id, backup, lvol_name) - except KeyError: - pass - task.function_result = f"Restore completed: {lvol_name}" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - elif state == "Failed": - fail_count = task.function_params.get("fail_count", 0) + 1 - task.function_params["fail_count"] = fail_count - reason = f"S3 transfer failed on data plane (attempt {fail_count})" - task.function_result = reason - if fail_count >= 3: - _set_lvol_restore_failed(task, reason) - try: - backup = db.get_backup_by_id(backup_id) - backup_events.backup_restore_failed( - task.cluster_id, node_id, backup, lvol_name, reason) - except KeyError: - logger.warning( - "Backup %s not found in DB; restore-failed event skipped for lvol %s", - backup_id, lvol_name) - task.status = JobSchedule.STATUS_DONE - else: - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - elif state == "No process": - task.function_params["recovery_started"] = False - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - else: - # "In progress" — still running, retry later - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - else: - # Unexpected response — retry - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) + if state == "Failed": + fail_count = task.function_params.get("fail_count", 0) + 1 + task.function_params["fail_count"] = fail_count + reason = f"S3 transfer failed on data plane (attempt {fail_count})" + if fail_count < 3: + raise TaskRetry(reason) + + _set_lvol_restore_failed(task, reason) + try: + backup = db.get_backup_by_id(backup_id) + backup_events.backup_restore_failed( + task.cluster_id, node_id, backup, lvol_name, reason) + except KeyError: + logger.warning( + "Backup %s not found in DB; restore-failed event skipped for lvol %s", + backup_id, lvol_name) + raise TaskAbort(reason) + + if state == "No process": + task.function_params["recovery_started"] = False + raise TaskDefer("No process, restarting recovery") + + raise TaskDefer("Restore in progress") def _run_merge(task): keep_backup_id = task.function_params.get("keep_backup_id") old_backup_id = task.function_params.get("old_backup_id") - merge_started = task.function_params.get("merge_started", False) try: keep_backup = db.get_backup_by_id(keep_backup_id) old_backup = db.get_backup_by_id(old_backup_id) except KeyError as e: - task.function_result = str(e) - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return - - try: - snode = db.get_storage_node_by_id(keep_backup.node_id) - except KeyError: - task.function_result = f"Node {keep_backup.node_id} not found" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return - - if snode.status != StorageNode.STATUS_ONLINE: - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskAbort(str(e)) + snode = _online_node(keep_backup.node_id) rpc_client = snode.rpc_client(timeout=30) - if not merge_started: + if not task.function_params.get("merge_started", False): try: - ret = rpc_client.bdev_lvol_s3_merge(keep_backup.s3_id, old_backup.s3_id, cluster_batch=16, lvs_name=snode.lvstore) - if not ret: - task.function_result = "bdev_lvol_s3_merge RPC failed" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + ret = rpc_client.bdev_lvol_s3_merge( + keep_backup.s3_id, old_backup.s3_id, cluster_batch=16, lvs_name=snode.lvstore) except RPCException as e: - task.function_result = f"RPC error: {e}" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskRetry(f"RPC error: {e}") + if not ret: + raise TaskRetry("bdev_lvol_s3_merge RPC failed") task.function_params["merge_started"] = True - task.write_to_db(db.kv_store) # Give the data plane time to complete the merge before finalizing - return + raise TaskDefer("Merge started") # The merge RPC is synchronous on the data plane — once it returned # successfully, the S3 data has been merged. Finalize: update the @@ -361,115 +263,78 @@ def _run_merge(task): old_backup.write_to_db() task.function_result = "Merge completed" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) logger.info(f"Merge completed: {old_backup_id} merged into {keep_backup_id}") -def _terminate_task(task, reason): - """Terminate a backup/restore/merge task and finalize its resource. +_HANDLERS = { + JobSchedule.FN_BACKUP: _run_backup, + JobSchedule.FN_BACKUP_RESTORE: _run_restore, + JobSchedule.FN_BACKUP_MERGE: _run_merge, +} + + +def process_task(task): + cluster = db.get_cluster_by_id(task.cluster_id) + backup_timeout_sec = getattr(cluster, 'backup_timeout_seconds', 0) or _DEFAULT_BACKUP_TIMEOUT_SEC + elapsed = int(time.time()) - task.date if task.date else 0 + if elapsed > backup_timeout_sec: + raise TaskAbort(f"timeout after {elapsed}s") + + _HANDLERS[task.function_name](task) + + +def finalize_resource(task): + """Release the backup/restore/merge the task was driving, once it is over. - Shared by the time-based timeout and the max_retry ceiling so both stop - the task the same way instead of leaving it to loop. + Reached on every terminal path, so it is written to be a no-op when the + handler completed the resource itself and to only act when the task ended + with the resource still in flight — a timeout, the retry ceiling, an abort + or a cancellation, none of which the handler sees. """ + reason = task.function_result + if task.function_name == JobSchedule.FN_BACKUP: - bid = task.function_params.get("backup_id") - if bid: - try: - b = db.get_backup_by_id(bid) - if b.status in (Backup.STATUS_PENDING, Backup.STATUS_IN_PROGRESS): - _fail_backup(b, task, reason) - return - except KeyError: - pass + backup_id = task.function_params.get("backup_id") + if not backup_id: + return + try: + backup = db.get_backup_by_id(backup_id) + except KeyError: + return + if backup.status in (Backup.STATUS_PENDING, Backup.STATUS_IN_PROGRESS): + backup.status = Backup.STATUS_FAILED + backup.error_message = reason + backup.write_to_db() + backup_events.backup_failed(backup.cluster_id, backup.node_id, backup) + elif task.function_name == JobSchedule.FN_BACKUP_RESTORE: _set_lvol_restore_failed(task, reason) - elif task.function_name == JobSchedule.FN_BACKUP_MERGE: - old_bid = task.function_params.get("old_backup_id") - if old_bid: - try: - ob = db.get_backup_by_id(old_bid) - if ob.status == Backup.STATUS_MERGING: - # Merge did not finish; leave the old backup intact. - ob.status = Backup.STATUS_COMPLETED - ob.write_to_db() - except KeyError: - pass - - task.function_result = reason - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - - -def process_task(task, cl): - """Advance a single backup task by one step, or terminate it. - - Terminates on cancellation, on the time-based timeout, or once the - max_retry ceiling is reached — the last is what stops a backup that keeps - crashing the data plane from re-issuing its RPC forever. - """ - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return - # Time-based backstop for a task that is stuck but not erroring (default 4h). - backup_timeout_sec = getattr(cl, 'backup_timeout_seconds', 0) or 14400 - elapsed = int(time.time()) - task.date if task.date else 0 - if elapsed > backup_timeout_sec: - _terminate_task(task, f"timeout after {elapsed}s") - return + elif task.function_name == JobSchedule.FN_BACKUP_MERGE: + old_backup_id = task.function_params.get("old_backup_id") + if not old_backup_id: + return + try: + old_backup = db.get_backup_by_id(old_backup_id) + except KeyError: + return + if old_backup.status == Backup.STATUS_MERGING: + # Merge did not finish; leave the old backup intact. + old_backup.status = Backup.STATUS_COMPLETED + old_backup.write_to_db() - # Retry ceiling: every other task runner enforces this. Without it a task - # whose step keeps failing (e.g. an RPC that crashes SPDK) loops until the - # timeout, re-triggering the failure each cycle. max_retry <= 0 means the - # task is intentionally unbounded and only the timeout applies. - if task.max_retry > 0 and task.retry >= task.max_retry: - _terminate_task(task, f"max retry reached ({task.retry}/{task.max_retry})") - return - try: - if task.function_name == JobSchedule.FN_BACKUP: - _run_backup(task) - elif task.function_name == JobSchedule.FN_BACKUP_RESTORE: - _run_restore(task) - elif task.function_name == JobSchedule.FN_BACKUP_MERGE: - _run_merge(task) - except Exception as e: - logger.error(f"Error running backup task {task.uuid}: {e}") - # Increment retry so the task eventually reaches max_retry - # instead of looping forever on non-RPCException errors - task.retry += 1 - task.function_result = f"Unhandled error: {e}" - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) +SPEC = RunnerSpec( + name="tasks-runner-backup", + function_names=list(_HANDLERS), + handler=process_task, + on_finish=finalize_resource, + is_eligible=lambda task, cluster: cluster.status != Cluster.STATUS_IN_ACTIVATION, +) def main(): - logger.info("Starting backup tasks runner...") - while True: - try: - db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue - clusters = db.get_clusters() - for cl in clusters: - if cl.status == Cluster.STATUS_IN_ACTIVATION: - continue - - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - for task in tasks: - if task.status == JobSchedule.STATUS_DONE or task.canceled: - continue - - # Re-fetch task for freshness - task = db.get_task_by_id(task.uuid) - process_task(task, cl) - - time.sleep(constants.TASK_EXEC_INTERVAL_SEC) + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_batch_migration.py b/simplyblock_core/services/tasks_runner_batch_migration.py index 6890a41805..afb8e8a05c 100644 --- a/simplyblock_core/services/tasks_runner_batch_migration.py +++ b/simplyblock_core/services/tasks_runner_batch_migration.py @@ -1307,12 +1307,7 @@ def main(): logger.info("Starting Batch Migration orchestrator task runner...") while True: - try: - clusters = db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue + clusters = db.get_clusters() if not clusters: logger.error("No clusters found!") diff --git a/simplyblock_core/services/tasks_runner_cluster_expand.py b/simplyblock_core/services/tasks_runner_cluster_expand.py index c9c17af6dc..70c4f39d96 100644 --- a/simplyblock_core/services/tasks_runner_cluster_expand.py +++ b/simplyblock_core/services/tasks_runner_cluster_expand.py @@ -1,8 +1,5 @@ # coding=utf-8 -import time - - -from simplyblock_core import db_controller, utils, constants +from simplyblock_core import db_controller, utils from simplyblock_core.controllers import tasks_controller from simplyblock_core.controllers.cluster_expansion.executor import ( integrate_new_node_into_cluster, @@ -14,6 +11,12 @@ ) from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.nvme_device import NVMeDevice +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskAbort, + TaskRetry, + serve, +) logger = utils.get_logger(__name__) @@ -23,132 +26,54 @@ def process_task(task): - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return False - - # max_retry < 0 means never give up (the plan must complete — an - # abandoned half-moved topology strands stale sec/tert pointers); - # cancel the task to stop it deliberately. - if 0 <= task.max_retry <= task.retry: - task.function_result = "max retry reached" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True - new_node_id = task.function_params.get("new_node_id") if not new_node_id: - task.function_result = "missing new_node_id in function_params" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True - - if task.status != JobSchedule.STATUS_RUNNING: - task.status = JobSchedule.STATUS_RUNNING - task.write_to_db(db.kv_store) - - try: - cluster = db.get_cluster_by_id(task.cluster_id) - new_snode = db.get_storage_node_by_id(new_node_id) - - # Retry-by-resume: a prior attempt that aborted left expand_state at - # the failed move's cursor. Flip it back to in_progress so the - # orchestrator re-attempts that move instead of recomputing a fresh - # diff against a topology the aborted run may have partially mutated. - if (cluster.expand_state or {}).get("phase") == EXPAND_PHASE_ABORTED: - cluster.expand_state = expand_state_rearm(cluster.expand_state) - cluster.write_to_db() - - integrate_new_node_into_cluster( - cluster, new_snode, db_controller=db, - manage_cluster_status=True) - - # integrate_new_node_into_cluster returns only on success; the - # orchestrator marks expand_state completed. Re-read to confirm. - cluster = db.get_cluster_by_id(task.cluster_id) - phase = (cluster.expand_state or {}).get("phase") - if phase == EXPAND_PHASE_COMPLETED: - # Queue new-device migration now that the rotation has landed and - # the cluster is back to ACTIVE — tasks are created against the - # post-rotation lvstore_stack (which includes the newcomer's - # primary distr). Mirrors the trigger the non-expansion add path - # runs inside add_node. - new_snode = db.get_storage_node_by_id(new_node_id) - for dev in new_snode.nvme_devices: - if dev.status == NVMeDevice.STATUS_ONLINE: - tasks_controller.add_new_device_mig_task(dev.get_id()) - task.function_result = f"expansion complete: {new_node_id}" - task.status = JobSchedule.STATUS_DONE - else: - # Shouldn't happen (no exception but not completed) — suspend and - # retry rather than silently mark done. - task.function_result = f"unexpected phase after run: {phase!r}" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return True - except Exception as e: - logger.error(e) - task.function_result = f"attempt {task.retry + 1} failed: {e}" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False + raise TaskAbort("missing new_node_id in function_params") + + cluster = db.get_cluster_by_id(task.cluster_id) + new_snode = db.get_storage_node_by_id(new_node_id) + + # Retry-by-resume: a prior attempt that aborted left expand_state at + # the failed move's cursor. Flip it back to in_progress so the + # orchestrator re-attempts that move instead of recomputing a fresh + # diff against a topology the aborted run may have partially mutated. + if (cluster.expand_state or {}).get("phase") == EXPAND_PHASE_ABORTED: + cluster.expand_state = expand_state_rearm(cluster.expand_state) + cluster.write_to_db() + + integrate_new_node_into_cluster( + cluster, new_snode, db_controller=db, + manage_cluster_status=True) + + # integrate_new_node_into_cluster returns only on success; the + # orchestrator marks expand_state completed. Re-read to confirm. + cluster = db.get_cluster_by_id(task.cluster_id) + phase = (cluster.expand_state or {}).get("phase") + if phase != EXPAND_PHASE_COMPLETED: + raise TaskRetry(f"unexpected phase after run: {phase!r}") + + # Queue new-device migration now that the rotation has landed and + # the cluster is back to ACTIVE — tasks are created against the + # post-rotation lvstore_stack (which includes the newcomer's + # primary distr). Mirrors the trigger the non-expansion add path + # runs inside add_node. + new_snode = db.get_storage_node_by_id(new_node_id) + for dev in new_snode.nvme_devices: + if dev.status == NVMeDevice.STATUS_ONLINE: + tasks_controller.add_new_device_mig_task(dev.get_id()) + + task.function_result = f"expansion complete: {new_node_id}" + + +SPEC = RunnerSpec( + name="tasks-runner-cluster-expand", + function_names=[JobSchedule.FN_CLUSTER_EXPAND], + handler=process_task, +) def main(): - logger.info("Starting Tasks runner cluster expand...") - while True: - try: - clusters = db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue - if not clusters: - logger.error("No clusters found!") - else: - for cl in clusters: - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - for task in tasks: - delay_seconds = constants.TASK_EXEC_INTERVAL_SEC - if task.function_name != JobSchedule.FN_CLUSTER_EXPAND: - continue - # Per-task isolation: a crash in process_task must not - # escape to the outer loop and kill the runner. - try: - while task.status != JobSchedule.STATUS_DONE: - # Re-fetch: the task may have been cancelled. - task = db.get_task_by_id(task.uuid) - # Lease gate: skip a task another live runner owns. - if not tasks_controller.claim_task(task): - logger.info( - f"Cluster-expand task {task.uuid} owned by " - f"another runner host; skipping") - break - with tasks_controller.task_lease_heartbeat(task): - res = process_task(task) - if res: - if task.status == JobSchedule.STATUS_DONE: - break - else: - # Cap the exponential backoff so a permanently - # failing expansion can't grow the sleep without - # bound. - delay_seconds = min( - delay_seconds * 2, - constants.RESTART_TASK_EXEC_INTERVAL_MAX_SEC, - ) - time.sleep(delay_seconds) - except Exception as e: - logger.error( - f"Cluster-expand task {task.uuid} processing " - f"crashed: {e}") - logger.exception(e) - - time.sleep(constants.TASK_EXEC_INTERVAL_SEC) + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_failed_migration.py b/simplyblock_core/services/tasks_runner_failed_migration.py index 4b591d3b22..d95efdbab3 100644 --- a/simplyblock_core/services/tasks_runner_failed_migration.py +++ b/simplyblock_core/services/tasks_runner_failed_migration.py @@ -1,172 +1,86 @@ # coding=utf-8 import time -from datetime import datetime from simplyblock_core import db_controller, utils, constants from simplyblock_core.controllers import tasks_controller, device_controller -from simplyblock_core.models.cluster import Cluster from simplyblock_core.models.job_schedule import JobSchedule +from simplyblock_core.models.storage_node import StorageNode +from simplyblock_core.services import migration_task_common as mig +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskAbort, + TaskRetry, + serve, +) +logger = utils.get_logger(__name__) - -from simplyblock_core.models.storage_node import StorageNode +# get DB controller +db = db_controller.DBController() def task_runner(task): - try: - snode = db.get_storage_node_by_id(task.node_id) - except KeyError: - task.status = JobSchedule.STATUS_DONE - task.function_result = f"Node not found: {task.node_id}" - task.write_to_db(db.kv_store) - return True - - cluster = db.get_cluster_by_id(task.cluster_id) - # Node removal DEPENDS on this runner: _decommission_node_devices blocks - # until every data device reaches FAILED_AND_MIGRATED, which only this - # runner sets — refusing IN_SHRINK deadlocks removal against its own status. - if cluster.status not in Cluster.OPERABLE_STATUSES: - task.function_result = "cluster is not active, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False - - # Expansion-first ordering: defer while a cluster expansion is open - # (see tasks_controller.defer_task_for_expansion). - if tasks_controller.defer_task_for_expansion(task): - return False - - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True - - if task.status in [JobSchedule.STATUS_NEW ,JobSchedule.STATUS_SUSPENDED]: - if task.status == JobSchedule.STATUS_NEW: - for node in db.get_storage_nodes_by_cluster_id(task.cluster_id): - if node.online_since: - try: - diff = datetime.now() - datetime.fromisoformat(node.online_since) - if diff.total_seconds() < 60: - task.function_result = "node is online < 1 min, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False - except Exception as e: - logger.error(f"Failed to get online since: {e}") - - task.status = JobSchedule.STATUS_RUNNING - task.write_to_db(db.kv_store) + snode = mig.require_node(task) + mig.require_active_cluster(task) + + if not mig.migration_started(task): + if not mig.nodes_settled(task.cluster_id): + raise TaskRetry("node is online < 1 min, retrying") if snode.status != StorageNode.STATUS_ONLINE: - task.function_result = "node is not online, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False - - active_f_task = tasks_controller.get_new_device_mig_task_for_device(task.cluster_id) - if active_f_task: - msg = "dev expansion task found, retry" - logger.info(msg) - task.function_result = msg - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False + raise TaskRetry("node is not online, retrying") + + if tasks_controller.get_new_device_mig_task_for_device(task.cluster_id): + raise TaskRetry("dev expansion task found, retry") rpc_client = snode.rpc_client(timeout=5, retry=2) - if "migration" not in task.function_params: + + if not mig.migration_started(task): try: device = db.get_storage_device_by_id(task.device_id) except KeyError: - task.status = JobSchedule.STATUS_DONE - task.function_result = "Device not found" - task.write_to_db(db.kv_store) - return True - - distr_name = task.function_params["distr_name"] - - qos_high_priority = False - if db.get_cluster_by_id(snode.cluster_id).is_qos_set(): - qos_high_priority = True - try: - rsp = rpc_client.distr_migration_failure_start( - distr_name, device.cluster_device_order, qos_high_priority, job_size=constants.MIG_JOB_SIZE, jobs=constants.MIG_PARALLEL_JOBS) - except Exception as e: - logger.error(e) - rsp = False - if not rsp: - logger.error(f"Failed to start device migration task, storage_ID: {device.cluster_device_order}") - task.function_result = "Failed to start device migration task" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False - - task.function_params['migration'] = {"name": distr_name} - task.write_to_db(db.kv_store) + raise TaskAbort("Device not found") + + started = mig.start_migration(task, lambda: rpc_client.distr_migration_failure_start( + task.function_params["distr_name"], device.cluster_device_order, + mig.qos_high_priority(snode.cluster_id), + job_size=constants.MIG_JOB_SIZE, jobs=constants.MIG_PARALLEL_JOBS)) + if started is None: + raise TaskAbort("canceled while starting migration") + task = started time.sleep(3) - try: - if "migration" in task.function_params: - mig_info = task.function_params["migration"] - res = rpc_client.distr_migration_status(**mig_info) - out = utils.handle_task_result(task, res) - dev_failed_task = tasks_controller.get_failed_device_mig_task(task.cluster_id, task.device_id) - if not dev_failed_task: - device_controller.device_set_failed_and_migrated(task.device_id) + mig.poll_migration(task, rpc_client) - return out - except Exception as e: - logger.error("Failed to get migration task status") - logger.exception(e) - task.function_result = "Failed to get migration status" - task.retry += 1 - task.write_to_db(db.kv_store) - return False +def tag_device_migrated(task): + """The device's data now lives elsewhere, so record that on the device. + Runs from the driver's on_finish rather than inline: the check below asks + whether any failed-migration task for this device is still open, and this + task only stops counting as open once it has been written DONE. Inline it + would always find itself and never tag anything — which the old code got + away with only because its handle_task_result wrote DONE first. + """ + if not mig.migration_started(task): + return # never got as far as moving any data + if tasks_controller.get_failed_device_mig_task(task.cluster_id, task.device_id): + return + device_controller.device_set_failed_and_migrated(task.device_id) -logger = utils.get_logger(__name__) -# get DB controller -db = db_controller.DBController() +SPEC = RunnerSpec( + name="tasks-runner-failed-migration", + function_names=[JobSchedule.FN_FAILED_DEV_MIG], + handler=task_runner, + on_finish=tag_device_migrated, + is_eligible=mig.sibling_eligibility, + interval=3, +) def main(): - logger.info("Starting Tasks runner...") - while True: - try: - db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue - time.sleep(3) - clusters = db.get_clusters() - if not clusters: - logger.error("No clusters found!") - else: - for cl in clusters: - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - for task in tasks: - if task.function_name == JobSchedule.FN_FAILED_DEV_MIG: - if task.status in [JobSchedule.STATUS_NEW, JobSchedule.STATUS_SUSPENDED]: - active_task = tasks_controller.get_active_node_mig_task( - task.cluster_id, task.node_id, task.function_params["distr_name"]) - if active_task: - logger.info("task found on same node, retry") - continue - if task.status != JobSchedule.STATUS_DONE: - # get new task object because it could be changed from cancel task - task = db.get_task_by_id(task.uuid) - res = task_runner(task) - if not res: - time.sleep(3) + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_fdb_backup.py b/simplyblock_core/services/tasks_runner_fdb_backup.py index bb2b565902..d94310fbf0 100644 --- a/simplyblock_core/services/tasks_runner_fdb_backup.py +++ b/simplyblock_core/services/tasks_runner_fdb_backup.py @@ -1,12 +1,9 @@ # coding=utf-8 -import time - - -from simplyblock_core import db_controller, utils, constants -from simplyblock_core.controllers import fdb_backup_controller -from simplyblock_core.controllers import fdb_backup_events +from simplyblock_core import db_controller, utils +from simplyblock_core.controllers import fdb_backup_controller, fdb_backup_events from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.cluster import Cluster +from simplyblock_core.services.task_runner_base import RunnerSpec, TaskRetry, serve logger = utils.get_logger(__name__) @@ -15,54 +12,34 @@ def process_fdb_backup_task(task): - task = db.get_task_by_id(task.uuid) - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return + if not fdb_backup_controller.create_backup(task.cluster_id): + raise TaskRetry("failed to create backup") - if task.retry >= task.max_retry: - task.function_result = "max retry reached, stopping task" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - fdb_backup_events.fdb_backup_failed(task.cluster_id, task.uuid) - return + task.function_result = "Backup created" - if task.status != JobSchedule.STATUS_RUNNING: - task.status = JobSchedule.STATUS_RUNNING - task.write_to_db(db.kv_store) - ret = fdb_backup_controller.create_backup(task.cluster_id) - if ret: - task.function_result = "Backup created" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - else: - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) +def report_exhausted(task): + """Announce a backup that gave up, in the cluster event log. + on_finish rather than on_failure: what an operator has to see is the task + running out of retries, not each attempt, and the driver finishes a task on + its ceiling without ever entering the handler. + """ + if 0 <= task.max_retry <= task.retry: + fdb_backup_events.fdb_backup_failed(task.cluster_id, task.uuid) -def main(): - logger.info("Starting Tasks runner fdb backup...") - while True: - clusters = db.get_clusters() - if not clusters: - logger.error("No clusters found!") - else: - for cl in clusters: - if cl.status == Cluster.STATUS_IN_ACTIVATION: - continue +SPEC = RunnerSpec( + name="tasks-runner-fdb-backup", + function_names=[JobSchedule.FN_FDB_BACKUP], + handler=process_fdb_backup_task, + on_finish=report_exhausted, + is_eligible=lambda task, cluster: cluster.status != Cluster.STATUS_IN_ACTIVATION, +) - tasks = db.get_job_tasks(cl.get_id()) - for task in tasks: - if task.status != JobSchedule.STATUS_DONE: - if task.function_name == JobSchedule.FN_FDB_BACKUP: - process_fdb_backup_task(task) - time.sleep(constants.TASK_EXEC_INTERVAL_SEC) +def main(): + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_jc_comp.py b/simplyblock_core/services/tasks_runner_jc_comp.py index 64580da519..af28c2e638 100644 --- a/simplyblock_core/services/tasks_runner_jc_comp.py +++ b/simplyblock_core/services/tasks_runner_jc_comp.py @@ -1,13 +1,17 @@ # coding=utf-8 -import time - - from simplyblock_core import db_controller, utils from simplyblock_core.controllers import tasks_controller from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.cluster import Cluster from simplyblock_core.models.storage_node import StorageNode from simplyblock_core.release_upgrades import jc_compression_upgrade +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskAbort, + TaskDefer, + TaskRetry, + serve, +) logger = utils.get_logger(__name__) @@ -15,123 +19,51 @@ db = db_controller.DBController() +def process_task(task): + # Release-upgrade guard (remove with the jc_compression_upgrade plugin): + # resumes are held until `cluster upgrade-complete`. + if jc_compression_upgrade.resume_is_held(db.get_cluster_by_id(task.cluster_id)): + raise TaskDefer("JC compression resume held: cluster upgrade in progress") + + try: + node = db.get_storage_node_by_id(task.node_id) + except KeyError: + raise TaskAbort("node not found") + + if node.status != StorageNode.STATUS_ONLINE: + raise TaskDefer(f"Node is {node.status}, retry task") + + if tasks_controller.get_active_node_tasks(task.cluster_id, task.node_id): + raise TaskDefer("Task found on same node") + + if any(n.status != StorageNode.STATUS_ONLINE + for n in db.get_storage_nodes_by_cluster_id(node.cluster_id)): + raise TaskDefer("Not all nodes are online, can not resume JC compression") + + logger.info("no task found on same node, resuming compression") + jm_vuid = task.function_params.get("jm_vuid", node.jm_vuid) + ret, err = node.rpc_client(timeout=5, retry=2).jc_suspend_compression( + jm_vuid=jm_vuid, suspend=False) + + if not ret: + if err: + raise TaskAbort(f"JC {node.jm_vuid} compression not needed") + raise TaskRetry("JC comp resume failed, retry task") + + task.function_result = f"JC {node.jm_vuid} compression resumed on node" + + +SPEC = RunnerSpec( + name="tasks-runner-jc-comp", + function_names=[JobSchedule.FN_JC_COMP_RESUME], + handler=process_task, + is_eligible=lambda task, cluster: cluster.status != Cluster.STATUS_IN_ACTIVATION, + interval=60, +) + + def main(): - logger.info("Starting Tasks runner...") - while True: - try: - db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue - clusters = db.get_clusters() - if not clusters: - logger.error("No clusters found!") - else: - for cl in clusters: - if cl.status == Cluster.STATUS_IN_ACTIVATION: - continue - - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - for task in tasks: - - if task.function_name == JobSchedule.FN_JC_COMP_RESUME: - if task.status != JobSchedule.STATUS_DONE: - - # get new task object because it could be changed from cancel task - task = db.get_task_by_id(task.uuid) - - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - continue - - # Release-upgrade guard (remove with the - # jc_compression_upgrade plugin): resumes are - # held until `cluster upgrade-complete`. - if jc_compression_upgrade.resume_is_held(cl): - msg = "JC compression resume held: cluster upgrade in progress" - logger.info(msg) - # Only write when something actually changes. The - # hold lasts until `cluster upgrade-complete` runs, - # so an unconditional write here costs one DB write - # per held task per poll for the whole upgrade. - if (task.status != JobSchedule.STATUS_SUSPENDED - or task.function_result != msg): - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - continue - - if task.retry >= task.max_retry: - task.function_result = "max retry reached, stopping task" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - continue - - try: - node = db.get_storage_node_by_id(task.node_id) - except KeyError: - task.function_result = "node not found" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - continue - - if node.status != StorageNode.STATUS_ONLINE: - msg = f"Node is {node.status}, retry task" - logger.info(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - continue - - node_task = tasks_controller.get_active_node_tasks(task.cluster_id, task.node_id) - if node_task: - msg="Task found on same node" - logger.info(msg) - task.retry += 1 - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - else: - logger.info("no task found on same node, resuming compression") - node = db.get_storage_node_by_id(task.node_id) - for n in db.get_storage_nodes_by_cluster_id(node.cluster_id): - if n.status != StorageNode.STATUS_ONLINE: - msg = "Not all nodes are online, can not resume JC compression" - logger.info(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - continue - - rpc_client = node.rpc_client(timeout=5, retry=2) - jm_vuid = node.jm_vuid - if "jm_vuid" in task.function_params: - jm_vuid = task.function_params["jm_vuid"] - try: - ret, err = rpc_client.jc_suspend_compression(jm_vuid=jm_vuid, suspend=False) - except Exception as e: - logger.error(e) - continue - if ret: - task.function_result = f"JC {node.jm_vuid} compression resumed on node" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - elif err: - task.function_result = f"JC {node.jm_vuid} compression not needed" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - else: - msg = "JC comp resume failed, retry task" - logger.info(msg) - task.retry += 1 - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - - time.sleep(60) + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_lvol_migration.py b/simplyblock_core/services/tasks_runner_lvol_migration.py index f6ac117b2c..a26c02059c 100644 --- a/simplyblock_core/services/tasks_runner_lvol_migration.py +++ b/simplyblock_core/services/tasks_runner_lvol_migration.py @@ -3025,7 +3025,7 @@ def task_runner(task): # Expansion-first ordering: defer while a cluster expansion is open — # even between the expand task's retries, when the cluster status is - # momentarily ACTIVE (see tasks_controller.defer_task_for_expansion). + # momentarily ACTIVE (see migration_task_common.require_active_cluster). if tasks_controller.get_active_cluster_expand_task(task.cluster_id): return _suspend_task( task, migration, "cluster expansion in progress, deferring", @@ -3874,12 +3874,6 @@ def main(): logger.info("Starting LVol Migration task runner...") while True: - try: - db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue clusters = db.get_clusters() if not clusters: logger.error("No clusters found!") diff --git a/simplyblock_core/services/tasks_runner_migration.py b/simplyblock_core/services/tasks_runner_migration.py index aeb5781b4d..a3a234eeb5 100644 --- a/simplyblock_core/services/tasks_runner_migration.py +++ b/simplyblock_core/services/tasks_runner_migration.py @@ -1,160 +1,83 @@ # coding=utf-8 -import time -from datetime import datetime, timezone - from simplyblock_core import db_controller, utils, constants from simplyblock_core.controllers import tasks_events, tasks_controller, lvol_controller -from simplyblock_core.models.cluster import Cluster from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.nvme_device import NVMeDevice from simplyblock_core.models.storage_node import StorageNode from simplyblock_core.release_upgrades import jc_compression_upgrade +from simplyblock_core.services import migration_task_common as mig +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskAbort, + TaskDefer, + TaskRetry, + serve, +) logger = utils.get_logger(__name__) -MIGRATION_WAIT_UNAVAILABLE_KEY = "wait_unavailable_before_retry" +# get DB controller +db = db_controller.DBController() +MIGRATION_WAIT_UNAVAILABLE_KEY = mig.MIGRATION_WAIT_UNAVAILABLE_KEY -def _cluster_unavailable_state(cluster_id): - unavailable = [] + +def _online_device_count(cluster_id, primaries_only=False): + online = 0 for node in db.get_storage_nodes_by_cluster_id(cluster_id): - if node.status in [StorageNode.STATUS_IN_CREATION, StorageNode.STATUS_REMOVED]: + if primaries_only and node.is_secondary_node: continue - if node.status not in [StorageNode.STATUS_ONLINE, StorageNode.STATUS_SUSPENDED]: - unavailable.append(f"node:{node.get_id()}") for dev in node.nvme_devices: - if dev.status in [NVMeDevice.STATUS_REMOVED, NVMeDevice.STATUS_FAILED_AND_MIGRATED]: - continue - if dev.status != NVMeDevice.STATUS_ONLINE: - unavailable.append(f"dev:{dev.get_id()}") - return sorted(unavailable) - + if dev.status == NVMeDevice.STATUS_ONLINE: + online += 1 + return online -def _migration_retry_allowed(task, unavailable): - previous = sorted(task.function_params.get(MIGRATION_WAIT_UNAVAILABLE_KEY, [])) - if not unavailable: - if previous: - task.function_params.pop(MIGRATION_WAIT_UNAVAILABLE_KEY, None) - task.write_to_db(db.kv_store) - return True - recovered = set(previous) - set(unavailable) - if previous and recovered: - task.function_params[MIGRATION_WAIT_UNAVAILABLE_KEY] = unavailable - task.write_to_db(db.kv_store) - logger.info( - "Migration retry allowed after recovery event for task %s: %s", - task.uuid, - sorted(recovered), - ) - return True +def _wait_for_cluster_recovery(task): + """Gate a not-yet-started migration on the cluster being whole enough. - task.function_params[MIGRATION_WAIT_UNAVAILABLE_KEY] = unavailable - task.function_result = ( - "waiting for unavailable nodes/devices to recover before restarting migration: " - f"{unavailable}" - ) - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False + Whether waiting costs a retry depends on why: with nothing unavailable the + hold-up is transient and counts against the budget, but while nodes or + devices are down it is the recovery we are waiting on, and burning retries + against it would terminate the migration before the cluster came back. + """ + unavailable = mig.cluster_unavailable_state(task.cluster_id) + if not unavailable: + raise TaskRetry(task.function_result or "waiting to start migration, retrying") + mig.require_recovery_progress(task, unavailable) def task_runner(task): + snode = mig.require_node(task) - task = db.get_task_by_id(task.uuid) - try: - snode = db.get_storage_node_by_id(task.node_id) - except KeyError: - task.status = JobSchedule.STATUS_DONE - task.function_result = f"Node not found: {task.node_id}" - task.write_to_db(db.kv_store) - return True - - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True + # Ahead of the online gate, which REMOVED also fails: a removed node never + # comes back, so deferring on it would keep the task alive forever. + if snode.status == StorageNode.STATUS_REMOVED: + raise TaskAbort(f"Node removed: {task.node_id}") if snode.status not in [StorageNode.STATUS_ONLINE, StorageNode.STATUS_SUSPENDED]: task.function_result = "node is not online, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - unavailable = _cluster_unavailable_state(task.cluster_id) - if not unavailable: - task.retry += 1 - task.write_to_db(db.kv_store) - else: - _migration_retry_allowed(task, unavailable) - return False - - if snode.status == StorageNode.STATUS_REMOVED: - task.status = JobSchedule.STATUS_DONE - task.function_result = f"Node found in r: {task.node_id}" - task.write_to_db(db.kv_store) - return True - - cluster = db.get_cluster_by_id(task.cluster_id) - if cluster.status not in Cluster.OPERABLE_STATUSES: - task.function_result = "cluster is not active, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False + _wait_for_cluster_recovery(task) + raise TaskDefer("node is not online, retrying") - # Expansion-first ordering: no data migration runs while a cluster - # expansion is open — even between the expand task's retries, when the - # cluster status is momentarily ACTIVE (see defer_task_for_expansion). - if tasks_controller.defer_task_for_expansion(task): - return False + mig.require_active_cluster(task) if tasks_controller.get_active_lvol_migration(task.node_id): - task.function_result = "LVol migration tasks found, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False - - if task.status in [JobSchedule.STATUS_NEW, JobSchedule.STATUS_SUSPENDED]: - current_online_devices = 0 - unavailable = _cluster_unavailable_state(task.cluster_id) - for node in db.get_storage_nodes_by_cluster_id(task.cluster_id): - if node.is_secondary_node: # pass - continue - for dev in node.nvme_devices: - if dev.status == NVMeDevice.STATUS_ONLINE: - current_online_devices += 1 - if node.status == StorageNode.STATUS_ONLINE and node.online_since: - try: - diff = datetime.now(timezone.utc) - datetime.fromisoformat(node.online_since) - if diff.total_seconds() < 60: - task.function_result = "node is online < 1 min, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False - except Exception as e: - logger.error(f"Failed to get online since: {e}") - - migration_devices = 0 - if "migration_devices" in task.function_params: - migration_devices = task.function_params["migration_devices"] - - if current_online_devices < migration_devices: - task.function_result = f"only {current_online_devices} devices online, waiting for more devices to be online" - task.status = JobSchedule.STATUS_SUSPENDED - if not unavailable: - task.retry += 1 - task.write_to_db(db.kv_store) - else: - _migration_retry_allowed(task, unavailable) - return False + raise TaskRetry("LVol migration tasks found, retrying") - if not _migration_retry_allowed(task, unavailable): - return False + if not mig.migration_started(task): + if not mig.nodes_settled(task.cluster_id, primaries_only=True): + raise TaskDefer("node is online < 1 min, retrying") - task.status = JobSchedule.STATUS_RUNNING - task.function_result = "" - task.write_to_db(db.kv_store) + online_devices = _online_device_count(task.cluster_id, primaries_only=True) + wanted = task.function_params.get("migration_devices", 0) + if online_devices < wanted: + task.function_result = (f"only {online_devices} devices online, waiting for " + f"more devices to be online") + _wait_for_cluster_recovery(task) + raise TaskDefer(task.function_result) + mig.require_recovery_progress(task, mig.cluster_unavailable_state(task.cluster_id)) rpc_client = snode.rpc_client(timeout=5, retry=2) @@ -162,184 +85,138 @@ def task_runner(task): # Migration IO triggers auto-leader promotion in the data plane, so # starting migration on a non-leader causes a split-brain write conflict. if not snode.is_secondary_node and not lvol_controller.is_node_leader(snode, snode.lvstore): - msg = f"Node {snode.get_id()} is not the leader for {snode.lvstore}, deferring migration" - logger.warning(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False - - if "migration" not in task.function_params: - current_online_devices = 0 - for node in db.get_storage_nodes_by_cluster_id(task.cluster_id): - for dev in node.nvme_devices: - if dev.status == NVMeDevice.STATUS_ONLINE: - current_online_devices += 1 - - distr_name = task.function_params["distr_name"] - - qos_high_priority = False - if db.get_cluster_by_id(snode.cluster_id).is_qos_set(): - qos_high_priority = True - try: - rsp = rpc_client.distr_migration_expansion_start(distr_name, qos_high_priority, job_size=constants.MIG_JOB_SIZE, - jobs=constants.MIG_PARALLEL_JOBS) - except Exception as e: - logger.error(e) - rsp = False - if not rsp: - msg = "Failed to start device migration task, retry later" - logger.error(msg) - task.function_result =msg - task.status = JobSchedule.STATUS_SUSPENDED - unavailable = _cluster_unavailable_state(task.cluster_id) - if not unavailable: - task.retry += 1 - task.write_to_db(db.kv_store) - else: - _migration_retry_allowed(task, unavailable) - return True - task.function_params['migration'] = {"name": distr_name} - task.function_params['migration_devices'] = current_online_devices - task.write_to_db(db.kv_store) - - try: - if "migration" in task.function_params: - allow_all_errors = False - for node in db.get_storage_nodes_by_cluster_id(task.cluster_id): - for dev in node.nvme_devices: - if dev.status in [NVMeDevice.STATUS_READONLY, NVMeDevice.STATUS_CANNOT_ALLOCATE, NVMeDevice.STATUS_FAILED]: - allow_all_errors = True - break - - mig_info = task.function_params["migration"] - res = rpc_client.distr_migration_status(**mig_info) - return utils.handle_task_result(task, res, allow_all_errors=allow_all_errors) - except Exception as e: - logger.error("Failed to get migration task status") - logger.exception(e) - task.function_result = "Failed to get migration status" - - task.retry += 1 - task.write_to_db(db.kv_store) - return False + raise TaskRetry(f"Node {snode.get_id()} is not the leader for {snode.lvstore}, " + f"deferring migration") + + if not mig.migration_started(task): + # Recorded alongside the start so a later poll can tell how much of the + # cluster the migration was sized against. + task.function_params["migration_devices"] = _online_device_count(task.cluster_id) + started = mig.start_migration(task, lambda: rpc_client.distr_migration_expansion_start( + task.function_params["distr_name"], mig.qos_high_priority(snode.cluster_id), + job_size=constants.MIG_JOB_SIZE, jobs=constants.MIG_PARALLEL_JOBS)) + if started is None: + raise TaskAbort("canceled while starting migration") + task = started + + mig.poll_migration(task, rpc_client, allow_all_errors=mig.allow_all_migration_errors( + task.cluster_id, [NVMeDevice.STATUS_READONLY, + NVMeDevice.STATUS_CANNOT_ALLOCATE, + NVMeDevice.STATUS_FAILED])) + + +def _is_eligible(task, cluster): + """One migration at a time per node and distr. + + Beyond the shared sibling rule, a device migration also waits behind a + SUSPENDED new-device migration for the same distr: that one is mid-flight + on the data plane even while its task is parked, and starting a second + migration over the same distr would collide with it. + """ + if mig.migration_started(task): + return True + distr_name = task.function_params.get("distr_name") + for sibling in db.get_job_tasks(task.cluster_id): + if sibling.function_name not in [JobSchedule.FN_FAILED_DEV_MIG, + JobSchedule.FN_DEV_MIG, + JobSchedule.FN_NEW_DEV_MIG]: + continue + if sibling.node_id != task.node_id or sibling.canceled: + continue + if sibling.function_params.get("distr_name") != distr_name: + continue + if sibling.status == JobSchedule.STATUS_RUNNING: + return False + if (sibling.status == JobSchedule.STATUS_SUSPENDED + and sibling.function_name == JobSchedule.FN_NEW_DEV_MIG): + return False + return True -# get DB controller -db = db_controller.DBController() +def update_master_tasks(cluster_id): + """Roll every master task's sub-task statuses up into it. -def update_master_task(task, cl): - master_task = None - tasks = {t.uuid: t for t in db.get_job_tasks(cl.get_id(), reverse=False)} - for t in tasks.values(): - if task.uuid in t.sub_tasks: - master_task = t - break + Per cycle rather than per sub-task attempt: the roll-up reads all of a + master's sub-tasks anyway, so running it once a cycle both covers every + status change (not only the attempts that finish a sub-task) and drops the + redundant re-computation each sibling used to trigger. + """ + tasks = {t.uuid: t for t in db.get_job_tasks(cluster_id, reverse=False)} + for master_task in list(tasks.values()): + if master_task.sub_tasks: + _roll_up(master_task, tasks) - def _set_master_task_status(master_task, status): - if master_task.status != status: - logger.info(f"_set_master_task_status: {status}") - master_task.status = status - master_task.function_result = status - master_task.write_to_db(db.kv_store) - tasks_events.task_updated(master_task) +def _roll_up(master_task, tasks): status_map = { JobSchedule.STATUS_DONE: 0, JobSchedule.STATUS_NEW: 0, JobSchedule.STATUS_SUSPENDED: 0, JobSchedule.STATUS_RUNNING: 0, } - if master_task: - for sub_task_id in master_task.sub_tasks: - sub_task = tasks[sub_task_id] - status_map[sub_task.status] = status_map.get(sub_task.status, 0) + 1 - - logger.info(f"master_task.sub_tasks: {len(master_task.sub_tasks)}") - logger.info(f"status_map: {status_map}") - - if status_map[JobSchedule.STATUS_DONE] == len(master_task.sub_tasks): # all tasks done - _set_master_task_status(master_task, JobSchedule.STATUS_DONE) - elif status_map[JobSchedule.STATUS_NEW] == len(master_task.sub_tasks): # all tasks new - _set_master_task_status(master_task, JobSchedule.STATUS_NEW) - elif status_map[JobSchedule.STATUS_SUSPENDED] == len(master_task.sub_tasks): # all tasks suspended - _set_master_task_status(master_task, JobSchedule.STATUS_SUSPENDED) - else: # set running - _set_master_task_status(master_task, JobSchedule.STATUS_RUNNING) - return True + for sub_task_id in master_task.sub_tasks: + sub_task = tasks.get(sub_task_id) + if sub_task is None: + return + status_map[sub_task.status] = status_map.get(sub_task.status, 0) + 1 + + total = len(master_task.sub_tasks) + for status in (JobSchedule.STATUS_DONE, JobSchedule.STATUS_NEW, JobSchedule.STATUS_SUSPENDED): + if status_map[status] == total: + rolled_up = status + break + else: + rolled_up = JobSchedule.STATUS_RUNNING + + if master_task.status == rolled_up: + return + logger.info(f"_set_master_task_status: {rolled_up}") + master_task.status = rolled_up + master_task.function_result = rolled_up + master_task.write_to_db(db.kv_store) + tasks_events.task_updated(master_task) + + +def resume_jc_compression(task): + """A finished migration frees the node for JC compression again.""" + if tasks_controller.get_active_node_tasks(task.cluster_id, task.node_id): + return + + # Release-upgrade guard (remove with the jc_compression_upgrade plugin): + # resumes are held until `cluster upgrade-complete`. + if jc_compression_upgrade.resume_is_held(db.get_cluster_by_id(task.cluster_id)): + logger.info("JC compression resume held: cluster upgrade in progress") + return + + logger.info("no task found on same node, resuming compression") + node = db.get_storage_node_by_id(task.node_id) + for peer in db.get_storage_nodes_by_cluster_id(node.cluster_id): + if peer.status not in [StorageNode.STATUS_ONLINE, StorageNode.STATUS_SUSPENDED]: + logger.warning("Not all nodes are online, can not resume JC compression") + try: + _, err = node.rpc_client(timeout=5, retry=2).jc_suspend_compression( + jm_vuid=node.jm_vuid, suspend=False) + if err: + logger.info("Failed to resume JC compression adding task...") + tasks_controller.add_jc_comp_resume_task(task.cluster_id, task.node_id, node.jm_vuid) + except Exception as e: + logger.error(e) + + +SPEC = RunnerSpec( + name="tasks-runner-migration", + function_names=[JobSchedule.FN_DEV_MIG], + handler=task_runner, + on_finish=resume_jc_compression, + on_cycle=lambda cluster: update_master_tasks(cluster.get_id()), + is_eligible=_is_eligible, + interval=3, +) def main(): - logger.info("Starting Tasks runner...") - while True: - try: - db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue - clusters = db.get_clusters() - if not clusters: - logger.error("No clusters found!") - else: - for cl in clusters: - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - for task in tasks: - if task.function_name == JobSchedule.FN_DEV_MIG and task.status != JobSchedule.STATUS_DONE: - task = db.get_task_by_id(task.uuid) - if task.status in [JobSchedule.STATUS_NEW, JobSchedule.STATUS_SUSPENDED]: - active_task = False - suspended_task= False - for t in db.get_job_tasks(task.cluster_id): - if t.function_name in [JobSchedule.FN_FAILED_DEV_MIG, JobSchedule.FN_DEV_MIG, - JobSchedule.FN_NEW_DEV_MIG] and t.node_id == task.node_id: - if "distr_name" in t.function_params and t.function_params[ - "distr_name"] == task.function_params['distr_name'] and t.canceled is False: - if t.status == JobSchedule.STATUS_RUNNING: - active_task = True - elif t.status == JobSchedule.STATUS_SUSPENDED and t.function_name == JobSchedule.FN_NEW_DEV_MIG: - suspended_task = True - if active_task and suspended_task: - break - if active_task or suspended_task: - logger.info("task found on same node, retry") - continue - elif task.status == JobSchedule.STATUS_RUNNING: - pass - - # Lease gate: skip a task another live runner host owns. - if not tasks_controller.claim_task(task): - logger.info(f"Migration task {task.uuid} owned by another runner host; skipping") - continue - with tasks_controller.task_lease_heartbeat(task): - res = task_runner(task) - update_master_task(task, cl) - if res: - node_task = tasks_controller.get_active_node_tasks(task.cluster_id, task.node_id) - # Release-upgrade guard (remove with the - # jc_compression_upgrade plugin): resumes are - # held until `cluster upgrade-complete`. - if not node_task and jc_compression_upgrade.resume_is_held(cl): - logger.info("JC compression resume held: cluster upgrade in progress") - elif not node_task: - logger.info("no task found on same node, resuming compression") - node = db.get_storage_node_by_id(task.node_id) - for n in db.get_storage_nodes_by_cluster_id(node.cluster_id): - if n.status not in [StorageNode.STATUS_ONLINE, StorageNode.STATUS_SUSPENDED]: - logger.warning("Not all nodes are online, can not resume JC compression") - continue - rpc_client = node.rpc_client(timeout=5, retry=2) - try: - ret, err = rpc_client.jc_suspend_compression(jm_vuid=node.jm_vuid, suspend=False) - if err: - logger.info("Failed to resume JC compression adding task...") - tasks_controller.add_jc_comp_resume_task(task.cluster_id, task.node_id, node.jm_vuid) - except Exception as e: - logger.error(e) - - time.sleep(3) + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_new_dev_migration.py b/simplyblock_core/services/tasks_runner_new_dev_migration.py index a7324c78c6..1a49b9632d 100644 --- a/simplyblock_core/services/tasks_runner_new_dev_migration.py +++ b/simplyblock_core/services/tasks_runner_new_dev_migration.py @@ -1,200 +1,101 @@ # coding=utf-8 import time -from datetime import datetime, timezone from simplyblock_core import db_controller, utils, constants -from simplyblock_core.controllers import tasks_controller -from simplyblock_core.models.cluster import Cluster from simplyblock_core.models.job_schedule import JobSchedule +from simplyblock_core.models.nvme_device import NVMeDevice +from simplyblock_core.models.storage_node import StorageNode +from simplyblock_core.services import migration_task_common as mig +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskAbort, + TaskDefer, + TaskRetry, + serve, +) +logger = utils.get_logger(__name__) +# get DB controller +db = db_controller.DBController() -from simplyblock_core.models.nvme_device import NVMeDevice -from simplyblock_core.models.storage_node import StorageNode + +def _recovery_migration_open(cluster_id): + """Recovery-before-expansion priority: the expansion migration (this task + family, queued when the role rebalance completes) must not run while any + outage-recovery data migration is open — an unexpected node outage during + the expansion queues those, and the required order is: expansion completes + -> outage device migration drains -> expansion migration runs.""" + for task in db.get_job_tasks(cluster_id): + if (task.function_name in (JobSchedule.FN_DEV_MIG, JobSchedule.FN_FAILED_DEV_MIG) + and task.status != JobSchedule.STATUS_DONE + and task.canceled is False): + return task + return None def task_runner(task): - try: - snode = db.get_storage_node_by_id(task.node_id) - except KeyError: - task.status = JobSchedule.STATUS_DONE - task.function_result = f"Node not found: {task.node_id}" - task.write_to_db(db.kv_store) - return True - - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True + snode = mig.require_node(task) if snode.status != StorageNode.STATUS_ONLINE: - task.function_result = "node is not online, retrying" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False - - cluster = db.get_cluster_by_id(task.cluster_id) - if cluster.status not in Cluster.OPERABLE_STATUSES: - task.function_result = "cluster is not active, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False - - # Expansion-first ordering: defer while a cluster expansion is open - # (see tasks_controller.defer_task_for_expansion). - if tasks_controller.defer_task_for_expansion(task): - return False - - # Recovery-before-expansion priority: the expansion migration (this - # task family, queued when the role rebalance completes) must not run - # while any outage-recovery data migration is open — an unexpected - # node outage during the expansion queues those, and the required - # order is: expansion completes -> outage device migration drains -> - # expansion migration runs. Deferral, not failure: no retry consumed. - for t in db.get_job_tasks(task.cluster_id): - if t.function_name in (JobSchedule.FN_DEV_MIG, JobSchedule.FN_FAILED_DEV_MIG) \ - and t.status != JobSchedule.STATUS_DONE and t.canceled is False: - task.function_result = ( - f"deferring: recovery migration {t.uuid} ({t.function_name}) is open") - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False - - if task.status in [JobSchedule.STATUS_NEW, JobSchedule.STATUS_SUSPENDED]: - for node in db.get_storage_nodes_by_cluster_id(task.cluster_id): - if node.is_secondary_node: # pass - continue - - if node.online_since: - try: - diff = datetime.now(timezone.utc) - datetime.fromisoformat(node.online_since) - if diff.total_seconds() < 60: - task.function_result = "node is online < 1 min, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False - except Exception as e: - logger.error(f"Failed to get online since: {e}") - - task.function_result = JobSchedule.STATUS_RUNNING - task.status = JobSchedule.STATUS_RUNNING - task.write_to_db(db.kv_store) + raise TaskRetry("node is not online, retrying") + + mig.require_active_cluster(task) + open_recovery = _recovery_migration_open(task.cluster_id) + if open_recovery is not None: + # Deferral, not failure: no retry consumed. + raise TaskDefer(f"deferring: recovery migration {open_recovery.uuid} " + f"({open_recovery.function_name}) is open") + if not mig.migration_started(task): + if not mig.nodes_settled(task.cluster_id, primaries_only=True): + raise TaskRetry("node is online < 1 min, retrying") rpc_client = snode.rpc_client(timeout=5, retry=2) - all_devs_online_or_failed = True - for node in db.get_storage_nodes_by_cluster_id(task.cluster_id): - for dev in node.nvme_devices: - if dev.status not in [NVMeDevice.STATUS_ONLINE, - NVMeDevice.STATUS_FAILED_AND_MIGRATED, - NVMeDevice.STATUS_CANNOT_ALLOCATE]: - all_devs_online_or_failed = False - break - if "migration" not in task.function_params: - if not all_devs_online_or_failed: - task.function_result = "Some devs are offline, retrying" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False + if not mig.migration_started(task): + if not _all_devices_online_or_written_off(task.cluster_id): + raise TaskRetry("Some devs are offline, retrying") try: - device = db.get_storage_device_by_id(task.device_id) + db.get_storage_device_by_id(task.device_id) except KeyError: - task.status = JobSchedule.STATUS_DONE - task.function_result = "Device not found" - task.write_to_db(db.kv_store) - return True - - distr_name = task.function_params["distr_name"] - - qos_high_priority = False - if db.get_cluster_by_id(snode.cluster_id).is_qos_set(): - qos_high_priority = True - try: - rsp = rpc_client.distr_migration_expansion_start( - distr_name, qos_high_priority, job_size=constants.MIG_JOB_SIZE,jobs=constants.MIG_PARALLEL_JOBS) - except Exception as e: - logger.error(f"Failed to start migration : {e}") - rsp = False - if not rsp: - logger.error(f"Failed to start device migration task, storage_ID: {device.cluster_device_order}") - task.function_result = "Failed to start device migration task" - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False - - task.function_params['migration'] = { - "name": distr_name - } - task.write_to_db(db.kv_store) + raise TaskAbort("Device not found") + + started = mig.start_migration(task, lambda: rpc_client.distr_migration_expansion_start( + task.function_params["distr_name"], mig.qos_high_priority(snode.cluster_id), + job_size=constants.MIG_JOB_SIZE, jobs=constants.MIG_PARALLEL_JOBS)) + if started is None: + raise TaskAbort("canceled while starting migration") + task = started time.sleep(3) - try: - if "migration" in task.function_params: - allow_all_errors = False - for node in db.get_storage_nodes_by_cluster_id(task.cluster_id): - for dev in node.nvme_devices: - if dev.status in [NVMeDevice.STATUS_READONLY, NVMeDevice.STATUS_CANNOT_ALLOCATE]: - allow_all_errors = True - break - - mig_info = task.function_params["migration"] - res = rpc_client.distr_migration_status(**mig_info) - return utils.handle_task_result(task, res, allow_all_errors=allow_all_errors) - except Exception as e: - logger.error("Failed to get migration task status") - logger.exception(e) - task.function_result = "Failed to get migration status" - - task.retry += 1 - task.write_to_db(db.kv_store) - return False + mig.poll_migration(task, rpc_client, allow_all_errors=mig.allow_all_migration_errors( + task.cluster_id, [NVMeDevice.STATUS_READONLY, NVMeDevice.STATUS_CANNOT_ALLOCATE])) -logger = utils.get_logger(__name__) +def _all_devices_online_or_written_off(cluster_id): + for node in db.get_storage_nodes_by_cluster_id(cluster_id): + for dev in node.nvme_devices: + if dev.status not in [NVMeDevice.STATUS_ONLINE, + NVMeDevice.STATUS_FAILED_AND_MIGRATED, + NVMeDevice.STATUS_CANNOT_ALLOCATE]: + return False + return True -# get DB controller -db = db_controller.DBController() + +SPEC = RunnerSpec( + name="tasks-runner-new-dev-migration", + function_names=[JobSchedule.FN_NEW_DEV_MIG], + handler=task_runner, + is_eligible=mig.sibling_eligibility, + interval=3, +) def main(): - logger.info("Starting Tasks runner...") - while True: - try: - db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue - time.sleep(3) - clusters = db.get_clusters() - if not clusters: - logger.error("No clusters found!") - else: - for cl in clusters: - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - for task in tasks: - if task.function_name == JobSchedule.FN_NEW_DEV_MIG: - if task.status in [JobSchedule.STATUS_NEW, JobSchedule.STATUS_SUSPENDED]: - active_task = tasks_controller.get_active_node_mig_task( - task.cluster_id, task.node_id, task.function_params["distr_name"]) - if active_task: - logger.info("task found on same node, retry") - continue - if task.status != JobSchedule.STATUS_DONE: - # get new task object because it could be changed from cancel task - task = db.get_task_by_id(task.uuid) - res = task_runner(task) - if not res: - time.sleep(2) + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_node_add.py b/simplyblock_core/services/tasks_runner_node_add.py index a6836bff53..83210f2e95 100644 --- a/simplyblock_core/services/tasks_runner_node_add.py +++ b/simplyblock_core/services/tasks_runner_node_add.py @@ -1,14 +1,17 @@ # coding=utf-8 import socket -import threading import time -from concurrent.futures import ThreadPoolExecutor from simplyblock_core import db_controller, storage_node_ops, utils, constants -from simplyblock_core.controllers import tasks_controller from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.cluster import Cluster +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskDefer, + TaskRetry, + serve, +) logger = utils.get_logger(__name__) @@ -24,67 +27,6 @@ # cluster-create / expansion fan-out can't exhaust the runner host. MAX_CONCURRENT_NODE_ADDS = constants.NODE_ADD_MAX_PARALLEL -# uuids of node-add tasks a worker on THIS host is currently driving, so the -# dispatch loop never hands the same task to two workers. (Cross-host -# duplicate execution is separately prevented by the per-task lease in -# tasks_controller.claim_task.) -_inflight = set() -_inflight_lock = threading.Lock() - -# target node_addr values currently being driven by a worker, guarded by the -# same lock. The concurrency model above ("different tasks target different -# nodes, no shared state") breaks if TWO task records ever target the SAME -# host — e.g. a caller's retried HTTP request creating a second FN_NODE_ADD -# task before tasks_controller._validate_new_task_node_add existed to block -# it, or any task created before that dedup shipped. Without this, both -# tasks' add_node() calls race the same host's config-slot classify-then- -# create logic milliseconds apart (2026-07-23: two threads, 4ms apart, -# produced 6 node records for a 4-slot host). This is belt-and-suspenders -# for the task-creation-time dedup — it protects against any duplicate task -# that already exists, regardless of how it got created. -_inflight_addrs = set() - - -def process_task(task, cl): - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return False - - if task.retry >= task.max_retry: - task.function_result = "max retry reached" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True - - if db.get_cluster_by_id(cl.get_id()).status == Cluster.STATUS_IN_ACTIVATION: - task.function_result = "Cluster is in_activation, waiting" - task.status = JobSchedule.STATUS_NEW - task.write_to_db(db.kv_store) - return False - - if task.status != JobSchedule.STATUS_RUNNING: - task.status = JobSchedule.STATUS_RUNNING - task.write_to_db(db.kv_store) - - try: - res = storage_node_ops.add_node(**task.function_params) - msg = f"Node add result: {res}" - logger.info(msg) - task.function_result = msg - if res: - task.status = JobSchedule.STATUS_DONE - else: - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return True - except Exception as e: - logger.error(e) - return False - - # Applying the CPU topology during add_node makes the node reboot # (kubeletconfig / MCP update). The in-flight attempt then fails, but the # right reaction is neither a quick blind retry (the node is down for @@ -96,11 +38,23 @@ def process_task(task, cl): NODE_REBOOT_POLL_SEC = 15 +def _node_addr(task): + """The target host of an add. Two task records must never drive the same + one concurrently: the concurrency model above ("different tasks target + different nodes, no shared state") breaks if they do — two add_node() calls + race the same host's config-slot classify-then-create logic milliseconds + apart (2026-07-23: two threads, 4ms apart, produced 6 node records for a + 4-slot host). tasks_controller._validate_new_task_node_add should stop such + a task from being created at all; this is the backstop for any that already + exists, however it got there.""" + return (task.function_params or {}).get("node_addr") + + def _node_api_reachable(task, timeout=5): """TCP-level reachability of the node agent (host:port from the task's node_addr). During add the StorageNode record may not exist yet, so this intentionally checks the address, not the DB object.""" - addr = (task.function_params or {}).get("node_addr", "") + addr = _node_addr(task) or "" if ":" not in addr: return True # can't tell — let the normal retry path decide host, _, port = addr.rpartition(":") @@ -111,7 +65,7 @@ def _node_api_reachable(task, timeout=5): return False -def _wait_node_reachable(task, task_uuid): +def _wait_node_reachable(task): """After a failed attempt, if the node is unreachable (rebooting for the CPU-topology change), wait for it to answer again — up to NODE_REBOOT_WAIT_MAX_SEC — instead of consuming retries against a node @@ -119,133 +73,56 @@ def _wait_node_reachable(task, task_uuid): if _node_api_reachable(task): return False logger.info( - f"Node-add task {task_uuid}: node agent unreachable (rebooting for " + f"Node-add task {task.uuid}: node agent unreachable (rebooting for " f"CPU topology?); waiting up to {NODE_REBOOT_WAIT_MAX_SEC}s for it to return") deadline = time.time() + NODE_REBOOT_WAIT_MAX_SEC while time.time() < deadline: time.sleep(NODE_REBOOT_POLL_SEC) if _node_api_reachable(task): - logger.info(f"Node-add task {task_uuid}: node agent reachable again; retrying add") + logger.info(f"Node-add task {task.uuid}: node agent reachable again; retrying add") return True logger.warning( - f"Node-add task {task_uuid}: node agent still unreachable after " + f"Node-add task {task.uuid}: node agent still unreachable after " f"{NODE_REBOOT_WAIT_MAX_SEC}s; resuming normal retry schedule") return True -def _run_task(task_uuid, cluster_id, node_addr): - """Worker thread: drive one node-add task to completion (or suspension), - then drop it from the in-flight set so a later cycle can retry it. - - Guarded against BaseException — add_node -> write_to_db calls exit(1) on a - DB write failure, which surfaces here as SystemExit; it must be logged and - contained to this worker, never allowed to kill the service loop or leave - the task stuck in the in-flight set. - """ - delay_seconds = constants.TASK_EXEC_INTERVAL_SEC +def process_task(task): try: - while True: - # Re-fetch for fresh FDB state (the task may have been canceled). - task = db.get_task_by_id(task_uuid) - if task is None or task.status == JobSchedule.STATUS_DONE: - break - cl = db.get_cluster_by_id(cluster_id) - # Lease gate: skip a task another live runner host owns. - if not tasks_controller.claim_task(task): - logger.info(f"Node-add task {task_uuid} owned by another runner host; skipping") - break - # add_node blocks for many minutes with no task writes; heartbeat - # the lease so another runner host never sees it stale mid-add. - retry_before = task.retry - with tasks_controller.task_lease_heartbeat(task): - res = process_task(task, cl) - if res: - if task.status == JobSchedule.STATUS_DONE: - break - # Reboot-aware handling: an attempt that failed because the node - # went down (CPU-topology reboot) is expected, not a real failure. - # add_node catches the interrupted spdk_process_start and RETURNS - # False, so process_task already consumed a retry — roll it back so - # the one guaranteed topology reboot per node doesn't eat the - # retry budget. Then wait for the agent to answer and retry - # promptly on a fresh schedule (no blind fast-retry, no runaway - # backoff). The re-run is idempotent: add_node cleans up its own - # stale IN_CREATION record on re-entry. - if _wait_node_reachable(task, task_uuid): - if task.retry > retry_before: - task.retry = retry_before - if task.status != JobSchedule.STATUS_DONE: - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - delay_seconds = constants.TASK_EXEC_INTERVAL_SEC - continue - if not res: - # Cap the exponential backoff so a permanently failing node-add - # can't grow the sleep without bound. - delay_seconds = min( - delay_seconds * 2, - constants.RESTART_TASK_EXEC_INTERVAL_MAX_SEC, - ) - time.sleep(delay_seconds) - except BaseException as e: - logger.error(f"Node-add task {task_uuid} processing crashed: {e}") - logger.exception(e) - finally: - with _inflight_lock: - _inflight.discard(task_uuid) - if node_addr: - _inflight_addrs.discard(node_addr) + res = storage_node_ops.add_node(**task.function_params) + msg = f"Node add result: {res}" + logger.info(msg) + except Exception as e: + logger.error(e) + res, msg = False, f"Node add raised: {e}" + + if res: + task.function_result = msg + return + + # The one guaranteed topology reboot per node must not eat the retry + # budget: add_node catches the interrupted spdk_process_start and reports + # failure, so wait for the agent to answer and re-attempt on a fresh + # schedule instead. The re-run is idempotent — add_node cleans up its own + # stale IN_CREATION record on re-entry. + if _wait_node_reachable(task): + raise TaskDefer(f"{msg} (node agent was rebooting)") + + raise TaskRetry(msg) + + +SPEC = RunnerSpec( + name="tasks-runner-node-add", + function_names=[JobSchedule.FN_NODE_ADD], + handler=process_task, + is_eligible=lambda task, cluster: cluster.status != Cluster.STATUS_IN_ACTIVATION, + concurrency=MAX_CONCURRENT_NODE_ADDS, + exclusion_key=_node_addr, +) def main(): - logger.info("Starting Tasks runner node add...") - - executor = ThreadPoolExecutor(max_workers=MAX_CONCURRENT_NODE_ADDS) - - while True: - try: - clusters = db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue - if not clusters: - logger.error("No clusters found!") - else: - for cl in clusters: - # An unhandled FDBError here (1031 transaction timeout) killed - # this runner at cluster start on 2026-07-16 — no auto-restart - # ran for the rest of the run. Log and retry next tick instead. - try: - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - except Exception as e: - logger.error(f"Failed to read tasks for cluster {cl.get_id()}: {e}") - continue - for task in tasks: - if task.function_name != JobSchedule.FN_NODE_ADD: - continue - if task.status == JobSchedule.STATUS_DONE: - continue - node_addr = (task.function_params or {}).get("node_addr") - # Dispatch to a worker once; skip if a worker on this host is - # already driving it. Excess tasks queue in the executor and - # run as workers free up. Also skip if a DIFFERENT task - # already targets the SAME node_addr: tasks_controller's - # creation-time dedup should prevent that task from ever - # existing, but this is the backstop — two tasks racing the - # same host's config-slot classify-then-create logic - # produced duplicate storage-node records (2026-07-23). - with _inflight_lock: - if task.uuid in _inflight: - continue - if node_addr and node_addr in _inflight_addrs: - continue - _inflight.add(task.uuid) - if node_addr: - _inflight_addrs.add(node_addr) - executor.submit(_run_task, task.uuid, cl.get_id(), node_addr) - - time.sleep(constants.TASK_EXEC_INTERVAL_SEC) + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_node_removal.py b/simplyblock_core/services/tasks_runner_node_removal.py index 460353e272..08a539f998 100644 --- a/simplyblock_core/services/tasks_runner_node_removal.py +++ b/simplyblock_core/services/tasks_runner_node_removal.py @@ -1,12 +1,15 @@ # coding=utf-8 -import time - - from simplyblock_core import db_controller, storage_node_ops, utils, constants -from simplyblock_core.controllers import tasks_controller from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.cluster import Cluster from simplyblock_core.models.storage_node import StorageNode +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskDefer, + TaskProgress, + serve, +) + logger = utils.get_logger(__name__) @@ -19,85 +22,35 @@ def process_task(task): node_removal_orchestrate is idempotent and resumable: it returns True only when the node is fully REMOVED, and False to mean "incomplete, retry later" - (most commonly: device failure-migration still in progress). On False we - suspend the task so the outer loop revisits it on the next tick instead of - busy-spinning here for what can be hours. + (most commonly: device failure-migration still in progress, which can take + hours). Incomplete is progress, not failure — it consumes no retry, and the + task stays RUNNING so the next tick picks it straight back up. """ - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True - cluster = db.get_cluster_by_id(task.cluster_id) if cluster.status == Cluster.STATUS_IN_ACTIVATION: - task.function_result = "cluster is in_activation, waiting" - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - storage_node_ops.set_node_status(task.node_id, StorageNode.STATUS_PENDING_REMOVAL, caused_by="remove") - return False - - if task.status != JobSchedule.STATUS_RUNNING: - task.status = JobSchedule.STATUS_RUNNING - task.write_to_db(db.kv_store) + # Checked here rather than through is_eligible: the wait also has to + # park the node in PENDING_REMOVAL, and eligibility must stay pure. + storage_node_ops.set_node_status( + task.node_id, StorageNode.STATUS_PENDING_REMOVAL, caused_by="remove") + raise TaskDefer("cluster is in_activation, waiting") force_remove = bool(task.function_params.get("force_remove", False)) - try: - done = storage_node_ops.node_removal_orchestrate(task.node_id, force_remove=force_remove) - except Exception as e: - logger.error(f"Node-removal task {task.uuid} raised: {e}") - logger.exception(e) - task.function_result = f"error: {e}" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False + if not storage_node_ops.node_removal_orchestrate(task.node_id, force_remove=force_remove): + raise TaskProgress("removal in progress, retrying") - if done: - task.function_result = "Node removed" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True - else: - # Incomplete: a phase asked us to retry (typically waiting on migration). - task.function_result = "removal in progress, retrying" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False + task.function_result = "Node removed" -def main(): - logger.info("Starting Tasks runner node removal...") +SPEC = RunnerSpec( + name="tasks-runner-node-removal", + function_names=[JobSchedule.FN_NODE_REMOVAL], + handler=process_task, + interval=constants.TASK_EXEC_INTERVAL_SEC, +) - while True: - time.sleep(constants.TASK_EXEC_INTERVAL_SEC) - try: - clusters = db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - continue - if not clusters: - logger.error("No clusters found!") - continue - for cl in clusters: - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - for task in tasks: - if task.function_name != JobSchedule.FN_NODE_REMOVAL: - continue - if task.status == JobSchedule.STATUS_DONE: - continue - # get a fresh object: cancel/other writers may have changed it - task = db.get_task_by_id(task.uuid) - # Lease gate: skip a task another live runner host owns. - if not tasks_controller.claim_task(task): - logger.info(f"Node-removal task {task.uuid} owned by another runner host; skipping") - continue - try: - process_task(task) - except Exception as e: - logger.error(f"Node-removal task {task.uuid} processing crashed: {e}") - logger.exception(e) + +def main(): + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_port_allow.py b/simplyblock_core/services/tasks_runner_port_allow.py index d2ff35ccd0..16bf4abaa6 100644 --- a/simplyblock_core/services/tasks_runner_port_allow.py +++ b/simplyblock_core/services/tasks_runner_port_allow.py @@ -12,6 +12,12 @@ from simplyblock_core.models.storage_node import StorageNode from simplyblock_core.models.nvme_device import NVMeDevice from simplyblock_core.models.lvol_model import LVol +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskAbort, + TaskDefer, + serve, +) logger = utils.get_logger(__name__) @@ -566,30 +572,13 @@ def _abort_recovering_node(node, reason): def exec_port_allow_task(task): - # get new task object because it could be changed from cancel task - task = db.get_task_by_id(task.uuid) - - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return - try: node = db.get_storage_node_by_id(task.node_id) except KeyError: - task.function_result = "node not found" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return + raise TaskAbort("node not found") if node.status not in [StorageNode.STATUS_DOWN, StorageNode.STATUS_ONLINE]: - msg = f"Node is {node.status}, retry task" - logger.info(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer(f"Node is {node.status}, retry task") # check node ping ping_check = health_controller._check_node_ping(node.mgmt_ip) @@ -600,12 +589,7 @@ def exec_port_allow_task(task): logger.info(f"Check 2: ping mgmt ip {node.mgmt_ip} ... {ping_check}") if not ping_check: - msg = "Node ping is false, retry task" - logger.info(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer("Node ping is false, retry task") # Data-NIC gate: mgmt reachability alone is not recovery — after a # partial partition the mgmt plane often returns first while the @@ -629,24 +613,14 @@ def exec_port_allow_task(task): logger.info(f"Check: ping data nic {data_nic.ip4_address} ... {data_ping}") data_results.append(data_ping) if data_results and not any(r is True for r in data_results): - msg = "Node data NIC not confirmed reachable, retry task" - logger.info(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer("Node data NIC not confirmed reachable, retry task") logger.info("connect to remote devices") # connect to remote devs try: remote_devices = storage_node_ops._connect_to_remote_devs(node, reattach=False) if not remote_devices: - msg = "Node unable to connect to remote devs, retry task" - logger.info(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer("Node unable to connect to remote devs, retry task") else: # Re-read fresh before writing to avoid overwriting concurrent changes node = db.get_storage_node_by_id(task.node_id) @@ -656,12 +630,7 @@ def exec_port_allow_task(task): logger.info("connect to remote JM devices") remote_jm_devices = storage_node_ops._connect_to_remote_jm_devs(node) if not remote_jm_devices or len(remote_jm_devices) < 2: - msg = "Node unable to connect to remote JMs, retry task" - logger.info(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer("Node unable to connect to remote JMs, retry task") else: # Re-read fresh before writing to avoid overwriting concurrent changes node = db.get_storage_node_by_id(task.node_id) @@ -669,14 +638,11 @@ def exec_port_allow_task(task): node.write_to_db() + except TaskDefer: + raise except Exception as e: logger.error(e) - msg = "Error when connect to remote devs, retry task" - logger.info(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer("Error when connect to remote devs, retry task") # After a network outage every distrib in the cluster is working from a # stale device view: the recovering node's distribs have stale REMOTE @@ -707,12 +673,7 @@ def exec_port_allow_task(task): f"port allow on {node.get_id()}") if not device_controller.device_set_online( dev.get_id(), cause=device_controller.CAUSE_NODE_RECOVERY): - msg = f"Device {dev.get_id()} re-admit refused, retry task" - logger.warning(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer(f"Device {dev.get_id()} re-admit refused, retry task") # Positive confirmation gate: every device-status event must be applied by # ALL of its target distribs before we touch hublvols / leadership / the @@ -722,12 +683,6 @@ def exec_port_allow_task(task): # held a stale device view, which is the placement/read-failure class this # whole sequence exists to prevent. On any device whose event is not # confirmed, suspend and retry — do NOT proceed. - def _fail(msg): - logger.warning(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - logger.info("Broadcasting local device status of recovering node to all nodes") node = db.get_storage_node_by_id(task.node_id) for dev in node.nvme_devices: @@ -736,12 +691,11 @@ def _fail(msg): try: ok = distr_controller.send_dev_status_event(dev, dev.status) except Exception as e: - _fail(f"Local device status broadcast for {dev.get_id()} failed: {e}, retry task") - return + raise TaskDefer(f"Local device status broadcast for {dev.get_id()} " + f"failed: {e}, retry task") if not ok: - _fail(f"Local device status for {dev.get_id()} not applied by all " - f"distribs, retry task") - return + raise TaskDefer(f"Local device status for {dev.get_id()} " + f"not applied by all distribs, retry task") logger.info("Sending other nodes' device status events to recovering node") for cluster_node in db.get_storage_nodes_by_cluster_id(task.cluster_id): @@ -753,14 +707,14 @@ def _fail(msg): try: ok = distr_controller.send_dev_status_event( dev, dev.status, target_node=node) + except TaskDefer: + raise except Exception as e: - _fail(f"Device status event for {dev.get_id()} to recovering node " - f"failed: {e}, retry task") - return + raise TaskDefer(f"Device status event for {dev.get_id()} to " + f"recovering node failed: {e}, retry task") if not ok: - _fail(f"Device status for {dev.get_id()} not applied by recovering " - f"node's distribs, retry task") - return + raise TaskDefer(f"Device status for {dev.get_id()} " + f"not applied by recovering node's distribs, retry task") logger.info("All device-status events confirmed applied by distribs") @@ -774,24 +728,14 @@ def _fail(msg): node = db.get_storage_node_by_id(task.node_id) own_hublvols_ok, own_msg = _reconnect_own_sec_tert_hublvols(node) if not own_hublvols_ok: - msg = f"Own (outbound) hublvol reconnect failed: {own_msg}, retry task" - logger.warning(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer(f"Own (outbound) hublvol reconnect failed: {own_msg}, retry task") # INBOUND: for lvstores where this node is the secondary, re-expose its # hub and ensure the tertiary's redirect path to it is connected (no # leadership action here — that stays in the failback step below). inbound_ok, inbound_msg = _reconnect_inbound_hublvols(node) if not inbound_ok: - msg = f"Inbound hublvol reconnect failed: {inbound_msg}, retry task" - logger.warning(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer(f"Inbound hublvol reconnect failed: {inbound_msg}, retry task") snode = db.get_storage_node_by_id(node.get_id()) sec_ids = [] @@ -821,17 +765,12 @@ def _fail(msg): snode.jm_vuid) except Exception as e: logger.error(e) - return + raise TaskDefer(f"JC compression check on peer failed: {e}, retry task") if node.lvstore_status == "ready": lvstore_check = health_controller._check_node_lvstore(node.lvstore_stack, node, auto_fix=True) if not lvstore_check: - msg = "Node LVolStore check fail, retry later" - logger.warning(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer("Node LVolStore check fail, retry later") sec_ids = [] if node.secondary_node_id: @@ -846,12 +785,7 @@ def _fail(msg): # primary-local recovery step, not a peer reconnect. primary_hublvol_check = health_controller._check_node_hublvol(node) if not primary_hublvol_check: - msg = "Node hublvol check fail, retry later" - logger.warning(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer("Node hublvol check fail, retry later") # Peer hublvol gate: for each ONLINE secondary/tertiary, drive # ``connect_to_hublvol`` (which re-attaches the bdev_nvme @@ -883,15 +817,8 @@ def _fail(msg): f"after {_HUBLVOL_MAX_ATTEMPTS} attempts: " + ", ".join(p[:8] for p in failing_peers)) _abort_recovering_node(node, reason) - task.function_result = ( + raise TaskAbort( f"Aborted recovering node {node.get_id()[:8]}: {reason}") - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return - - if task.status != JobSchedule.STATUS_RUNNING: - task.status = JobSchedule.STATUS_RUNNING - task.write_to_db(db.kv_store) try: # wait for lvol sync delete @@ -903,9 +830,11 @@ def _fail(msg): port_number = task.function_params["port_number"] + except TaskDefer: + raise except Exception as e: logger.error(e) - return + raise TaskDefer(f"Waiting for lvol sync delete failed: {e}, retry task") # --- Leadership failback, BEFORE the port is unblocked -------------- # @@ -954,12 +883,8 @@ def _fail(msg): leadership_read_failed = f"{peer.get_id()[:8]}: {e}" if current_leader is None and leadership_read_failed: - msg = f"Leadership read failed on peer {leadership_read_failed}, retry task" - logger.warning(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer( + f"Leadership read failed on peer {leadership_read_failed}, retry task") if current_leader is not None: failback_ok, failback_msg = _failback_leadership_to_primary( @@ -974,12 +899,7 @@ def _fail(msg): failback_ok, failback_msg = True, "" if not failback_ok: - msg = f"Leadership failback incomplete: {failback_msg}, retry task" - logger.warning(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return + raise TaskDefer(f"Leadership failback incomplete: {failback_msg}, retry task") # Unblock ALL of the node's LVS subsystem ports (own primary + every # follower LVS), not just the single port_number the task was created @@ -1061,46 +981,26 @@ def _fail(msg): logger.error(f"Device re-admit after port allow failed: {e}") task.function_result = f"Port {port_number} allowed on node" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) + + +SPEC = RunnerSpec( + name="tasks-runner-port-allow", + function_names=[JobSchedule.FN_PORT_ALLOW], + handler=exec_port_allow_task, + # No IN_ACTIVATION gate, deliberately, unlike every other runner: + # port_allow is the final step of a node's recovery, and activation NEEDS + # those ports open (2026-07-16 full-fleet reboot: a task suspended seconds + # before activation started froze for the entire 30-minute activation, + # while the activation's hublvol attaches to that node failed against its + # still-blocked port). The task's own gates — node status, mgmt/data-NIC + # pings, verified-open hublvols — decide whether a retry can proceed; one + # that still can't just defers again. + interval=5, +) def main(): - logger.info("Starting Tasks runner...") - while True: - try: - db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue - clusters = db.get_clusters() - if not clusters: - logger.error("No clusters found!") - else: - for cl in clusters: - # Deliberately NOT skipped while the cluster is IN_ACTIVATION: - # port_allow is the final step of a node's recovery, and - # activation NEEDS those ports open (2026-07-16 full-fleet - # reboot: a task suspended seconds before activation started - # froze at 0/8 retries for the entire 30-minute activation, - # while the activation's hublvol attaches to that node failed - # against its still-blocked port). The task's own gates (node - # status, mgmt/data-NIC pings, verified-open hublvols) decide - # whether a retry can proceed; a retry that still can't just - # re-suspends. - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - for task in tasks: - if task.function_name == JobSchedule.FN_PORT_ALLOW: - if task.status != JobSchedule.STATUS_DONE: - # Lease gate: skip a task another live runner host owns. - if not tasks_controller.claim_task(task): - logger.info(f"Port-allow task {task.uuid} owned by another runner host; skipping") - continue - with tasks_controller.task_lease_heartbeat(task): - exec_port_allow_task(task) - - time.sleep(5) + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_replication_final.py b/simplyblock_core/services/tasks_runner_replication_final.py index 6e21839450..fee5725907 100644 --- a/simplyblock_core/services/tasks_runner_replication_final.py +++ b/simplyblock_core/services/tasks_runner_replication_final.py @@ -24,12 +24,13 @@ import uuid as uuid_lib from datetime import datetime -from simplyblock_core import constants, db_controller, utils +from simplyblock_core import db_controller, utils from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.lvol_model import LVol, LVolReplication from simplyblock_core.models.snapshot import SnapShot from simplyblock_core.models.storage_node import StorageNode from simplyblock_core.services import replication_final_step +from simplyblock_core.services.task_runner_base import RunnerSpec, TaskDefer, TaskRetry, serve logger = utils.get_logger(__name__) utils.init_sentry_sdk(__name__) @@ -37,145 +38,121 @@ db = db_controller.DBController() -def _finalize(task, ok, err): - if ok: - replication_id = task.function_params.get("replication_id") - final_state = task.function_params.get("final_state", LVolReplication.STATE_CUTOVER_DONE) - if replication_id: - try: - rep = db.get_lvol_replication_by_id(replication_id) - rep.state = final_state - rep.write_to_db(db.kv_store) - except Exception as e: - logger.error(f"Failed to update replication state: {e}") - task.function_result = "cutover done" - task.status = JobSchedule.STATUS_DONE - task.function_params["end_time"] = int(time.time()) - task.write_to_db(db.kv_store) - - # The hand-off is complete: the target serves the data from here on, - # so the SOURCE must stop replicating. Nothing else clears its cadence - # config, and a retired source otherwise keeps taking internal - # snapshots and shipping them to the very target it handed off to - # (observed 2026-08-21: replication_final done "cutover done" while - # the source volumes kept replicating). In-flight transfers drain - # naturally; this stops NEW cadence snapshots at the gate the monitor - # reads (do_replicate / replication_interval_min). - try: - src_lvol = db.get_lvol_by_id(task.function_params.get("lvol_id")) - if src_lvol.do_replicate: - src_lvol.do_replicate = False - src_lvol.replication_interval_min = 0 - src_lvol.replication_policy_id = "" - src_lvol.write_to_db() - logger.info(f"Cutover done: stopped replication on source " - f"volume {src_lvol.get_id()}") - except KeyError: - pass # source already gone (e.g. deleted out of band) - except Exception as e: - logger.error(f"Could not stop replication on the source after " - f"cutover: {e}") - - # Optional migration semantics: the source volume has served its - # purpose once the client runs on the target, so `replication-commit - # --delete-source` retires it here — strictly AFTER the cutover state - # is durable, so a crash in between leaves a completed cutover with - # the source still present (retryable by hand), never a deleted - # source with an uncommitted cutover. The relationship record is what - # later look-ups (target-by-source, active side) resolve through, and - # it survives the volume's deletion. - if task.function_params.get("delete_source"): - src_lvol_id = task.function_params.get("lvol_id") - try: - from simplyblock_core.controllers import lvol_controller - src_lvol = db.get_lvol_by_id(src_lvol_id) - logger.info(f"Cutover committed with --delete-source: deleting " - f"source volume {src_lvol_id}") - lvol_controller.delete_lvol(src_lvol) - except Exception as e: - # The cutover itself succeeded; a failed source delete is - # reported loudly but does not un-succeed the task. - logger.error(f"Source volume {src_lvol_id} could not be " - f"deleted after the cutover: {e}") - return True - - task.function_result = err or "cutover failed, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False +def _record_cutover_done(task): + replication_id = task.function_params.get("replication_id") + if not replication_id: + return + + final_state = task.function_params.get("final_state", LVolReplication.STATE_CUTOVER_DONE) + try: + rep = db.get_lvol_replication_by_id(replication_id) + rep.state = final_state + rep.write_to_db(db.kv_store) + except Exception as e: + logger.error(f"Failed to update replication state: {e}") + + +def _stop_source_replication(task): + """The hand-off is complete: the target serves the data from here on, so the + SOURCE must stop replicating. Nothing else clears its cadence config, and a + retired source otherwise keeps taking internal snapshots and shipping them + to the very target it handed off to (observed 2026-08-21: replication_final + done "cutover done" while the source volumes kept replicating). In-flight + transfers drain naturally; this stops NEW cadence snapshots at the gate the + monitor reads (do_replicate / replication_interval_min). + """ + try: + src_lvol = db.get_lvol_by_id(task.function_params.get("lvol_id")) + if src_lvol.do_replicate: + src_lvol.do_replicate = False + src_lvol.replication_interval_min = 0 + src_lvol.replication_policy_id = "" + src_lvol.write_to_db() + logger.info(f"Cutover done: stopped replication on source " + f"volume {src_lvol.get_id()}") + except KeyError: + pass # source already gone (e.g. deleted out of band) + except Exception as e: + logger.error(f"Could not stop replication on the source after " + f"cutover: {e}") + + +def _delete_source_if_requested(task): + """Optional migration semantics: the source volume has served its purpose + once the client runs on the target, so `replication-commit --delete-source` + retires it here — strictly AFTER the cutover state is durable, so a crash in + between leaves a completed cutover with the source still present (retryable + by hand), never a deleted source with an uncommitted cutover. The + relationship record is what later look-ups (target-by-source, active side) + resolve through, and it survives the volume's deletion. + """ + if not task.function_params.get("delete_source"): + return + + src_lvol_id = task.function_params.get("lvol_id") + try: + from simplyblock_core.controllers import lvol_controller + src_lvol = db.get_lvol_by_id(src_lvol_id) + logger.info(f"Cutover committed with --delete-source: deleting " + f"source volume {src_lvol_id}") + lvol_controller.delete_lvol(src_lvol) + except Exception as e: + # The cutover itself succeeded; a failed source delete is reported + # loudly but does not un-succeed the task. + logger.error(f"Source volume {src_lvol_id} could not be " + f"deleted after the cutover: {e}") def task_runner(task: JobSchedule): params = task.function_params lvol_id = params.get("lvol_id") if not lvol_id: - return _finalize(task, False, "missing lvol_id in task params") - - if task.retry >= task.max_retry or task.canceled is True: - task.function_result = "task cancelled" if task.canceled else "max retry reached" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True + raise TaskRetry("missing lvol_id in task params") try: lvol = db.get_lvol_by_id(lvol_id) tgt_node = db.get_storage_node_by_id(params["tgt_node_id"]) except KeyError as e: - return _finalize(task, False, f"object not found: {e}") + raise TaskRetry(f"object not found: {e}") + + if tgt_node.status != StorageNode.STATUS_ONLINE: + raise TaskRetry("target node not online, retrying") - # The source may be gone entirely (fail-over after cluster loss); cutover - # proceeds with a best-effort ANA flip in that case. - src_node = None try: src_node = db.get_storage_node_by_id(params["src_node_id"]) except KeyError: - pass + raise TaskRetry("source node not found for cutover") - if tgt_node.status != StorageNode.STATUS_ONLINE: - task.function_result = "target node not online, retrying" - task.status = JobSchedule.STATUS_SUSPENDED - task.retry += 1 - task.write_to_db(db.kv_store) - return False - - if src_node is None: - return _finalize(task, False, "source node not found for cutover") - - if task.status in [JobSchedule.STATUS_NEW, JobSchedule.STATUS_SUSPENDED, JobSchedule.STATUS_RUNNING]: - task.status = JobSchedule.STATUS_RUNNING - task.function_params.setdefault("start_time", int(time.time())) - task.write_to_db(db.kv_store) - - # ---- SHRINK PHASE ----------------------------------------------- # - if "shrink_snap_id" in params and params.get("shrink_round", 0) > 0: - done, err = _shrink_step(task, lvol) - if err: - return _finalize(task, False, err) - if not done: - # waiting on replication of the current shrink snapshot; come - # back next pass WITHOUT burning a retry (bounded by deadline) - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - return False - - # ---- CUTOVER PHASE (immediately after the last shrink round) ---- # - if "tgt_lvol_composite" not in params: - err = _prepare_cutover(task, lvol, src_node, tgt_node) - if err: - return _finalize(task, False, err) - params = task.function_params - - try: - ok, err = replication_final_step.run_cutover( - src_node, tgt_node, lvol, - params["tgt_lvol_composite"], params["tgt_map_id"], - params["tgt_snap_composite"], operation=params.get("operation", "replicate")) - except Exception as e: - logger.error(f"Cutover raised: {e}", exc_info=True) - return _finalize(task, False, str(e)) - return _finalize(task, ok, err) - return True + params.setdefault("start_time", int(time.time())) + + # ---- SHRINK PHASE --------------------------------------------------- # + if "shrink_snap_id" in params and params.get("shrink_round", 0) > 0: + _shrink_step(task, lvol) + + # ---- CUTOVER PHASE (immediately after the last shrink round) -------- # + if "tgt_lvol_composite" not in params: + _prepare_cutover(task, lvol, src_node, tgt_node) + + ok, err = replication_final_step.run_cutover( + src_node, tgt_node, lvol, + params["tgt_lvol_composite"], params["tgt_map_id"], + params["tgt_snap_composite"], operation=params.get("operation", "replicate")) + if not ok: + raise TaskRetry(err or "cutover failed, retrying") + + _record_cutover_done(task) + params["end_time"] = int(time.time()) + task.function_result = "cutover done" + + _stop_source_replication(task) + _delete_source_if_requested(task) + + +SPEC = RunnerSpec( + name="tasks-runner-replication-final", + function_names=[JobSchedule.FN_REPLICATION_FINAL], + handler=task_runner, +) SHRINK_ROUNDS = 2 @@ -184,28 +161,29 @@ def task_runner(task: JobSchedule): def _shrink_step(task, lvol): """Advance the delta-shrink state machine one step. - Returns (done, error): done=True when all rounds are replicated and the - cutover may start IMMEDIATELY; error aborts the task. + Returns once all rounds are replicated and the cutover may start + IMMEDIATELY. Raises :class:`TaskDefer` while a round is still replicating — + waiting for the target is not a failure, so it must not consume a retry; the + wait is bounded by ``shrink_deadline`` instead. """ params = task.function_params if int(time.time()) > params.get("shrink_deadline", 0): - return False, "shrink phase timed out waiting for replication" + raise TaskRetry("shrink phase timed out waiting for replication") snap_id = params["shrink_snap_id"] try: snap = db.get_snapshot_by_id(snap_id) except KeyError: - return False, f"shrink snapshot {snap_id} disappeared" + raise TaskRetry(f"shrink snapshot {snap_id} disappeared") # target_replicated_snap_uuid is set at replicate-finish AFTER the target # copy is chained and converted — replicated AND converted in one signal. if not snap.target_replicated_snap_uuid: - task.function_result = (f"shrink round {params['shrink_round']}: waiting " - f"for {snap_id[:8]} to replicate") - return False, None + raise TaskDefer(f"shrink round {params['shrink_round']}: waiting " + f"for {snap_id[:8]} to replicate") if params["shrink_round"] >= SHRINK_ROUNDS: - return True, None + return # Round replicated — IMMEDIATELY take the next snapshot: its delta covers # only the wait window of the previous round. @@ -214,12 +192,10 @@ def _shrink_step(task, lvol): lvol.get_id(), f"repl_commit_{uuid_lib.uuid4()}", snap_type=SnapShot.TYPE_INTERNAL) if err: - return False, f"shrink round {params['shrink_round'] + 1} snapshot failed: {err}" + raise TaskRetry(f"shrink round {params['shrink_round'] + 1} snapshot failed: {err}") params["shrink_round"] += 1 params["shrink_snap_id"] = new_snap - task.function_result = f"shrink round {params['shrink_round']}: snapshot taken" - task.write_to_db(db.kv_store) - return False, None + raise TaskDefer(f"shrink round {params['shrink_round']}: snapshot taken") def _prepare_cutover(task, lvol, src_node, tgt_node): @@ -239,7 +215,7 @@ def _prepare_cutover(task, lvol, src_node, tgt_node): new_lvol, snapshot, error = lvol_controller._clone_from_last_replicated( db, lvol.get_id(), lvol, tgt_node, target_pool_uuid, src_node.cluster_id) if error: - return f"cutover clone failed: {error}" + raise TaskRetry(f"cutover clone failed: {error}") new_lvol.status = LVol.STATUS_ONLINE new_lvol.write_to_db(db.kv_store) @@ -250,7 +226,7 @@ def _prepare_cutover(task, lvol, src_node, tgt_node): if tgt_map_id is None: lvol_controller.delete_lvol_from_node(new_lvol, tgt_node) db.release_lvol_ns_slot(new_lvol) - return "could not resolve target map_id" + raise TaskRetry("could not resolve target map_id") rep = LVolReplication() rep.uuid = str(uuid_lib.uuid4()) @@ -272,34 +248,15 @@ def _prepare_cutover(task, lvol, src_node, tgt_node): "tgt_snap_composite": snapshot.snap_bdev, "replication_id": rep.get_id(), }) + # Checkpoint the task itself, against the driver's usual ownership of task + # writes: the clone and the LVolReplication record above already exist, so a + # crash before the cutover completes must not let the next attempt build a + # second clone. task.write_to_db(db.kv_store) - return None def main(): - logger.info("Starting replication-final tasks runner...") - while True: - try: - clusters = db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue - for cl in clusters: - for task in db.get_job_tasks(cl.get_id(), reverse=False): - if task.function_name != JobSchedule.FN_REPLICATION_FINAL: - continue - if task.status == JobSchedule.STATUS_DONE: - continue - task = db.get_task_by_id(task.uuid) - try: - res = task_runner(task) - except Exception as e: - logger.error(f"replication-final task {task.uuid} failed: {e}", exc_info=True) - res = False - if not res: - time.sleep(3) - time.sleep(constants.TASK_EXEC_INTERVAL_SEC) + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_restart.py b/simplyblock_core/services/tasks_runner_restart.py index 82d3a75497..d266f696b2 100644 --- a/simplyblock_core/services/tasks_runner_restart.py +++ b/simplyblock_core/services/tasks_runner_restart.py @@ -1,7 +1,5 @@ # coding=utf-8 -import datetime import time -from concurrent.futures import ThreadPoolExecutor from simplyblock_core import constants, db_controller, storage_node_ops, utils from simplyblock_core.controllers import device_controller, health_controller, tasks_controller @@ -9,6 +7,14 @@ from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.nvme_device import NVMeDevice from simplyblock_core.models.storage_node import StorageNode +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskAbort, + TaskDefer, + TaskRetry, + checkpoint, + serve, +) from simplyblock_core.snode_client import SNodeClientException @@ -20,6 +26,43 @@ utils.init_sentry_sdk() +def _parallel_restart_allowed(node): + """The two sanctioned cases for restarting nodes in parallel: a drained + suspension (full-cluster recovery — every node offline, no client IO), and + a fully-dead failure domain (no domain member ONLINE, so parallel recovery + cannot touch served IO). + + Decides both the dispatch mode and whether the peer-exclusion pre-check in + task_runner_node applies. The two must agree: a task fanned out in parallel + would otherwise immediately defer on the very peers it was dispatched + alongside. + """ + cluster = db.get_cluster_by_id(node.cluster_id) + if cluster.status == Cluster.STATUS_SUSPENDED and cluster.suspend_drain_complete: + return True + return storage_node_ops.fd_dead_recovery_allowed(db, node) + + +def _is_eligible(task, cluster): + """Suspend recovery: while a SUSPENDED cluster is still being drained to + all-offline, pause node restarts. Executing one now would fight the + auto-shutdown and re-create the wedged half-restarted state we are fixing. + The task re-polls without consuming a retry and runs once the drain + completes.""" + if task.function_name != JobSchedule.FN_NODE_RESTART: + return True + return not tasks_controller.is_auto_restart_paused(db.get_cluster_by_id(cluster.get_id())) + + +def _serialize(task, cluster): + if task.function_name != JobSchedule.FN_NODE_RESTART or not task.node_id: + return True + try: + return not _parallel_restart_allowed(db.get_storage_node_by_id(task.node_id)) + except KeyError: + return True + + def _get_node_unavailable_devices_count(node_id): node = db.get_storage_node_by_id(node_id) devices = [] @@ -46,62 +89,6 @@ def _validate_no_task_node_restart(cluster_id, node_id): return True -def _task_finish(task, result): - """Terminal task write via atomic CAS: set DONE + result on the FRESH row. - - A plain ``task.write_to_db()`` of the runner's in-memory copy writes the - whole stale object: it erased a concurrent cancellation and wiped the - owner lease (2026-07-29 double restart). This never overwrites an - already-DONE task (a concurrent cancellation's result wins) and preserves - ``owner`` / ``canceled`` as found. - - Returns True if this call performed the transition, False if the task was - already finished (or gone) — callers gating side effects (give-up - OFFLINE flip, re-queue) on having won the transition must check it. - """ - now = str(datetime.datetime.now(datetime.timezone.utc)) - wrote = {"done": False} - - def _mutate(t): - if t.status == JobSchedule.STATUS_DONE: - return False - t.function_result = result - t.status = JobSchedule.STATUS_DONE - t.updated_at = now - wrote["done"] = True - return True - - if db.atomic_update(task, _mutate) is None: - return False - return wrote["done"] - - -def _task_update(task, mutate): - """Non-terminal task write via atomic CAS. Applies ``mutate(t)`` to the - FRESH row, so it can neither resurrect a concurrently canceled/finished - task nor clobber concurrently-written fields (owner lease, cancellation) - the way a full-object ``write_to_db()`` of a stale copy does. - - Returns the fresh post-write task object, or None when the task is - done/canceled/gone — the caller must stop driving the task in that case. - ``mutate`` may be replayed on transaction conflict; it must only mutate - the object passed to it (no I/O, no other writes). - """ - now = str(datetime.datetime.now(datetime.timezone.utc)) - - def _mutate(t): - if t.status == JobSchedule.STATUS_DONE or t.canceled: - return False - mutate(t) - t.updated_at = now - return True - - fresh = db.atomic_update(task, _mutate) - if fresh is None or fresh.status == JobSchedule.STATUS_DONE or fresh.canceled: - return None - return fresh - - def _ensure_spdk_killed(node): """Best-effort kill of the SPDK process on the node before we mark it OFFLINE. Without this, flipping the status to OFFLINE while SPDK is still @@ -241,53 +228,24 @@ def task_runner(task): def task_runner_device(task): device = _get_device(task) - if task.retry >= constants.TASK_EXEC_RETRY_COUNT: - task.function_result = "max retry reached" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - device_controller.device_set_unavailable(device.get_id()) - device_controller.device_set_retries_exhausted(device.get_id(), True) - return True - if not _validate_no_task_node_restart(task.cluster_id, task.node_id): - task.function_result = "canceled: node restart found" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) + # The node-level restart supersedes this device-level one. device_controller.device_set_unavailable(device.get_id()) - return True - - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - device_controller.device_set_retries_exhausted(device.get_id(), True) - return True + raise TaskAbort("canceled: node restart found") node = db.get_storage_node_by_id(task.node_id) if node.status != StorageNode.STATUS_ONLINE: logger.error(f"Node is not online: {node.get_id()}, retry") - task.function_result = "Node is offline" - task.retry += 1 - task.write_to_db(db.kv_store) - return False + raise TaskRetry("Node is offline") if device.status == NVMeDevice.STATUS_ONLINE and device.io_error is False: logger.info(f"Device is online: {device.get_id()}") task.function_result = "Device is online" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True + return if device.status in [NVMeDevice.STATUS_REMOVED, NVMeDevice.STATUS_FAILED]: logger.info(f"Device is not unavailable: {device.get_id()}, {device.status} , stopping task") - task.function_result = f"stopped because dev is {device.status}" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - return True - - if task.status != JobSchedule.STATUS_RUNNING: - task.status = JobSchedule.STATUS_RUNNING - task.write_to_db(db.kv_store) + raise TaskAbort(f"stopped because dev is {device.status}") # set device online for the first 3 retries if task.retry < 3: @@ -301,58 +259,81 @@ def task_runner_device(task): # check device status time.sleep(5) device = _get_device(task) - if device.status == NVMeDevice.STATUS_ONLINE and device.io_error is False: - logger.info(f"Device is online: {device.get_id()}") - task.function_result = "done" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) + if device.status != NVMeDevice.STATUS_ONLINE or device.io_error is not False: + raise TaskRetry(f"Device is {device.status}, retry") - tasks_controller.add_device_mig_task_for_node(task.node_id) + logger.info(f"Device is online: {device.get_id()}") + task.function_result = "done" + tasks_controller.add_device_mig_task_for_node(task.node_id) - return True - task.retry += 1 - task.write_to_db(db.kv_store) - return False +def _give_up_on_device(task): + """The retry ceiling has terminated a device-restart task.""" + device = _get_device(task) + if device is None: + return + device_controller.device_set_unavailable(device.get_id()) + device_controller.device_set_retries_exhausted(device.get_id(), True) + + +def _abandon_task(task): + """Driver on_finish: the terminal paths a handler never reaches. + + Only two need anything. The retry ceiling gives up on the target, which for + a node means parking it OFFLINE and re-queueing, and for a device means + marking it unavailable and out of retries. A cancellation stops a device + task retrying forever. Everything else — success, the aborts the handler + raises itself — leaves the target alone. + """ + ceiling_reached = 0 <= task.max_retry <= task.retry + + if task.function_name == JobSchedule.FN_NODE_RESTART: + if ceiling_reached and not task.canceled: + _give_up_on_node(task) + return + + if task.canceled: + device = _get_device(task) + if device is not None: + device_controller.device_set_retries_exhausted(device.get_id(), True) + elif ceiling_reached: + _give_up_on_device(task) + + +def _give_up_on_node(task): + """The retry ceiling has terminated a node-restart task. Reached through + the driver's on_finish, since the handler never sees that path.""" + # restart_cleanup: this task ran try_set_node_restarting earlier + # and is the lock owner; tagging unblocks the RESTARTING-lock + # guard so the giving-up flip lands. + storage_node_ops.set_node_status( + task.node_id, StorageNode.STATUS_OFFLINE, caused_by="restart_cleanup") + # Re-queue a fresh auto-restart task so the node does not get + # stranded in OFFLINE forever. Without this, the legitimate + # auto-restart trigger (set_node_offline) won't fire either — + # it skips when status is already OFFLINE — so the only path + # back is operator intervention. Hours-of-backoff exhaustion + # almost always means a long peer-side recovery is in flight; + # once it clears, the new task can succeed. + try: + node_obj = db.get_storage_node_by_id(task.node_id) + tasks_controller.add_node_to_auto_restart(node_obj) + except KeyError: + logger.debug( + f"Node {task.node_id} no longer exists, skipping auto-restart re-queue") + except Exception as exc: + logger.error(f"Failed to re-queue auto-restart for {task.node_id}: {exc}") def task_runner_node(task): try: node = db.get_storage_node_by_id(task.node_id) except KeyError: - _task_finish(task, "node not found") - return True - - if task.retry >= task.max_retry: - if not _task_finish(task, "max retry reached"): - # Concurrently finished/canceled — someone else owns the outcome; - # the give-up side effects (OFFLINE flip + re-queue) must not run. - return True - # restart_cleanup: this task ran try_set_node_restarting earlier - # and is the lock owner; tagging unblocks the RESTARTING-lock - # guard so the giving-up flip lands. - storage_node_ops.set_node_status( - task.node_id, StorageNode.STATUS_OFFLINE, caused_by="restart_cleanup") - # Re-queue a fresh auto-restart task so the node does not get - # stranded in OFFLINE forever. Without this, the legitimate - # auto-restart trigger (set_node_offline) won't fire either — - # it skips when status is already OFFLINE — so the only path - # back is operator intervention. Hours-of-backoff exhaustion - # almost always means a long peer-side recovery is in flight; - # once it clears, the new task can succeed. - try: - node_obj = db.get_storage_node_by_id(task.node_id) - tasks_controller.add_node_to_auto_restart(node_obj) - except KeyError: - pass - except Exception as exc: - logger.error(f"Failed to re-queue auto-restart for {task.node_id}: {exc}") - return True + raise TaskAbort("node not found") if node.status in [StorageNode.STATUS_REMOVED, StorageNode.STATUS_SCHEDULABLE]: logger.info(f"Node is {node.status}, stopping task") - _task_finish(task, f"Node is {node.status}, stopping") - return True + raise TaskAbort(f"Node is {node.status}, stopping") # DOWN used to short-circuit here too. After removing the monitor's # set_node_online (which previously did DOWN -> ONLINE on health-check # pass), DOWN must be handled by this runner: shutdown + restart drives @@ -384,34 +365,28 @@ def task_runner_node(task): # a still-False health_check from auxiliary checks. if node.status == StorageNode.STATUS_ONLINE: logger.info(f"Node is online: {node.get_id()}") - _task_finish(task, "Node is online") - return True + task.function_result = "Node is online" + return - if task.canceled: - _task_finish(task, "canceled") - return True - - if task.status != JobSchedule.STATUS_RUNNING: - if node.status == StorageNode.STATUS_RESTARTING: - logger.info("Node is restarting, stopping task") - _task_finish(task, "Node is restarting") - return True - updated = _task_update( - task, lambda t: setattr(t, "status", JobSchedule.STATUS_RUNNING)) - if updated is None: - logger.info(f"Task {task.uuid} canceled/finished concurrently; stopping") - return True - task = updated + # A restart already in flight makes this task redundant — unless it is our + # own from an earlier attempt. This used to be inferred from the task still + # being NEW/SUSPENDED, which no longer distinguishes anything now that the + # driver moves the task to RUNNING before calling; the marker below records + # the fact directly, and also covers an attempt that issued the restart and + # then died, whose RESTARTING is ours to finish rather than defer to. + if (node.status == StorageNode.STATUS_RESTARTING + and not task.function_params.get("restart_issued")): + logger.info("Node is restarting, stopping task") + raise TaskAbort("Node is restarting") # Peer-restart mutual-exclusion pre-check: if any peer is RESTARTING # or IN_SHUTDOWN we cannot proceed (try_set_node_restarting in the # restart impl uses an FDB-tx with the same predicate and would fail # acquisition). This is purely transient — burning a retry on a lock - # we know we can't acquire just collapses the backoff budget. Return - # False without incrementing task.retry; the runner's outer loop - # will sleep with exponential backoff and re-call us. Once the peer - # finishes its transition, this check passes and we proceed with a - # fresh budget. + # we know we can't acquire just collapses the backoff budget, so it + # defers instead: no retry consumed, re-polled on the next short pass. + # Once the peer finishes its transition, this check passes and we + # proceed with a fresh budget. # # Skipped only for a SUSPENDED **and drained** cluster: recovery restarts # run in parallel then (see the dispatch loop below) so peers in @@ -419,10 +394,7 @@ def task_runner_node(task): # restart_storage_node is relaxed the same way (allow_concurrent_peers). # An operator-caused suspension never drains — its survivors still serve # IO — so it keeps the full pre-check. - cluster_obj = db.get_cluster_by_id(node.cluster_id) - if not (cluster_obj.status == Cluster.STATUS_SUSPENDED - and cluster_obj.suspend_drain_complete) \ - and not storage_node_ops.fd_dead_recovery_allowed(db, node): + if not _parallel_restart_allowed(node): # Strict one-restart-at-a-time outside the two sanctioned cases: # drained suspension, and a fully-dead failure domain # (fd_dead_recovery_allowed — no domain member ONLINE, so parallel @@ -435,17 +407,8 @@ def task_runner_node(task): continue if peer.status in (StorageNode.STATUS_RESTARTING, StorageNode.STATUS_IN_SHUTDOWN): - msg = (f"Peer {peer.get_id()[:8]} is {peer.status}; " - f"deferring (no retry consumed)") - logger.info(msg) - # Atomic CAS, NOT write_to_db of the stale in-memory copy: - # this exact write un-canceled a task that set_node_status - # (ONLINE) had canceled seconds earlier and erased its owner - # lease, resurrecting the restart (2026-07-29 double restart). - if _task_update( - task, lambda t, m=msg: setattr(t, "function_result", m)) is None: - return True - return False + raise TaskDefer(f"Peer {peer.get_id()[:8]} is {peer.status}; " + f"deferring (no retry consumed)") # is node reachable? ping_check = health_controller._check_node_ping(node.mgmt_ip) @@ -465,15 +428,8 @@ def task_runner_node(task): if data_ping_check is True: node_data_nic_ping_check = True if not ping_check or not node_api_check or not node_data_nic_ping_check: - # node is unreachable, retry logger.info(f"Node is not reachable: {task.node_id}, retry") - - def _unreachable(t): - t.function_result = "Node is unreachable, retry" - t.retry += 1 - if _task_update(task, _unreachable) is None: - return True - return False + raise TaskRetry("Node is unreachable, retry") # Last-line defense before the destructive shutdown/restart sequence: # everything above ran against reads taken seconds ago (the reachability @@ -481,17 +437,13 @@ def _unreachable(t): # (set_node_status(ONLINE) -> cancel_pending_node_restart_tasks) must stop # this entry HERE — in the 2026-07-29 double restart the second entry never # re-checked and force-shut a node that was back up and serving. - try: - task = db.get_task_by_id(task.uuid) - except KeyError: - return True - if task.canceled or task.status == JobSchedule.STATUS_DONE: + fresh = db.get_task_by_id(task.uuid) + if fresh is None or fresh.canceled or fresh.status == JobSchedule.STATUS_DONE: logger.info( f"Task {task.uuid} was canceled/finished concurrently; " f"stopping before shutdown") - if task.status != JobSchedule.STATUS_DONE: - _task_finish(task, "canceled") - return True + raise TaskAbort("canceled") + task = fresh # Cross-actor claim check on a fresh node read: a live driver (e.g. a # manual `sn restart`) mid-transition on this node holds the per-node @@ -505,18 +457,12 @@ def _unreachable(t): try: node = db.get_storage_node_by_id(task.node_id) except KeyError: - _task_finish(task, "node not found") - return True + raise TaskAbort("node not found") if node.status in (StorageNode.STATUS_RESTARTING, StorageNode.STATUS_IN_SHUTDOWN): claim_holder = db_controller.restart_claim_active(node) if claim_holder: - msg = (f"Node restart claim held by {claim_holder}; " - f"deferring (no retry consumed)") - logger.info(msg) - if _task_update( - task, lambda t, m=msg: setattr(t, "function_result", m)) is None: - return True - return False + raise TaskDefer(f"Node restart claim held by {claim_holder}; " + f"deferring (no retry consumed)") # Cleanup shutdown before the restart — but only when there is something # to clean: a node that is already OFFLINE had SPDK confirmed gone (that @@ -544,23 +490,25 @@ def _unreachable(t): if ret: logger.info("Node shutdown succeeded") shutdown_succeeded = True - - def _mark_cleanup(t): - t.function_params = dict(t.function_params) - t.function_params["cleanup_shutdown_done"] = True - updated = _task_update(task, _mark_cleanup) + updated = checkpoint(task, cleanup_shutdown_done=True) if updated is None: # Canceled under us right after the shutdown; do not # drive the restart of a canceled task. The monitor's # offline re-queue scan picks the node up again. - return True + raise TaskAbort("canceled during cleanup shutdown") task = updated else: logger.error("Node shutdown returned False; will retry after reset") time.sleep(3) + except (TaskAbort, TaskDefer, TaskRetry): + raise except Exception as e: logger.error(e) - return False + # Preserved as a defer, not a failure: this branch never + # consumed a retry, and restart's give-up has side effects + # (OFFLINE flip + re-queue) that a changed verdict would start + # triggering where it previously could not. + raise TaskDefer(f"cleanup shutdown raised: {e}") else: logger.info( f"Skipping cleanup shutdown for {node.get_id()}: " @@ -571,9 +519,7 @@ def _mark_cleanup(t): # of a half-shutdown node produced the in_restart hang we're guarding # against. Let the outer retry reattempt the whole cycle. if not shutdown_succeeded: - if _task_update(task, lambda t: setattr(t, "retry", t.retry + 1)) is None: - return True - return False + raise TaskRetry("Node shutdown did not succeed") try: # resetting node @@ -586,12 +532,21 @@ def _mark_cleanup(t): # never recognized as "ours" — masked only because this call # uses force=True, which proceeds past the guard regardless and # just logged a spurious "Restart task found" error every time. + # Recorded before the call: a restart that starts and then loses + # this process still owns the node's RESTARTING state, and the next + # attempt must recognise it as ours rather than stopping for it. + updated = checkpoint(task, restart_issued=True) + if updated is None: + raise TaskAbort("canceled before restart") + task = updated ret = storage_node_ops.restart_storage_node(node.get_id(), force=True, current_restart_task_id=task.uuid) if ret: logger.info("Node restart succeeded") + except (TaskAbort, TaskDefer, TaskRetry): + raise except Exception as e: logger.error(e) - return False + raise TaskDefer(f"restart raised: {e}") time.sleep(3) node = db.get_storage_node_by_id(task.node_id) @@ -625,12 +580,10 @@ def _mark_cleanup(t): # False at the moment we re-read the DB. if node.status == StorageNode.STATUS_ONLINE: logger.info(f"Node is online: {node.get_id()}") - _task_finish(task, "done") - return True + task.function_result = "done" + return - if _task_update(task, lambda t: setattr(t, "retry", t.retry + 1)) is None: - return True - return False + raise TaskRetry("Node did not come back online") finally: # On any non-success exit from the shutdown/restart sequence, make sure # we don't leave the node pinned in STATUS_IN_SHUTDOWN or @@ -646,21 +599,11 @@ def _mark_cleanup(t): logger.error(f"Post-task status reset check failed: {exc}") -# Per-task restart scheduling (in-memory; this runner is a single long-lived -# process). Maps task uuid -> epoch time when the task is next eligible to run. -# This lets the runner round-robin all restart tasks instead of pinning the -# thread on one task's blocking retry loop: a task that is waiting — deferred on -# a concurrent peer restart, or in failure backoff — no longer blocks the other -# pending restart tasks behind it (incident 2026-06-25: a single task deferring -# on a peer that was briefly in_restart sat in a growing backoff and starved a -# second node's brand-new restart task indefinitely). -_restart_next_attempt: dict = {} - # A genuine restart FAILURE first retries at a steady 1-minute cadence for a few # attempts (so a node that just needs a moment to come back recovers quickly), -# then falls back to the existing exponential backoff capped at +# then falls back to exponential backoff capped at # RESTART_TASK_EXEC_INTERVAL_MAX_SEC. A DEFER (peer-restart mutual exclusion) is -# NOT a failure and does not back off at all — see the loop below. +# NOT a failure and does not back off at all — the driver re-polls it next pass. RESTART_LEAD_IN_RETRIES = 3 RESTART_LEAD_IN_INTERVAL_SEC = 60 @@ -668,8 +611,8 @@ def _mark_cleanup(t): def _restart_backoff_seconds(retry): """Delay before the next attempt of a FAILED restart (one that consumed a retry). First RESTART_LEAD_IN_RETRIES attempts use a constant 1-minute - cadence; after that the existing exponential backoff applies, continuing - upward from the lead-in interval and capped at the configured maximum.""" + cadence; after that exponential backoff applies, continuing upward from the + lead-in interval and capped at the configured maximum.""" if retry <= RESTART_LEAD_IN_RETRIES: return RESTART_LEAD_IN_INTERVAL_SEC exp = RESTART_LEAD_IN_INTERVAL_SEC * (2 ** (retry - RESTART_LEAD_IN_RETRIES)) @@ -752,188 +695,37 @@ def _watchdog_orphaned_transitional_nodes(cluster_id): logger.info(f"Queued auto-restart for recovered node {node_id}") -# Parallel restart execution for SUSPENDED clusters: during full-cluster -# recovery every node is offline and no client IO flows, so node restarts -# cannot violate FTT and are fanned out on this pool (~70 s each; strictly -# sequential recovery of a 32-node cluster took ~38 min, 2026-07-08). The -# per-primary consistency of the cross-node connect section is preserved by -# storage_node_ops._remote_connect_gate, and the peer-exclusion guards -# (task_runner_node pre-check + try_set_node_restarting) are relaxed only -# while the cluster is SUSPENDED. Online clusters never dispatch here. -_restart_pool = ThreadPoolExecutor( - max_workers=constants.NODE_RESTART_MAX_PARALLEL_SUSPENDED, - thread_name_prefix="node-restart") -_restart_inflight: dict = {} # task uuid -> Future -# node_id -> Future. Parallel dispatch MUST also be exclusive per NODE, not -# only per task: multiple node_restart tasks can be queued for the same node -# (escalation + requeue paths), and keying inflight by task uuid alone let -# them run concurrently — each kill-and-restarting the same SPDK out from -# under the other, flipping the node offline/in_restart in a loop (observed -# 2026-07-10 mass-reboot recovery: 79 concurrent same-node dispatches, nodes -# stuck bouncing for 10+ minutes). -_node_inflight: dict = {} - - -def _process_restart_task(task_uuid): - """Claim and drive one restart task, including the per-task backoff - bookkeeping. Runs inline (serialized) normally, or on the - suspended-cluster parallel pool. Never raises: a crash in one task must - not kill recovery of every other node.""" - try: - # Re-read (it may have been canceled / changed concurrently). - task = db.get_task_by_id(task_uuid) - if task.status == JobSchedule.STATUS_DONE: - _restart_next_attempt.pop(task_uuid, None) - return - # Lease gate: do not drive a task another live runner host - # already owns (prevents a second replica issuing a - # concurrent shutdown/restart). - if not tasks_controller.claim_task(task): - logger.info(f"Restart task {task_uuid} owned by another runner host; skipping") - return - retry_before = task.retry - # Device restarts (and parts of node restarts outside the - # restart_storage_node wrapper) block without task writes; heartbeat - # the lease so it never goes stale mid-execution and gets stolen by - # another runner host. - with tasks_controller.task_lease_heartbeat(task): - res = task_runner(task) - task = db.get_task_by_id(task_uuid) - if res or task.status == JobSchedule.STATUS_DONE: - _restart_next_attempt.pop(task_uuid, None) - elif task.retry > retry_before: - # Genuine failure (retry consumed): 1-min lead-in, then - # exponential backoff. - _restart_next_attempt[task_uuid] = ( - time.time() + _restart_backoff_seconds(task.retry)) - else: - # Defer (peer-restart mutual exclusion; retry NOT - # consumed): not a failure — do not back off. Re-poll on - # the next short pass so this picks up immediately once - # the blocking restart finishes. - _restart_next_attempt[task_uuid] = ( - time.time() + constants.RESTART_TASK_EXEC_INTERVAL_SEC) - except Exception as e: - logger.error(f"Restart task {task_uuid} processing crashed: {e}") - logger.exception(e) - try: - retry = db.get_task_by_id(task_uuid).retry - except Exception: - retry = 0 - _restart_next_attempt[task_uuid] = ( - time.time() + _restart_backoff_seconds(retry)) +SPEC = RunnerSpec( + name="tasks-runner-restart", + function_names=[JobSchedule.FN_DEV_RESTART, JobSchedule.FN_NODE_RESTART], + handler=task_runner, + on_finish=_abandon_task, + on_cycle=lambda cluster: _watchdog_orphaned_transitional_nodes(cluster.get_id()), + is_eligible=_is_eligible, + interval=constants.TASK_EXEC_INTERVAL_SEC, + # Parallel restart execution for SUSPENDED clusters: during full-cluster + # recovery every node is offline and no client IO flows, so node restarts + # cannot violate FTT and are fanned out (~70 s each; strictly sequential + # recovery of a 32-node cluster took ~38 min, 2026-07-08). The per-primary + # consistency of the cross-node connect section is preserved by + # storage_node_ops._remote_connect_gate, and the peer-exclusion guards + # (the pre-check in task_runner_node + try_set_node_restarting) are relaxed + # under exactly the same condition. Online clusters stay sequential. + concurrency=constants.NODE_RESTART_MAX_PARALLEL_SUSPENDED, + serialize=_serialize, + # Never two restart tasks for the same node at once: multiple node_restart + # tasks can be queued for one node (escalation + requeue paths), and + # excluding by task alone let them run concurrently — each kill-and- + # restarting the same SPDK out from under the other, flipping the node + # offline/in_restart in a loop (2026-07-10 mass-reboot recovery: 79 + # concurrent same-node dispatches, nodes bouncing for 10+ minutes). + exclusion_key=lambda task: task.node_id or None, + backoff=_restart_backoff_seconds, +) def main(): - logger.info("Starting Tasks runner...") - while True: - try: - db.get_clusters() - except Exception as e: - logger.error(f"Failed to get clusters: {e}") - time.sleep(3) - continue - clusters = db.get_clusters() - if not clusters: - logger.error("No clusters found!") - else: - for cl in clusters: - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - for task in tasks: - if task.function_name not in [JobSchedule.FN_DEV_RESTART, JobSchedule.FN_NODE_RESTART]: - continue - if task.status == JobSchedule.STATUS_DONE: - _restart_next_attempt.pop(task.uuid, None) - _restart_inflight.pop(task.uuid, None) - continue - # Round-robin: skip a task that is not yet due so a waiting task - # (deferred on a concurrent peer restart, or in failure backoff) - # does NOT block the other pending restart tasks behind it. The - # outer loop revisits every task each pass (TASK_EXEC_INTERVAL_SEC). - if time.time() < _restart_next_attempt.get(task.uuid, 0): - continue - # Suspend recovery: while a SUSPENDED cluster is still being - # drained to all-offline, pause node restarts. Executing one now - # would fight the auto-shutdown and re-create the wedged - # half-restarted state we are fixing. Re-poll soon without - # consuming a retry; the task runs once the drain completes - # (suspend_drain_complete). - dispatch_parallel = False - if task.function_name == JobSchedule.FN_NODE_RESTART: - cl_fresh = db.get_cluster_by_id(cl.get_id()) - if tasks_controller.is_auto_restart_paused(cl_fresh): - logger.info( - "Cluster %s suspended and draining; deferring " - "node-restart task %s", cl.get_id(), task.uuid) - _restart_next_attempt[task.uuid] = ( - time.time() + constants.RESTART_TASK_EXEC_INTERVAL_SEC) - continue - # SUSPENDED and drained: full-cluster recovery — fan node - # restarts out on the pool (see _restart_pool). Online - # clusters stay strictly sequential. suspend_drain_complete - # is required: it certifies every (non operator-stopped) - # node went OFFLINE, i.e. no client IO — an operator-caused - # suspension never drains, its survivors are still serving, - # and its restarts must stay sequential with full guards. - # Parallel dispatch in the two sanctioned cases only: - # drained suspension (full-cluster recovery), and a - # fully-dead failure domain (fd_dead_recovery_allowed: - # no domain member ONLINE, no cross-domain restart in - # flight — recovery of a rebooted domain fans out - # instead of 16 x single-restart serially). The former - # relaxation that fanned out while the domain was - # still SERVING (2026-07-16 violation) stays removed. - dispatch_parallel = (cl_fresh.status == Cluster.STATUS_SUSPENDED - and cl_fresh.suspend_drain_complete) - if not dispatch_parallel and task.node_id: - try: - dispatch_parallel = storage_node_ops.fd_dead_recovery_allowed( - db, db.get_storage_node_by_id(task.node_id)) - except KeyError: - pass - - # Single dispatch path: EVERY execution goes through the - # pool with _restart_inflight/_node_inflight as the - # mutual-exclusion authority. The former split — a parallel - # branch that consulted the inflight maps and an inline - # branch that consulted neither — let a dispatch-mode flip - # mid-restart (fd_dead_recovery_allowed going false as the - # first peers of the domain came back ONLINE) re-enter the - # SAME task that was still running on the pool, which then - # force-shut the already-recovered node (2026-07-29 double - # restart). Serialized mode submits identically and just - # waits for the future, so the bookkeeping is the same in - # both modes and mode flips are harmless in both directions. - inflight = _restart_inflight.get(task.uuid) - if inflight is not None and not inflight.done(): - continue - # Per-node exclusion: never run two restart tasks for the - # same node concurrently (see _node_inflight above). The - # duplicate task re-polls next pass; by then the winner - # has usually completed and marked it obsolete. - node_inflight = _node_inflight.get(task.node_id) - if node_inflight is not None and not node_inflight.done(): - _restart_next_attempt[task.uuid] = ( - time.time() + constants.RESTART_TASK_EXEC_INTERVAL_SEC) - continue - fut = _restart_pool.submit(_process_restart_task, task.uuid) - _restart_inflight[task.uuid] = fut - if task.node_id: - _node_inflight[task.node_id] = fut - if not dispatch_parallel: - # Inline (serialized) execution: wait for this task - # before dispatching the next. _process_restart_task - # never raises, so a crash in one task cannot escape to - # the outer `while True` and kill recovery of every - # other node. - fut.result() - - try: - _watchdog_orphaned_transitional_nodes(cl.get_id()) - except Exception as e: - logger.error(f"Orphaned-node watchdog failed for cluster {cl.get_id()}: {e}") - - time.sleep(constants.TASK_EXEC_INTERVAL_SEC) + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/services/tasks_runner_sync_lvol_del.py b/simplyblock_core/services/tasks_runner_sync_lvol_del.py index 71b8eaafd1..897192f409 100644 --- a/simplyblock_core/services/tasks_runner_sync_lvol_del.py +++ b/simplyblock_core/services/tasks_runner_sync_lvol_del.py @@ -1,19 +1,38 @@ # coding=utf-8 -import sys +"""Task runner for the deferred per-node lvol operations. + +Two task families share this runner, both of them work that could not be done +inline on the owning node at the time it was requested: + +- ``FN_LVOL_SYNC_OP`` — re-apply a create-registration or a resize on a + non-leader node (incident 2026-07-10: an in-memory deferral queue was never + drained, so a volume's tertiary subsystem was never created). +- ``FN_LVOL_SYNC_DEL`` — delete a replica bdev on a secondary node, holding the + primary's del-sync lock until the task ends. +""" import time -from typing import Optional +from typing import NoReturn, Optional -from simplyblock_core import constants, db_controller, utils -from simplyblock_core.controllers import events_controller, snapshot_controller, tasks_controller +from simplyblock_core import constants, db_controller, storage_node_ops, utils +from simplyblock_core.controllers import events_controller, snapshot_controller from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.cluster import Cluster +from simplyblock_core.models.lvol_model import LVol from simplyblock_core.models.storage_node import StorageNode +from simplyblock_core.services.task_runner_base import ( + RunnerSpec, + TaskAbort, + TaskDefer, + TaskRetry, + serve, +) logger = utils.get_logger(__name__) # get DB controller db = db_controller.DBController() + def get_primary_node(task) -> Optional[StorageNode]: if "primary_node" in task.function_params: return db.get_storage_node_by_id(task.function_params["primary_node"]) @@ -24,256 +43,247 @@ def get_primary_node(task) -> Optional[StorageNode]: return None -# A persistent FDB read failure — or an unexpectedly empty cluster list — on this -# long-lived process means the FDB client is wedged: fdb.open() caches the -# Database and the network thread is per-process, so re-reading the same handle -# never recovers; only a fresh process does. Incident -# mass_create_delete_docker-20260629: this runner sat idle ~8h logging -# "No clusters found!" after a startup FDBError(1031), so 2,664 lvols never -# drained from in_deletion. Exit after this many consecutive failures so the -# orchestrator restarts us with a clean FDB connection (~3 min at the 3s tick). -_DB_FAILURE_RESTART_THRESHOLD = 60 - +def _is_already_gone(err): + """True when an RPC error means "the thing is not there" (-19/ENODEV). -def _count_failure_and_maybe_alert(task, node, what, msg): - """Count one genuine failure of this leg and escalate past the threshold. - - task.retry was never incremented by this runner (max_retry was declared - and never enforced), so it is free to use as what it actually is here: a - failure counter. It does not terminate anything -- these tasks are - unbounded by design, because giving up leaks the volume -- it only - decides when to raise an alert. - - The event fires once on crossing the threshold and again whenever the - error CHANGES, so a node that stays broken does not write an event every - 3 seconds. + For a delete that is success; for a register it is still a failure, but + it is not an actionable one to alert on repeatedly -- the object the + caller wanted registered has gone away underneath it. """ - previous = task.function_result - task.retry = (task.retry or 0) + 1 - if task.retry <= constants.TASK_FAILURE_ALERT_THRESHOLD: + if err is None: + return False + if isinstance(err, dict): + return err.get("code") == -19 + return "-19" in str(err) + + +# Last threshold-alert message reported per task. The driver clears +# ``function_result`` before every attempt, so the previous attempt's message +# cannot be read off the task; without this a node that stays broken would +# write one identical event per poll. Unlike the SYNC_DELETE_FAILED event, +# which dedupes through the driver's on_failure, this alert must also fire +# once on crossing the threshold with an UNCHANGED message, so it keeps its +# own memo. +_last_alerted_failure: dict = {} + + +def _alert_repeated_failure(task, node_id, what, msg): + """Escalate a failure that keeps repeating to the cluster event log. + + Both task families here are unbounded by design (``max_retry=-1``) -- + giving up leaks the volume -- so nothing else ever surfaces a leg that + fails permanently. Fires once on crossing the threshold and again only + when the error CHANGES. + """ + # The driver consumes this attempt's retry only after the handler returns. + attempts = (task.retry or 0) + 1 + threshold = constants.TASK_FAILURE_ALERT_THRESHOLD + if attempts <= threshold: return - if task.retry > constants.TASK_FAILURE_ALERT_THRESHOLD + 1 and previous == msg: + previous = _last_alerted_failure.get(task.uuid) + if attempts > threshold + 1 and previous == msg: return + _last_alerted_failure[task.uuid] = msg + try: events_controller.log_event_cluster( - cluster_id=node.cluster_id, + cluster_id=task.cluster_id, domain=events_controller.DOMAIN_STORAGE, event="LVOL_TASK_FAILING_REPEATEDLY", db_object=task, caused_by=events_controller.CAUSED_BY_MONITOR, - message=(f"{what} has failed {task.retry} times on node " - f"{node.get_id()}; it will keep retrying, but this is " - f"not a transient condition and needs attention. " - f"Last error: {msg}"), - node_id=node.get_id(), + message=(f"{what} has failed {attempts} times on node {node_id}; " + f"it will keep retrying, but this is not a transient " + f"condition and needs attention. Last error: {msg}"), + node_id=node_id, event_level="Critical") except Exception as event_error: logger.warning(f"Could not log repeated-failure event: {event_error}") -def _log_sync_delete_failure(task, node, lvol_bdev_name, msg): - """Record a failed sync delete in the cluster event log. + +def _run_sync_op(task): + lvol_id = task.function_params.get("lvol_id") + op = task.function_params.get("op") + + try: + lvol = db.get_lvol_by_id(lvol_id) + except KeyError: + # A missing record is normally obsolescence (the lvol was deleted) and + # completes the task. But add_lvol_ha queues these register tasks in + # its pre-check, BEFORE it writes the lvol record at the end of the + # create -- so this runner can see the task while the create is still + # running and permanently DROP the registration, leaving the replica + # missing and every later snapshot on that node failing (the LVOL_109 + # class). Inside the grace window, defer instead: once the record + # appears, the status check below takes over. + try: + age = time.time() - float(task.date or 0) + except (TypeError, ValueError): + # Unknown age: fall back to treating the record as genuinely gone, + # rather than deferring forever. + age = float("inf") + if age < constants.LVOL_SYNC_OP_RECORD_GRACE_SEC: + raise TaskDefer("lvol record not written yet, retrying") + raise TaskAbort("lvol no longer exists") + + if lvol.status == LVol.STATUS_IN_DELETION: + raise TaskAbort("lvol is being deleted") + if lvol.status != LVol.STATUS_ONLINE: + raise TaskDefer(f"lvol status is {lvol.status}, retrying") + + try: + node = db.get_storage_node_by_id(task.node_id) + except KeyError: + raise TaskAbort("node no longer exists") + + if node.get_id() not in lvol.nodes: + raise TaskAbort("node no longer hosts this lvol (topology moved)") + if node.status != StorageNode.STATUS_ONLINE: + raise TaskDefer(f"node is {node.status}, retrying") + if storage_node_ops.get_restart_phase(task.node_id, lvol.lvs_name): + # The owning flow (restart/activation/expansion) re-registers + # lvols itself; re-check once it has released the LVS. + raise TaskDefer("LVS owned by a restart/activation/expansion, retrying") + + if op == "register": + ok, err = storage_node_ops.repair_lvol_registration_on_non_leader( + lvol, node, task.function_params.get("secondary_index", 0)) + if not ok: + msg = f"registration failed: {err}" + if _is_already_gone(err): + # -19 / ENODEV: nothing to act on, do not count it. + raise TaskDefer(msg) + _alert_repeated_failure( + task, task.node_id, f"Deferred lvol {op} for {lvol_id}", msg) + raise TaskRetry(msg) + task.function_result = f"registered lvol {lvol_id} on {task.node_id}" + elif op == "resize": + # Converge to the CURRENT DB size — resize_lvol persists the new + # size after the fan-out, so this always applies the latest + # target even if the lvol was resized again meanwhile. + size_in_mib = utils.convert_size(lvol.size, 'MiB') + if not node.rpc_client(timeout=10, retry=2).bdev_lvol_resize( + f"{lvol.lvs_name}/{lvol.lvol_bdev}", size_in_mib): + raise TaskDefer("resize RPC failed, retrying") + task.function_result = f"resized lvol {lvol_id} on {task.node_id} to {size_in_mib} MiB" + else: + raise TaskAbort(f"unknown op {op!r}") + + +def alert_sync_delete_failure(task, reason): + """Report a failed sync delete in the cluster event log. This is the case an operator cannot see from the volume list alone: the async delete already SUCCEEDED, so the data is going away, but a node still - holds its replica bdev and the volume is pinned in_deletion until this - task drains. Emitted only when the failure message CHANGES (first failure, - or a different error), not on every 3s retry -- a node that stays down - would otherwise write thousands of identical events. + holds its replica bdev and the volume is pinned in_deletion until this task + drains. """ - if task.function_result == msg: + if task.function_name != JobSchedule.FN_LVOL_SYNC_DEL: return + + events_controller.log_event_cluster( + cluster_id=task.cluster_id, + domain=events_controller.DOMAIN_STORAGE, + event="SYNC_DELETE_FAILED", + db_object=task, + caused_by=events_controller.CAUSED_BY_MONITOR, + message=(f"Sync delete of {task.function_params['lvol_bdev_name']} failed on " + f"node {task.node_id} after a successful async delete; the volume " + f"stays in_deletion until this drains. {reason}"), + node_id=task.node_id, + event_level="Error") + + +def _fail_sync_del(task, node, lvol_bdev_name, msg) -> NoReturn: + _alert_repeated_failure( + task, node.get_id(), f"Sync delete of {lvol_bdev_name}", msg) + raise TaskRetry(msg) + + +def _run_sync_del(task): try: - events_controller.log_event_cluster( - cluster_id=node.cluster_id, - domain=events_controller.DOMAIN_STORAGE, - event="SYNC_DELETE_FAILED", - db_object=task, - caused_by=events_controller.CAUSED_BY_MONITOR, - message=(f"Sync delete of {lvol_bdev_name} failed on node " - f"{node.get_id()} after a successful async delete; the " - f"volume stays in_deletion until this drains. {msg}"), - node_id=node.get_id(), - event_level="Error") - except Exception as event_error: - logger.warning(f"Could not log sync-delete failure event: {event_error}") + node = db.get_storage_node_by_id(task.node_id) + except KeyError: + raise TaskAbort("node not found") + + # The node must be ONLINE, not merely "not in a transitional state". DOWN + # was accepted here, so the runner fired the RPC at a node that cannot + # answer; it failed and suspended, which works but is pure noise. register + # (_run_sync_op) already requires ONLINE. + if node.status != StorageNode.STATUS_ONLINE: + raise TaskDefer(f"Node is {node.status}, retry task") + + lvol_bdev_name = task.function_params["lvol_bdev_name"] + + # Re-check the restart phase. This leg was queued BECAUSE a restart owned + # the LVS state (check_non_leader_for_operation returns "queue" for + # PRE_BLOCK/BLOCKED/POST_UNBLOCK); draining it without re-testing that + # condition undoes the deferral the inline path just made. register already + # guards on this. + if storage_node_ops.get_restart_phase( + node.get_id(), lvol_bdev_name.split("/")[0]): + raise TaskDefer("LVS owned by a restart, retry task") + + logger.info(f"Sync delete bdev: {lvol_bdev_name} from node: {node.get_id()}") + try: + # Per-node lvstore lock: the sync delete mutates the replica blob tree + # and must not interleave with a create/register of another object on + # this node. + with snapshot_controller.lvstore_op_lock( + node.cluster_id, + lvol_bdev_name.split("/")[0], + node_id=node.get_id()): + ret, err = node.rpc_client().delete_lvol(lvol_bdev_name, sync=True) + except Exception as e: + _fail_sync_del( + task, node, lvol_bdev_name, + f"Sync delete of {lvol_bdev_name} on {node.get_id()} failed: {e}; will retry") + + if not ret: + if "code" not in err or err["code"] != -19: + # -19 never reaches here: it is handled below as success (the peer + # is already clean), so it is never counted. + _fail_sync_del( + task, node, lvol_bdev_name, + f"Failed to sync delete bdev: {lvol_bdev_name} from node: {node.get_id()}") + logger.error(f"Sync delete completed with error: {err}") + + task.function_result = f"bdev {lvol_bdev_name} deleted" + + +def process_task(task): + if task.function_name == JobSchedule.FN_LVOL_SYNC_OP: + _run_sync_op(task) + else: + _run_sync_del(task) + + +def release_del_sync_lock(task): + """Free the primary's del-sync lock once the delete task is over, however + it ended — the lock is keyed on there being no active task left.""" + _last_alerted_failure.pop(task.uuid, None) + + if task.function_name != JobSchedule.FN_LVOL_SYNC_DEL: + return + primary_node = get_primary_node(task) + if primary_node: + primary_node.lvol_del_sync_lock_reset() -def main(): - logger.info("Starting Tasks runner...") - _consecutive_db_failures = 0 - while True: - try: - clusters = db.get_clusters() - except Exception as e: - _consecutive_db_failures += 1 - logger.error(f"Failed to get clusters ({_consecutive_db_failures}): {e}") - if _consecutive_db_failures >= _DB_FAILURE_RESTART_THRESHOLD: - logger.error("FDB unreadable for too long; exiting for a clean restart") - sys.exit(1) - time.sleep(3) - continue - - if not clusters: - _consecutive_db_failures += 1 - logger.error(f"No clusters found! ({_consecutive_db_failures})") - if _consecutive_db_failures >= _DB_FAILURE_RESTART_THRESHOLD: - logger.error("No clusters readable for too long (FDB client likely wedged); " - "exiting for a clean restart") - sys.exit(1) - else: - for cl in clusters: - if cl.status == Cluster.STATUS_IN_ACTIVATION: - continue - - # An unhandled FDBError here (e.g. 1031 transaction timeout, - # 2026-07-16 run) killed the whole runner — count it toward - # the same wedge-detection threshold as get_clusters instead. - try: - tasks = db.get_job_tasks(cl.get_id(), reverse=False) - except Exception as e: - _consecutive_db_failures += 1 - logger.error(f"Failed to read tasks for cluster {cl.get_id()} " - f"({_consecutive_db_failures}): {e}") - if _consecutive_db_failures >= _DB_FAILURE_RESTART_THRESHOLD: - logger.error("FDB unreadable for too long; exiting for a clean restart") - sys.exit(1) - continue - _consecutive_db_failures = 0 - for task in tasks: - if task.function_name == JobSchedule.FN_LVOL_SYNC_OP: - if task.status != JobSchedule.STATUS_DONE: - # Re-read (it may have been canceled concurrently). - task = db.get_task_by_id(task.uuid) - if task.status == JobSchedule.STATUS_DONE: - continue - try: - tasks_controller.run_lvol_sync_op_task(task) - except Exception as e: - logger.error(f"lvol sync-op task {task.uuid} crashed: {e}") - continue - - if task.function_name == JobSchedule.FN_LVOL_SYNC_DEL: - if task.status != JobSchedule.STATUS_DONE: - - # get new task object because it could be changed from cancel task - task = db.get_task_by_id(task.uuid) - - if task.canceled: - task.function_result = "canceled" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - primary_node = get_primary_node(task) - if primary_node: - primary_node.lvol_del_sync_lock_reset() - continue - - node = db.get_storage_node_by_id(task.node_id) - - if not node: - task.function_result = "node not found" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - primary_node = db.get_storage_node_by_id(task.function_params["primary_node"]) - primary_node.lvol_del_sync_lock_reset() - continue - - # The node must be ONLINE, not merely "not in a - # transitional state". DOWN was accepted here, so - # the runner fired the RPC at a node that cannot - # answer; it failed and suspended, which works but - # is pure noise. register (run_lvol_sync_op_task) - # already requires ONLINE. - if node.status != StorageNode.STATUS_ONLINE: - msg = f"Node is {node.status}, retry task" - logger.info(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - continue - - # Re-check the restart phase. This leg was queued - # BECAUSE a restart owned the LVS state - # (check_non_leader_for_operation returns "queue" - # for PRE_BLOCK/BLOCKED/POST_UNBLOCK); draining it - # without re-testing that condition undoes the - # deferral the inline path just made. register - # already guards on this. - # Imported here, not at module scope: - # storage_node_ops pulls in the controllers that - # import this module (tasks_controller does the - # same for the identical reason). - from simplyblock_core import storage_node_ops - lvs_name_of_task = task.function_params[ - "lvol_bdev_name"].split("/")[0] - if storage_node_ops.get_restart_phase( - node.get_id(), lvs_name_of_task): - msg = "LVS owned by a restart, retry task" - logger.info(msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - continue - - if task.status != JobSchedule.STATUS_RUNNING: - task.status = JobSchedule.STATUS_RUNNING - task.write_to_db(db.kv_store) - - lvol_bdev_name = task.function_params["lvol_bdev_name"] - - logger.info(f"Sync delete bdev: {lvol_bdev_name} from node: {node.get_id()}") - try: - # Per-node lvstore lock: the sync delete mutates - # the replica blob tree and must not interleave - # with a create/register of another object on - # this node. The try also keeps a dead node from - # killing the runner: on 2026-07-16 an unhandled - # RPCException ('connection error') here took the - # whole service down and no deferred sync delete - # ever ran again. - with snapshot_controller.lvstore_op_lock( - node.cluster_id, - lvol_bdev_name.split("/")[0], - node_id=node.get_id()): - ret, err = node.rpc_client().delete_lvol(lvol_bdev_name, sync=True) - except Exception as e: - msg = (f"Sync delete of {lvol_bdev_name} on {node.get_id()} " - f"failed: {e}; will retry") - logger.error(msg) - _log_sync_delete_failure(task, node, lvol_bdev_name, msg) - _count_failure_and_maybe_alert( - task, node, - f"Sync delete of {lvol_bdev_name}", msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - continue - if not ret: - if "code" in err and err["code"] == -19: - logger.error(f"Sync delete completed with error: {err}") - else: - msg = f"Failed to sync delete bdev: {lvol_bdev_name} from node: {node.get_id()}" - logger.error(msg) - _log_sync_delete_failure(task, node, lvol_bdev_name, msg) - # -19 never reaches here: it is handled - # above as success (the peer is already - # clean), so it is never counted. - _count_failure_and_maybe_alert( - task, node, - f"Sync delete of {lvol_bdev_name}", msg) - task.function_result = msg - task.status = JobSchedule.STATUS_SUSPENDED - task.write_to_db(db.kv_store) - continue - - task.function_result = f"bdev {lvol_bdev_name} deleted" - task.status = JobSchedule.STATUS_DONE - task.write_to_db(db.kv_store) - primary_node = get_primary_node(task) - if primary_node: - primary_node.lvol_del_sync_lock_reset() - - time.sleep(3) +SPEC = RunnerSpec( + name="tasks-runner-sync-lvol-del", + function_names=[JobSchedule.FN_LVOL_SYNC_OP, JobSchedule.FN_LVOL_SYNC_DEL], + handler=process_task, + on_finish=release_del_sync_lock, + on_failure=alert_sync_delete_failure, + is_eligible=lambda task, cluster: cluster.status != Cluster.STATUS_IN_ACTIVATION, + interval=3, +) + + +def main(): + serve(SPEC) if __name__ == "__main__": diff --git a/simplyblock_core/storage_node_ops.py b/simplyblock_core/storage_node_ops.py index 17f30a7f1f..58fb68681c 100644 --- a/simplyblock_core/storage_node_ops.py +++ b/simplyblock_core/storage_node_ops.py @@ -7069,16 +7069,10 @@ def shutdown_storage_node(node_id, force=False, keep_auto_restart=False, snode.write_to_db(db_controller.kv_store) # Step 2: cancel migration tasks while controllers are still up. - pending_tasks = db_controller.get_job_tasks(snode.cluster_id) - for task in pending_tasks: - if task.node_id != node_id or task.status == JobSchedule.STATUS_DONE: - continue - if task.function_name in [ - JobSchedule.FN_DEV_MIG, - JobSchedule.FN_NEW_DEV_MIG, - ]: - task.canceled = True - task.write_to_db(db_controller.kv_store) + tasks_controller.cancel_node_tasks(snode.cluster_id, node_id, [ + JobSchedule.FN_DEV_MIG, + JobSchedule.FN_NEW_DEV_MIG, + ]) if not force: # Step 3 (Loop 1): broadcast device-unavailable events. The diff --git a/simplyblock_core/test/test_replication_relationship_and_delete_source.py b/simplyblock_core/test/test_replication_relationship_and_delete_source.py index a297391a0e..f91ee5ae9b 100644 --- a/simplyblock_core/test/test_replication_relationship_and_delete_source.py +++ b/simplyblock_core/test/test_replication_relationship_and_delete_source.py @@ -211,36 +211,38 @@ def _delete(lvol, **kw): monkeypatch.setattr(lvol_controller, "delete_lvol", _delete) task = _Task(delete_source) - ok = trf._finalize(task, True, "") - return ok, task, events + # The handler's success path. The task's terminal state is written by the + # task runner afterwards, so what this observes is the ordering of the + # source retirement against the durable cutover state. + trf._record_cutover_done(task) + task.function_result = "cutover done" + trf._stop_source_replication(task) + trf._delete_source_if_requested(task) + return task, events def test_cutover_stops_replication_on_the_source(monkeypatch): """Observed 2026-08-21: after "cutover done" the source kept taking and replicating cadence snapshots. The hand-off must stop the source.""" - ok, _task, events = _run_finalize(monkeypatch, delete_source=False) - assert ok is True + _task, events = _run_finalize(monkeypatch, delete_source=False) assert ("src_config", False, 0, "") in events, "the source's replication config must be cleared at cutover" def test_source_deleted_only_after_cutover_state_is_durable(monkeypatch): - ok, task, events = _run_finalize(monkeypatch, delete_source=True) - assert ok is True + task, events = _run_finalize(monkeypatch, delete_source=True) assert ("delete", "SRC1") in events state_at = events.index(("state", LVolReplication.STATE_CUTOVER_DONE)) delete_at = events.index(("delete", "SRC1")) assert state_at < delete_at, "the cutover state must be durable BEFORE the delete" - assert task.status == JobSchedule.STATUS_DONE + assert task.function_result == "cutover done" def test_source_kept_without_the_flag(monkeypatch): - ok, _task, events = _run_finalize(monkeypatch, delete_source=False) - assert ok is True + _task, events = _run_finalize(monkeypatch, delete_source=False) assert not any(e[0] == "delete" for e in events) def test_failed_source_delete_does_not_unsucceed_the_cutover(monkeypatch): - ok, task, _events = _run_finalize(monkeypatch, delete_source=True, - delete_raises=True) - assert ok is True - assert task.status == JobSchedule.STATUS_DONE + task, _events = _run_finalize(monkeypatch, delete_source=True, + delete_raises=True) + assert task.function_result == "cutover done" diff --git a/simplyblock_core/test/test_tasks_runner_replication_final.py b/simplyblock_core/test/test_tasks_runner_replication_final.py index 787ec9f69e..5b812309e7 100644 --- a/simplyblock_core/test/test_tasks_runner_replication_final.py +++ b/simplyblock_core/test/test_tasks_runner_replication_final.py @@ -1,7 +1,16 @@ -"""D6 unit tests for the replication-final task runner lifecycle.""" +"""D6 unit tests for the replication-final task handler. + +The runner sits on the shared driver (``task_runner_base``), so the handler +is void: it returns on success, raises ``TaskRetry`` for every outcome the +driver should suspend and re-attempt, and ``TaskDefer`` while the delta-shrink +phase waits on the target without burning a retry. Task status/retry transitions +themselves belong to the driver and are covered by +tests/unit/tasks/test_task_runner_base.py. +""" import pytest from simplyblock_core.services import tasks_runner_replication_final as runner +from simplyblock_core.services.task_runner_base import TaskDefer, TaskRetry from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.lvol_model import LVol, LVolReplication from simplyblock_core.models.storage_node import StorageNode @@ -76,58 +85,57 @@ def _run_cutover(src, tgt, lvol, comp, map_id, snap, operation="replicate"): return calls -def test_happy_path_marks_done_and_updates_state(monkeypatch): +def test_happy_path_returns_and_updates_state(monkeypatch): rep = LVolReplication() rep.state = LVolReplication.STATE_CUTOVER_PENDING nodes = {"S1": _node("S1"), "T1": _node("T1")} calls = _install(monkeypatch, nodes, rep, (True, None)) - res = runner.task_runner(_task()) + task = _task() + assert runner.task_runner(task) is None - assert res is True assert len(calls) == 1 assert calls[0][5] == "replicate" assert rep.state == LVolReplication.STATE_CUTOVER_DONE + assert task.function_result == "cutover done" + assert task.function_params["start_time"] <= task.function_params["end_time"] -def test_failure_suspends_and_retries(monkeypatch): +def test_failed_cutover_is_retryable(monkeypatch): rep = LVolReplication() nodes = {"S1": _node("S1"), "T1": _node("T1")} _install(monkeypatch, nodes, rep, (False, "boom")) - task = _task() - res = runner.task_runner(task) - - assert res is False - assert task.status == JobSchedule.STATUS_SUSPENDED - assert task.retry == 1 - assert task.function_result == "boom" + with pytest.raises(TaskRetry, match="boom"): + runner.task_runner(_task()) -def test_max_retry_marks_done_without_cutover(monkeypatch): +def test_target_offline_is_retryable_without_cutover(monkeypatch): rep = LVolReplication() - nodes = {"S1": _node("S1"), "T1": _node("T1")} + nodes = {"S1": _node("S1"), "T1": _node("T1", status=StorageNode.STATUS_OFFLINE)} calls = _install(monkeypatch, nodes, rep, (True, None)) - task = _task() - task.retry = 5 # == max_retry - res = runner.task_runner(task) + with pytest.raises(TaskRetry): + runner.task_runner(_task()) + assert calls == [] - assert res is True - assert task.status == JobSchedule.STATUS_DONE - assert calls == [] # cutover never attempted +def test_missing_source_node_is_retryable_without_cutover(monkeypatch): + rep = LVolReplication() + calls = _install(monkeypatch, {"T1": _node("T1")}, rep, (True, None)) -def test_target_offline_suspends(monkeypatch): + with pytest.raises(TaskRetry, match="source node not found"): + runner.task_runner(_task()) + assert calls == [] + + +def test_missing_lvol_id_is_retryable(monkeypatch): rep = LVolReplication() - nodes = {"S1": _node("S1"), "T1": _node("T1", status=StorageNode.STATUS_OFFLINE)} + nodes = {"S1": _node("S1"), "T1": _node("T1")} calls = _install(monkeypatch, nodes, rep, (True, None)) - task = _task() - res = runner.task_runner(task) - - assert res is False - assert task.status == JobSchedule.STATUS_SUSPENDED + with pytest.raises(TaskRetry, match="missing lvol_id"): + runner.task_runner(_task(lvol_id="")) assert calls == [] @@ -181,9 +189,8 @@ def test_shrink_waits_until_replicated(monkeypatch): runner, task = _mk(monkeypatch, {"S1": _ShrinkSnap(replicated=False)}, {"shrink_round": 1, "shrink_snap_id": "S1", "shrink_deadline": 2**60}) - done, err = runner._shrink_step(task, _ShrinkLvol()) - assert (done, err) == (False, None) - assert "waiting" in task.function_result + with pytest.raises(TaskDefer, match="waiting"): + runner._shrink_step(task, _ShrinkLvol()) def test_shrink_takes_next_snapshot_immediately(monkeypatch): @@ -198,8 +205,8 @@ def _add(lid, name, snap_type="user"): import simplyblock_core.controllers.snapshot_controller as sc monkeypatch.setattr(sc, "add", _add) - done, err = runner._shrink_step(task, _ShrinkLvol()) - assert (done, err) == (False, None) + with pytest.raises(TaskDefer): + runner._shrink_step(task, _ShrinkLvol()) assert taken == [("LV1", "internal")] or taken[0][0] == "LV1" assert task.function_params["shrink_round"] == 2 assert task.function_params["shrink_snap_id"] == "S2" @@ -209,8 +216,7 @@ def test_shrink_completes_after_last_round(monkeypatch): runner, task = _mk(monkeypatch, {"S2": _ShrinkSnap(replicated=True)}, {"shrink_round": runner_rounds(), "shrink_snap_id": "S2", "shrink_deadline": 2**60}) - done, err = runner._shrink_step(task, _ShrinkLvol()) - assert (done, err) == (True, None), "cutover must start IMMEDIATELY after the last round" + runner._shrink_step(task, _ShrinkLvol()) # returns => cutover starts IMMEDIATELY def runner_rounds(): @@ -222,5 +228,5 @@ def test_shrink_deadline_aborts(monkeypatch): runner, task = _mk(monkeypatch, {"S1": _ShrinkSnap(replicated=False)}, {"shrink_round": 1, "shrink_snap_id": "S1", "shrink_deadline": 1}) - done, err = runner._shrink_step(task, _ShrinkLvol()) - assert done is False and err and "timed out" in err + with pytest.raises(TaskRetry, match="timed out"): + runner._shrink_step(task, _ShrinkLvol()) diff --git a/simplyblock_core/utils/__init__.py b/simplyblock_core/utils/__init__.py index f7000e32c1..e854e358f8 100644 --- a/simplyblock_core/utils/__init__.py +++ b/simplyblock_core/utils/__init__.py @@ -32,7 +32,6 @@ from simplyblock_core import constants from simplyblock_core import shell_utils -from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.nvme_device import NVMeDevice from simplyblock_web import node_utils @@ -940,50 +939,6 @@ def strfdelta_seconds(remainder: int) -> str: return out.strip() -def handle_task_result(task: JobSchedule, res: dict, allowed_error_codes=None, allow_all_errors=False): - if res: - if not allowed_error_codes: - allowed_error_codes = [0] - - res_data = res[0] - migration_status = res_data.get("status") - error_code = res_data.get("error", -1) - progress = res_data.get("progress", -1) - if migration_status == "completed": - if error_code == 0: - task.function_result = "Done" - task.status = JobSchedule.STATUS_DONE - elif error_code in allowed_error_codes or allow_all_errors: - task.function_result = f"mig completed with status: {error_code}" - task.status = JobSchedule.STATUS_DONE - else: - task.function_result = f"mig error: {error_code}, retrying" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - del task.function_params['migration'] - - task.write_to_db() - return True - - elif migration_status == "failed": - task.status = JobSchedule.STATUS_DONE - task.function_result = migration_status - task.write_to_db() - return True - - elif migration_status == "none": - task.function_result = "mig retry after restart" - task.retry += 1 - task.status = JobSchedule.STATUS_SUSPENDED - del task.function_params['migration'] - task.write_to_db() - return True - - else: - task.function_result = f"Status: {migration_status}, progress:{progress}" - task.write_to_db() - else: - logger.error("Failed to get mig status") logger = get_logger(__name__) @@ -3258,11 +3213,11 @@ def patch_prometheus_configmap(username: str, password: str): return False -def create_docker_service(cluster_docker: DockerClient, service_name: str, service_file: str, service_image: str): +def create_docker_service(cluster_docker: DockerClient, service_name: str, command: List[str], service_image: str): logger.info(f"Creating service: {service_name}") cluster_docker.services.create( image=service_image, - command=service_file, + command=list(command), name=service_name, mounts=["/etc/foundationdb:/etc/foundationdb"], env=["SIMPLYBLOCK_LOG_LEVEL=DEBUG"], @@ -3275,7 +3230,7 @@ def create_docker_service(cluster_docker: DockerClient, service_name: str, servi def create_k8s_service(namespace: str, deployment_name: str, - container_name: str, service_file: str, container_image: str): + container_name: str, command: List[str], container_image: str): logger.info(f"Creating deployment: {deployment_name} in namespace {namespace}") load_kube_config_with_fallback() apps_v1 = client.AppsV1Api() @@ -3308,7 +3263,7 @@ def create_k8s_service(namespace: str, deployment_name: str, container = V1Container( name=container_name, image=container_image, - command=["python", service_file], + command=list(command), env=env_list, volume_mounts=volume_mounts, resources=V1ResourceRequirements( diff --git a/tests/integration/test_port_allow_recovery.py b/tests/integration/test_port_allow_recovery.py index da53989864..01a66e46c8 100644 --- a/tests/integration/test_port_allow_recovery.py +++ b/tests/integration/test_port_allow_recovery.py @@ -63,6 +63,7 @@ from simplyblock_core.models.storage_node import StorageNode from simplyblock_core.models.nvme_device import NVMeDevice from simplyblock_core.models.lvol_model import LVol +from simplyblock_core.services.task_runner_base import TaskAbort, TaskDefer def _make_lvol(uuid, status=None): @@ -327,7 +328,6 @@ def _run(self): def test_full_cluster_map_push_is_gone(self): self._run() self.cluster_map_push.assert_not_called() - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_readmit_uses_node_recovery_cause_and_precedes_unblock(self): self._run() @@ -387,9 +387,8 @@ def test_readmit_refused_suspends_task(self): "simplyblock_core.services.tasks_runner_port_allow." "device_controller.device_set_online", return_value=False, - ): + ), self.assertRaises(TaskDefer): exec_port_allow_task(self.task) - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) self.assertEqual(self._node_allows(), [], "no firewall allow when the pre-unblock re-admit " "was refused") @@ -403,9 +402,8 @@ def test_broadcast_failure_suspends_task(self): "simplyblock_core.services.tasks_runner_port_allow." "distr_controller.send_dev_status_event", side_effect=Exception("distrib send failed"), - ): + ), self.assertRaises(TaskDefer): exec_port_allow_task(self.task) - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) self.assertEqual(self._node_allows(), [], "no firewall allow when the device-status refresh " "failed") @@ -443,21 +441,20 @@ def _node_allows(self): ] def test_data_nic_down_suspends(self): - self._run_with_data_ping(False) - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) + with self.assertRaises(TaskDefer): + self._run_with_data_ping(False) self.assertEqual(self._node_allows(), [], "the port must not open while the node's data NIC is down") def test_data_nic_inconclusive_suspends(self): - self._run_with_data_ping(None) - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) + with self.assertRaises(TaskDefer): + self._run_with_data_ping(None) self.assertEqual(self._node_allows(), [], "an inconclusive data-NIC ping must not allow the port — " "recovery requires positive confirmation") def test_data_nic_up_proceeds(self): self._run_with_data_ping(True) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) self.assertEqual(len(self._node_allows()), 1) @@ -560,7 +557,6 @@ def test_no_failback_when_no_peer_leader(self): jc_calls = [c for c in self.calls if c[0] == "jc_disable_replication"] self.assertEqual(jc_calls, [], "no quiesce without an acting leader") self.assertEqual(len(self._node_allows()), 1) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_zero_leader_left_alone(self): # Nobody holds leadership (no-abort outage aftermath): the runner @@ -576,7 +572,6 @@ def test_zero_leader_left_alone(self): "a zero-leader LVS is left for the primary's " "promotion-on-first-IO — no CP-side healing") self.assertEqual(len(self._node_allows()), 1) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_quiesce_loop_retries_locally_then_succeeds(self): # jc_disable_replication False = active replication -> re-quiesce @@ -597,7 +592,6 @@ def _jc(vuid): demotes = [c for c in self.calls if c[0] == "bdev_lvol_set_leader" and c[1] == "sec"] self.assertEqual(len(demotes), 1) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_quiesce_loop_exhaustion_suspends(self): from simplyblock_core.services.tasks_runner_port_allow import ( @@ -609,11 +603,11 @@ def _jc(vuid): return False self.sec_rpc.jc_disable_replication.side_effect = _jc - self._run() + with self.assertRaises(TaskDefer): + self._run() jc_calls = [c for c in self.calls if c[0] == "jc_disable_replication"] self.assertEqual(len(jc_calls), _REPL_SUSPEND_MAX_ATTEMPTS, "the quiesce+disable loop is bounded locally") - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) demotes = [c for c in self.calls if c[0] == "bdev_lvol_set_leader"] self.assertEqual(demotes, [], "no demote against active journal replication") @@ -622,8 +616,8 @@ def _jc(vuid): def test_jc_disable_raise_suspends(self): self.sec_rpc.jc_disable_replication.side_effect = Exception("jc rpc timeout") - self._run() - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) + with self.assertRaises(TaskDefer): + self._run() demotes = [c for c in self.calls if c[0] == "bdev_lvol_set_leader"] self.assertEqual(demotes, []) self.assertEqual(self._node_allows(), []) @@ -638,9 +632,8 @@ def test_down_acting_leader_hublvol_is_still_a_hard_gate(self): "simplyblock_core.services.tasks_runner_port_allow." "_verify_or_reconnect_peer_hublvol", return_value=False, - ): + ), self.assertRaises(TaskDefer): self._run() - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) demotes = [c for c in self.calls if c[0] == "bdev_lvol_set_leader"] self.assertEqual(demotes, [], "no demote while the acting leader's hublvol to the " @@ -722,7 +715,6 @@ def test_demote_lands_on_tertiary_only_and_is_plain(self): if c[0] == "bdev_distrib_force_to_non_leader"] self.assertEqual(forces, [], "no force_to_non_leader in the minimal failback") - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_quiesce_verify_demote_order_with_tertiary_leader(self): self._run() @@ -1021,7 +1013,7 @@ def test_strict_verify_exhausts_retries_aborts_recovering_node(self): ) as reconnect_mock, patch( "simplyblock_core.services.tasks_runner_port_allow._abort_recovering_node", side_effect=lambda n, r: self.calls.append(("abort_recovering_node", n.uuid, r)), - ) as abort_mock: + ) as abort_mock, self.assertRaises(TaskAbort): exec_port_allow_task(self.task) abort_mock.assert_called_once() @@ -1037,8 +1029,8 @@ def test_strict_verify_exhausts_retries_aborts_recovering_node(self): if c[0] == "firewall_set_port" and c[3] == "allow"] self.assertEqual(allow_calls, []) - # Task ended in DONE (not SUSPENDED — the retries already ran). - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) + # TaskAbort, not TaskDefer: the retries already ran, so the driver + # must finish the task rather than schedule another attempt. # 5 reconnect attempts were issued (one per retry iteration). self.assertEqual(reconnect_mock.call_count, 5) @@ -1141,22 +1133,23 @@ def test_abort_helper_present_and_used(self): self.assertIn("_abort_recovering_node(node, reason)", self.src) def test_no_port_allowed_after_abort(self): - # Source-level invariant: when the abort path runs the task returns - # without falling through to firewall_set_port/port_allowed. - # We assert that abort sets task status DONE and the function - # returns before the firewall/port_allowed lines. + # Source-level invariant: when the abort path runs, the handler + # leaves via TaskAbort without falling through to + # firewall_set_port/port_allowed. The raise is what makes the + # fall-through impossible — it replaced an explicit STATUS_DONE + + # return when the runner moved onto the task runner base. i_abort = self.src.find("_abort_recovering_node(node, reason)") - i_done = self.src.find("STATUS_DONE", i_abort) - i_return = self.src.find("return", i_done) + i_raise = self.src.find("raise TaskAbort", i_abort) # The invariant is about the port_allowed occurrence in - # exec_port_allow_task AFTER the abort path's return. + # exec_port_allow_task AFTER the abort path's raise. i_allow_event = self.src.find("tcp_ports_events.port_allowed", i_abort) self.assertGreater(i_abort, 0) - self.assertGreater(i_done, i_abort) - self.assertGreater(i_return, i_done) - self.assertGreater(i_allow_event, i_return, + self.assertGreater(i_raise, i_abort, + "the abort path must terminate the handler with " + "TaskAbort, not fall through") + self.assertGreater(i_allow_event, i_raise, "tcp_ports_events.port_allowed must appear after " - "the abort path's return so it cannot fire on the " + "the abort path's raise so it cannot fire on the " "abort code path") @@ -1257,9 +1250,8 @@ def test_pre_unblock_refusal_suspends_task(self): "simplyblock_core.services.tasks_runner_port_allow." "device_controller.device_set_online", return_value=False, - ): + ), self.assertRaises(TaskDefer): self._run() - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) node_allows = [ c for c in self.calls if c[0] == "firewall_set_port" and c[1] == self.node.uuid @@ -1291,7 +1283,6 @@ def _refuse_epilogue(dev_id, *a, **kw): self.assertTrue( any("refused" in line for line in logs.output), "a refused re-admit must be logged, never silent") - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) class _SecRoleReconnectBase(_BasePortAllowTest): @@ -1405,21 +1396,21 @@ def test_tertiary_path_failure_is_a_hard_gate(self): self.node_rpc.subsystem_list.return_value = [{"nqn": "nqn-p-hub"}] self.node_rpc.subsystem_get.return_value = {"nqn": "nqn-p-hub"} self.tert.add_hublvol_failover_path = MagicMock(return_value=False) - self._run_with_verify(lambda *a: True) + with self.assertRaises(TaskDefer): + self._run_with_verify(lambda *a: True) node_allows = [ c for c in self.calls if c[0] == "firewall_set_port" and c[1] == self.node.uuid and c[3] == "allow" ] self.assertEqual(node_allows, [], "the port must not open with a broken tertiary redirect") - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) def test_outbound_failure_suspends_task(self): def _fail_only_own(peer, primary): # Fail the node->prim direction; pass the peer-gate direction. return not (peer is self.node and primary is self.prim) - self._run_with_verify(_fail_only_own) - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) + with self.assertRaises(TaskDefer): + self._run_with_verify(_fail_only_own) node_allows = [ c for c in self.calls if c[0] == "firewall_set_port" and c[1] == self.node.uuid and c[3] == "allow" @@ -1439,7 +1430,6 @@ def test_offline_primary_outbound_skipped(self): self.assertEqual(own_calls, [], "no OUTBOUND reconnect toward a non-ONLINE primary — " "its own recovery path re-drives that leg") - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_follower_recovery_is_hublvol_wiring_only(self): # Scenario (iv): the recovering node is a plain follower of an @@ -1462,7 +1452,6 @@ def test_follower_recovery_is_hublvol_wiring_only(self): "triggers no leadership action at all") ana_mock.assert_not_called() self.assertEqual(len(self._node_allows()), 1) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_no_ana_promotion_when_primary_online(self): # The deferred post-unblock ANA promotion is gated on the primary @@ -1478,7 +1467,6 @@ def test_no_ana_promotion_when_primary_online(self): ) as ana_mock: self._run_with_verify(lambda *a: True) ana_mock.assert_not_called() - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) class TestStaleLeaderConvergence(_SecRoleReconnectBase): @@ -1538,7 +1526,6 @@ def test_stale_claim_not_converged_by_runner(self): if c[0] == "bdev_distrib_force_to_non_leader" and c[1] == "node"] self.assertEqual(forces, []) self.assertEqual(len(self._node_allows()), 1) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_no_demote_when_primary_not_leading(self): # The node is then the legitimate acting leader; demoting it would @@ -1547,13 +1534,11 @@ def test_no_demote_when_primary_not_leading(self): self.prim_rpc.bdev_lvol_get_lvstores.return_value = [{"lvs leadership": False}] self._run() self.assertEqual(self._node_demotes(), []) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_no_action_without_stale_claim(self): self.lvs_leadership_on_node["LVS_P"] = False self._run() self.assertEqual(self._node_demotes(), []) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) class TestNoFollowerPortFencing(_SecRoleReconnectBase): @@ -1583,13 +1568,12 @@ def test_down_follower_never_fenced(self): "at first contact — the fencing was removed on 2026-07-07") self.assertNotIn("fenced_ports", self.task.function_params, "no fence record is ever written into the task") - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_suspension_leaves_no_block_behind(self): def _fail_own(peer, primary): return not (peer is self.node and primary is self.prim) - self._run_with_verify(_fail_own) - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) + with self.assertRaises(TaskDefer): + self._run_with_verify(_fail_own) self.assertEqual( self._blocks(), [], "a suspended recovery must not leave any port blocked — the " @@ -1637,12 +1621,11 @@ def test_tertiary_path_gated_before_allow(self): "the tertiary->secondary hublvol must be connected " "BEFORE the port opens — the tertiary is the acting " "leader and must be able to redirect here") - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_tertiary_path_failure_suspends(self): self.tert.add_hublvol_failover_path = MagicMock(return_value=False) - self._run_with_verify(lambda *a: True) - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) + with self.assertRaises(TaskDefer): + self._run_with_verify(lambda *a: True) node_allows = [ c for c in self.calls if c[0] == "firewall_set_port" and c[1] == self.node.uuid and c[3] == "allow" @@ -1659,7 +1642,6 @@ def test_down_primary_also_gates_tertiary_path(self): self._run_with_verify(lambda *a: True) tert_paths = [c for c in self.calls if c[0] == "tert_path"] self.assertEqual(len(tert_paths), 1) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) # --- deferred post-unblock ANA promotion (primary OFFLINE) ---------- @@ -1705,7 +1687,6 @@ def test_ana_switchback_promotes_secondary_listeners_post_unblock(self): "must never gate the recovery") self.assertLess(i_event, i_ana, "the promotion runs after the port_allowed event") - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_ana_promotion_failure_is_best_effort(self): # A failing per-lvol promotion must not fail the (already @@ -1717,7 +1698,6 @@ def _boom(lvol, target, state): raise Exception("listener set-ana rpc failed") self._run_with_ana(ana_side=_boom) self.assertEqual(len(self._node_allows()), 1) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) @unittest.skip("pending design call 2026-07: deferred ANA DEMOTE (a " @@ -1779,12 +1759,11 @@ def test_redirect_rewired_toward_acting_leader(self): self.assertIs(call.args[0], self.sec1) self.assertEqual(call.kwargs.get("role"), "tertiary") self.assertIs(call.kwargs.get("lvs_node"), self.prim2) - self.assertEqual(self.task.status, JobSchedule.STATUS_DONE) def test_failure_suspends_before_port_opens(self): self.node.connect_to_hublvol = MagicMock(return_value=False) - self._run() - self.assertEqual(self.task.status, JobSchedule.STATUS_SUSPENDED) + with self.assertRaises(TaskDefer): + self._run() node_allows = [ c for c in self.calls if c[0] == "firewall_set_port" and c[1] == self.node.uuid and c[3] == "allow" diff --git a/tests/integration/test_tasks_runner_cluster_expand.py b/tests/integration/test_tasks_runner_cluster_expand.py index 7fae28f68c..98e70afa1c 100644 --- a/tests/integration/test_tasks_runner_cluster_expand.py +++ b/tests/integration/test_tasks_runner_cluster_expand.py @@ -4,6 +4,10 @@ No FDB / SPDK: ``integrate_new_node_into_cluster`` and the DB handle are mocked, so these run in milliseconds. This is the "fast tier" that lets expansion logic be developed without the multi-hour real-FDB simulation. + +Scope is the handler only. Cancellation, max-retry, status transitions and +the retry counter belong to the task runner driver and are tested once for +every runner in ``tests/unit/tasks/test_task_runner_base.py``. """ import unittest @@ -16,6 +20,7 @@ EXPAND_PHASE_COMPLETED, EXPAND_PHASE_IN_PROGRESS, ) +from simplyblock_core.services.task_runner_base import TaskAbort, TaskRetry import simplyblock_core.services.tasks_runner_cluster_expand as runner @@ -65,27 +70,10 @@ def setUp(self): self.tc = patch.object(runner, "tasks_controller").start() - def test_canceled_marks_done(self): - task = _task(canceled=True) - res = runner.process_task(task) - self.assertFalse(res) - self.assertEqual(task.status, JobSchedule.STATUS_DONE) - self.assertEqual(task.function_result, "canceled") - self.integrate.assert_not_called() - - def test_max_retry_marks_done(self): - task = _task(retry=3, max_retry=3) - res = runner.process_task(task) - self.assertTrue(res) - self.assertEqual(task.status, JobSchedule.STATUS_DONE) - self.assertEqual(task.function_result, "max retry reached") - self.integrate.assert_not_called() - - def test_missing_new_node_id_marks_done(self): + def test_missing_new_node_id_aborts(self): task = _task(new_node_id=None) - res = runner.process_task(task) - self.assertTrue(res) - self.assertEqual(task.status, JobSchedule.STATUS_DONE) + with self.assertRaises(TaskAbort): + runner.process_task(task) self.integrate.assert_not_called() def test_happy_path_completes_and_queues_dev_mig(self): @@ -101,26 +89,34 @@ def _integrate(c, snode, **kw): self.integrate.side_effect = _integrate task = _task() - res = runner.process_task(task) + runner.process_task(task) - self.assertTrue(res) - self.assertEqual(task.status, JobSchedule.STATUS_DONE) self.integrate.assert_called_once() + self.assertIn("expansion complete", task.function_result) # Only the two ONLINE devices get a migration task. self.assertEqual(self.tc.add_new_device_mig_task.call_count, 2) - def test_failure_suspends_and_increments_retry(self): + def test_failure_propagates_for_the_driver_to_retry(self): self.db.get_cluster_by_id.return_value = _cluster() self.db.get_storage_node_by_id.return_value = _node_with_devices() self.integrate.side_effect = RuntimeError("boom") task = _task(retry=0) - res = runner.process_task(task) + # Suspending and counting the retry is the driver's half of the + # contract; the handler only has to not swallow the failure. + with self.assertRaises(RuntimeError): + runner.process_task(task) + + self.tc.add_new_device_mig_task.assert_not_called() + + def test_unexpected_phase_after_run_is_retried(self): + self.db.get_cluster_by_id.return_value = _cluster() + self.db.get_storage_node_by_id.return_value = _node_with_devices() + self.integrate.side_effect = lambda c, snode, **kw: None + + with self.assertRaises(TaskRetry): + runner.process_task(_task()) - self.assertFalse(res) - self.assertEqual(task.status, JobSchedule.STATUS_SUSPENDED) - self.assertEqual(task.retry, 1) - self.assertIn("boom", task.function_result) self.tc.add_new_device_mig_task.assert_not_called() def test_aborted_state_is_rearmed_before_resume(self): diff --git a/tests/integration/test_tasks_runner_restart_shortcircuit.py b/tests/integration/test_tasks_runner_restart_shortcircuit.py index 6e2bdba861..4134743dcf 100644 --- a/tests/integration/test_tasks_runner_restart_shortcircuit.py +++ b/tests/integration/test_tasks_runner_restart_shortcircuit.py @@ -39,6 +39,7 @@ from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.storage_node import StorageNode from simplyblock_core.models.nvme_device import NVMeDevice +from simplyblock_core.services.task_runner_base import TaskAbort, TaskRetry _RUNNER_PATH = os.path.join( @@ -126,9 +127,7 @@ def test_online_and_healthy_with_no_devices_skips_restart(self): node = _mk_node(status=StorageNode.STATUS_ONLINE, health_check=True, nvme_devices=[]) with patch.object(mod, "db", _mk_db(node)): - ret = mod.task_runner_node(task) - self.assertTrue(ret) - self.assertEqual(task.status, JobSchedule.STATUS_DONE) + mod.task_runner_node(task) self.assertIn("online", task.function_result.lower()) def test_online_and_healthy_with_unavailable_devices_still_skips(self): @@ -143,9 +142,8 @@ def test_online_and_healthy_with_unavailable_devices_still_skips(self): node = _mk_node(status=StorageNode.STATUS_ONLINE, health_check=True, nvme_devices=[bad_dev]) with patch.object(mod, "db", _mk_db(node)): - ret = mod.task_runner_node(task) - self.assertTrue(ret) - self.assertEqual(task.status, JobSchedule.STATUS_DONE) + mod.task_runner_node(task) + self.assertIn("online", task.function_result.lower()) def test_online_but_unhealthy_still_skips_restart(self): """Critical regression: an ONLINE node with health_check=False @@ -158,9 +156,7 @@ def test_online_but_unhealthy_still_skips_restart(self): node = _mk_node(status=StorageNode.STATUS_ONLINE, health_check=False, nvme_devices=[]) with patch.object(mod, "db", _mk_db(node)): - ret = mod.task_runner_node(task) - self.assertTrue(ret) - self.assertEqual(task.status, JobSchedule.STATUS_DONE) + mod.task_runner_node(task) self.assertIn("online", task.function_result.lower()) @@ -180,8 +176,10 @@ def test_offline_does_not_short_circuit(self): mock_health._check_node_ping.return_value = False mock_health._check_node_api.return_value = False mock_health._check_ping_from_node.return_value = False - _ = mod.task_runner_node(task) - self.assertNotEqual(task.status, JobSchedule.STATUS_DONE) + # Not a short-circuit: it falls through to the reachability + # checks and asks the driver for another attempt. + with self.assertRaises(TaskRetry): + mod.task_runner_node(task) class TestTerminalStatusesStillDoneImmediately(unittest.TestCase): @@ -201,9 +199,10 @@ def test_removed_short_circuits_without_restart(self): task = _mk_task() node = _mk_node(status=StorageNode.STATUS_REMOVED) with patch.object(mod, "db", _mk_db(node)): - ret = mod.task_runner_node(task) - self.assertTrue(ret) - self.assertEqual(task.status, JobSchedule.STATUS_DONE) + # TaskAbort is the terminal, non-retryable stop the driver turns + # into DONE — reached before any shutdown/restart call. + with self.assertRaises(TaskAbort): + mod.task_runner_node(task) if __name__ == "__main__": diff --git a/tests/unit/tasks/test_lvol_lock_and_retry.py b/tests/unit/tasks/test_lvol_lock_and_retry.py index 9476daa53b..8af7410dc0 100644 --- a/tests/unit/tasks/test_lvol_lock_and_retry.py +++ b/tests/unit/tasks/test_lvol_lock_and_retry.py @@ -31,6 +31,20 @@ from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.storage_node import StorageNode from simplyblock_core.exceptions import PreconditionError +from simplyblock_core.services import tasks_runner_sync_lvol_del as runner +from simplyblock_core.services.task_runner_base import ( + TaskAbort, + TaskDefer, + TaskRetry, +) + + +@pytest.fixture(autouse=True) +def _clear_failure_memo(): + """The runner remembers the last reported message per task in-process.""" + runner._last_alerted_failure.clear() + yield + runner._last_alerted_failure.clear() # --- 1. sync delete is explicitly unbounded ----------------------------- @@ -58,6 +72,8 @@ def test_sync_delete_matches_register_retry_semantics(): def _task(age_seconds): t = mock.MagicMock() + t.uuid = "task-1" + t.function_name = JobSchedule.FN_LVOL_SYNC_OP t.canceled = False t.date = int(time.time()) - age_seconds t.function_params = {"lvol_id": "lvol-1", "op": "register", @@ -68,20 +84,18 @@ def _task(age_seconds): def test_register_defers_while_the_create_is_still_writing_the_record(): task = _task(age_seconds=5) - with mock.patch.object(tasks_controller.db, "get_lvol_by_id", + with mock.patch.object(runner.db, "get_lvol_by_id", side_effect=KeyError("not yet")): - tasks_controller.run_lvol_sync_op_task(task) - assert task.status == JobSchedule.STATUS_SUSPENDED, ( - "inside the grace window a missing record means the create has not " - "committed yet, not that the volume was deleted") + with pytest.raises(TaskDefer): + runner.SPEC.handler(task) def test_register_completes_once_the_record_is_genuinely_gone(): task = _task(age_seconds=constants.LVOL_SYNC_OP_RECORD_GRACE_SEC + 60) - with mock.patch.object(tasks_controller.db, "get_lvol_by_id", + with mock.patch.object(runner.db, "get_lvol_by_id", side_effect=KeyError("deleted")): - tasks_controller.run_lvol_sync_op_task(task) - assert task.status == JobSchedule.STATUS_DONE + with pytest.raises(TaskAbort): + runner.SPEC.handler(task) # --- 3. force delete takes the locks best-effort, never bypasses them --- @@ -120,6 +134,8 @@ def test_force_delete_wait_is_bounded(): def _register_task(retry=0): t = mock.MagicMock() + t.uuid = "task-1" + t.function_name = JobSchedule.FN_LVOL_SYNC_OP t.canceled = False t.date = int(time.time()) t.retry = retry @@ -132,8 +148,12 @@ def _register_task(retry=0): return t -def _run_register_failing(task, err="boom"): - """Drive run_lvol_sync_op_task to its genuine-failure branch.""" +def _run_register_failing(task, err="boom", expect=TaskRetry): + """Drive the register handler to its genuine-failure branch. + + The driver owns the status transition and the retry increment, so the + handler is asserted on the signal it raises, not on task state. + """ lvol = mock.MagicMock() lvol.status = "online" lvol.nodes = ["node-1", "node-2"] @@ -143,9 +163,9 @@ def _run_register_failing(task, err="boom"): node.status = "online" with contextlib.ExitStack() as stack: stack.enter_context(mock.patch.object( - tasks_controller.db, "get_lvol_by_id", return_value=lvol)) + runner.db, "get_lvol_by_id", return_value=lvol)) stack.enter_context(mock.patch.object( - tasks_controller.db, "get_storage_node_by_id", return_value=node)) + runner.db, "get_storage_node_by_id", return_value=node)) stack.enter_context(mock.patch( "simplyblock_core.storage_node_ops.get_restart_phase", return_value=None)) @@ -154,18 +174,19 @@ def _run_register_failing(task, err="boom"): "repair_lvol_registration_on_non_leader", return_value=(False, err))) event = stack.enter_context(mock.patch.object( - tasks_controller.events_controller, "log_event_cluster")) - tasks_controller.run_lvol_sync_op_task(task) + runner.events_controller, "log_event_cluster")) + with pytest.raises(expect): + runner.SPEC.handler(task) return event def test_first_failures_do_not_alert(): """Transient failures are normal; only persistence is abnormal.""" for retry in range(constants.TASK_FAILURE_ALERT_THRESHOLD): + runner._last_alerted_failure.clear() task = _register_task(retry=retry) event = _run_register_failing(task) assert event.call_count == 0, f"alerted at retry={retry + 1}" - assert task.status == JobSchedule.STATUS_SUSPENDED def test_alert_once_past_the_threshold(): @@ -173,20 +194,21 @@ def test_alert_once_past_the_threshold(): event = _run_register_failing(task) assert event.call_count == 1 assert event.call_args.kwargs["event_level"] == "Critical" - assert task.retry == constants.TASK_FAILURE_ALERT_THRESHOLD + 1 + assert (f"{constants.TASK_FAILURE_ALERT_THRESHOLD + 1} times" + in event.call_args.kwargs["message"]) def test_same_error_does_not_alert_every_cycle(): """A node that stays broken must not write an event every 3 seconds.""" task = _register_task(retry=constants.TASK_FAILURE_ALERT_THRESHOLD + 5) - task.function_result = "registration failed: boom" + runner._last_alerted_failure[task.uuid] = "registration failed: boom" event = _run_register_failing(task, err="boom") assert event.call_count == 0 def test_changed_error_alerts_again(): task = _register_task(retry=constants.TASK_FAILURE_ALERT_THRESHOLD + 5) - task.function_result = "registration failed: something else" + runner._last_alerted_failure[task.uuid] = "registration failed: something else" event = _run_register_failing(task, err="boom") assert event.call_count == 1 @@ -194,10 +216,8 @@ def test_changed_error_alerts_again(): def test_enodev_is_never_counted_or_alerted(): """-19 means the object is already gone; not an actionable alert.""" task = _register_task(retry=constants.TASK_FAILURE_ALERT_THRESHOLD + 5) - event = _run_register_failing(task, err={"code": -19}) + event = _run_register_failing(task, err={"code": -19}, expect=TaskDefer) assert event.call_count == 0 - assert task.retry == constants.TASK_FAILURE_ALERT_THRESHOLD + 5, ( - "-19 must not advance the failure counter") # --- 5. a peer restart is waited out under the lock, but not forever ---- diff --git a/tests/unit/tasks/test_max_retry_semantics.py b/tests/unit/tasks/test_max_retry_semantics.py index 801f7711be..a7e5287981 100644 --- a/tests/unit/tasks/test_max_retry_semantics.py +++ b/tests/unit/tasks/test_max_retry_semantics.py @@ -10,9 +10,10 @@ The invariant under test is the basic task-scheduler semantic: a task whose ``retry`` has reached its ceiling must *terminate* (``STATUS_DONE``) instead of looping forever, and it must not perform its side-effecting work on that final -poll. Below the ceiling the task still advances. See -:mod:`test_retry_ceiling` for the backup runner's version of the same guard and -the static cross-runner check. +poll. Below the ceiling the task still advances. See :mod:`test_retry_ceiling` +for the cross-runner discovery check, and +:mod:`tests.unit.tasks.test_task_runner_base` for the ceiling as enforced by the +shared driver, which the migrated runners delegate it to. """ from unittest.mock import MagicMock @@ -23,6 +24,7 @@ from simplyblock_core.models.storage_node import StorageNode import simplyblock_core.services.tasks_runner_node_add as node_add_runner +from simplyblock_core.services.task_runner_base import TaskDefer, TaskRetry import simplyblock_core.services.tasks_runner_restart as restart_runner @@ -49,115 +51,148 @@ def _task(function_name, retry, max_retry, **params): # -------------------------------------------------------------------------- # tasks_runner_node_add.process_task +# +# The ceiling itself now lives in the shared driver (see +# tests/unit/tasks/test_task_runner_base.py); what this runner still decides is +# whether a failed add counts against it at all. # -------------------------------------------------------------------------- -def test_node_add_max_retry_finishes_without_adding(monkeypatch): - """retry >= max_retry must finish the task and never call add_node.""" +def test_node_add_success_completes(monkeypatch): sops = MagicMock() + sops.add_node.return_value = True monkeypatch.setattr(node_add_runner, "storage_node_ops", sops) - monkeypatch.setattr(node_add_runner, "db", MagicMock()) - - task = _task(JobSchedule.FN_NODE_ADD, retry=3, max_retry=3, node_id="node-1") - res = node_add_runner.process_task(task, MagicMock()) - assert res is True - assert task.status == JobSchedule.STATUS_DONE - assert "max retry" in task.function_result - sops.add_node.assert_not_called() + task = _task(JobSchedule.FN_NODE_ADD, retry=0, max_retry=3, node_id="node-1") + assert node_add_runner.process_task(task) is None + sops.add_node.assert_called_once_with(node_id="node-1") -def test_node_add_below_ceiling_dispatches(monkeypatch): - """Below the ceiling the task still runs its step (add_node).""" +def test_node_add_failure_counts_a_retry(monkeypatch): + """A failed add against a responsive node advances retry, so the driver's + ceiling can eventually bind.""" sops = MagicMock() - sops.add_node.return_value = True + sops.add_node.return_value = False monkeypatch.setattr(node_add_runner, "storage_node_ops", sops) - # get_cluster_by_id().status is a Mock != STATUS_IN_ACTIVATION, so the - # in-activation gate is not taken and the task proceeds to add_node. - monkeypatch.setattr(node_add_runner, "db", MagicMock()) + monkeypatch.setattr(node_add_runner, "_wait_node_reachable", lambda task: False) - task = _task(JobSchedule.FN_NODE_ADD, retry=0, max_retry=3, node_id="node-1") - res = node_add_runner.process_task(task, MagicMock()) - - assert res is True - sops.add_node.assert_called_once_with(node_id="node-1") - assert task.status == JobSchedule.STATUS_DONE + task = _task(JobSchedule.FN_NODE_ADD, retry=1, max_retry=3, node_id="node-1") + with pytest.raises(TaskRetry): + node_add_runner.process_task(task) -def test_node_add_failure_below_ceiling_suspends_and_counts_retry(monkeypatch): - """A failed add below the ceiling suspends and advances retry (so the - ceiling can eventually bind), rather than finishing the task.""" +def test_node_add_failure_during_a_reboot_does_not_count(monkeypatch): + """The CPU-topology reboot is expected, not a failure: waiting it out must + not burn one of the task's retries.""" sops = MagicMock() sops.add_node.return_value = False monkeypatch.setattr(node_add_runner, "storage_node_ops", sops) - monkeypatch.setattr(node_add_runner, "db", MagicMock()) + monkeypatch.setattr(node_add_runner, "_wait_node_reachable", lambda task: True) task = _task(JobSchedule.FN_NODE_ADD, retry=1, max_retry=3, node_id="node-1") - res = node_add_runner.process_task(task, MagicMock()) + with pytest.raises(TaskDefer): + node_add_runner.process_task(task) - assert res is True # processed; the loop keeps polling - assert task.status == JobSchedule.STATUS_SUSPENDED - assert task.retry == 2 + +def test_node_add_exception_is_handled_like_a_failed_add(monkeypatch): + sops = MagicMock() + sops.add_node.side_effect = RuntimeError("boom") + monkeypatch.setattr(node_add_runner, "storage_node_ops", sops) + monkeypatch.setattr(node_add_runner, "_wait_node_reachable", lambda task: False) + + task = _task(JobSchedule.FN_NODE_ADD, retry=0, max_retry=3, node_id="node-1") + with pytest.raises(TaskRetry, match="boom"): + node_add_runner.process_task(task) # -------------------------------------------------------------------------- -# tasks_runner_restart.task_runner_node +# tasks_runner_restart give-up side effects +# +# The ceiling itself is the driver's (test_task_runner_base); what the runner +# still owns is what to do about the target when the ceiling terminates a task +# — a path its handler never sees, so it hangs off the driver's on_finish. # -------------------------------------------------------------------------- -def test_restart_node_max_retry_finishes_and_marks_offline(monkeypatch): - """retry >= max_retry finishes the task, flips the node OFFLINE via the - restart_cleanup path, and re-queues a fresh auto-restart.""" - node = MagicMock() - node.status = StorageNode.STATUS_ONLINE +def _restart_task(function_name, retry, max_retry, canceled=False): + task = _task(function_name, retry, max_retry, node_id="node-1") + task.canceled = canceled + return task + + +def test_restart_node_give_up_marks_offline_and_requeues(monkeypatch): + """Exhausting the retries parks the node OFFLINE and queues a fresh + auto-restart, so it is not stranded until an operator intervenes.""" fake_db = MagicMock() - fake_db.get_storage_node_by_id.return_value = node - # Mirror DBController.atomic_update's contract: apply the mutator to the - # (fresh) object and return it. The restart runner's task writes go - # through this instead of write_to_db (stale-copy lost-update fix). - fake_db.atomic_update.side_effect = lambda obj, fn: (fn(obj), obj)[1] monkeypatch.setattr(restart_runner, "db", fake_db) - sops = MagicMock() tasks_ctrl = MagicMock() monkeypatch.setattr(restart_runner, "storage_node_ops", sops) monkeypatch.setattr(restart_runner, "tasks_controller", tasks_ctrl) - task = _task(JobSchedule.FN_NODE_RESTART, retry=5, max_retry=5, node_id="node-1") - res = restart_runner.task_runner_node(task) + restart_runner.SPEC.on_finish( + _restart_task(JobSchedule.FN_NODE_RESTART, retry=5, max_retry=5)) - assert res is True - assert task.status == JobSchedule.STATUS_DONE - assert "max retry" in task.function_result sops.set_node_status.assert_called_once() assert sops.set_node_status.call_args.args[1] == StorageNode.STATUS_OFFLINE tasks_ctrl.add_node_to_auto_restart.assert_called_once() -# -------------------------------------------------------------------------- -# tasks_runner_restart.task_runner_device -# -------------------------------------------------------------------------- +def test_restart_node_success_leaves_the_node_alone(monkeypatch): + """A task that finished below the ceiling succeeded — the node is online + and must not be flipped OFFLINE by the cleanup path.""" + monkeypatch.setattr(restart_runner, "db", MagicMock()) + sops = MagicMock() + tasks_ctrl = MagicMock() + monkeypatch.setattr(restart_runner, "storage_node_ops", sops) + monkeypatch.setattr(restart_runner, "tasks_controller", tasks_ctrl) + + restart_runner.SPEC.on_finish( + _restart_task(JobSchedule.FN_NODE_RESTART, retry=2, max_retry=5)) + + sops.set_node_status.assert_not_called() + tasks_ctrl.add_node_to_auto_restart.assert_not_called() + + +def test_restart_node_unbounded_never_gives_up(monkeypatch): + monkeypatch.setattr(restart_runner, "db", MagicMock()) + sops = MagicMock() + monkeypatch.setattr(restart_runner, "storage_node_ops", sops) + monkeypatch.setattr(restart_runner, "tasks_controller", MagicMock()) + + restart_runner.SPEC.on_finish( + _restart_task(JobSchedule.FN_NODE_RESTART, retry=100, max_retry=-1)) -def test_restart_device_max_retry_finishes_and_exhausts(monkeypatch): - """retry >= TASK_EXEC_RETRY_COUNT finishes the task and marks the device - unavailable / retries-exhausted instead of restarting it again.""" + sops.set_node_status.assert_not_called() + + +def test_restart_device_give_up_marks_unavailable_and_exhausted(monkeypatch): device = MagicMock() device.get_id.return_value = "dev-1" monkeypatch.setattr(restart_runner, "_get_device", lambda task: device) monkeypatch.setattr(restart_runner, "db", MagicMock()) - dc = MagicMock() monkeypatch.setattr(restart_runner, "device_controller", dc) - task = _task( + restart_runner.SPEC.on_finish(_restart_task( JobSchedule.FN_DEV_RESTART, retry=constants.TASK_EXEC_RETRY_COUNT, - max_retry=constants.TASK_EXEC_RETRY_COUNT, - node_id="node-1", - ) - res = restart_runner.task_runner_device(task) - - assert res is True - assert task.status == JobSchedule.STATUS_DONE - assert "max retry" in task.function_result + max_retry=constants.TASK_EXEC_RETRY_COUNT)) + dc.device_set_unavailable.assert_called_once_with("dev-1") dc.device_set_retries_exhausted.assert_called_once_with("dev-1", True) dc.restart_device.assert_not_called() + + +def test_restart_device_cancel_exhausts_retries(monkeypatch): + """A canceled device task must not be picked up and retried forever.""" + device = MagicMock() + device.get_id.return_value = "dev-1" + monkeypatch.setattr(restart_runner, "_get_device", lambda task: device) + monkeypatch.setattr(restart_runner, "db", MagicMock()) + dc = MagicMock() + monkeypatch.setattr(restart_runner, "device_controller", dc) + + restart_runner.SPEC.on_finish(_restart_task( + JobSchedule.FN_DEV_RESTART, retry=0, max_retry=5, canceled=True)) + + dc.device_set_retries_exhausted.assert_called_once_with("dev-1", True) + dc.device_set_unavailable.assert_not_called() diff --git a/tests/unit/tasks/test_retry_ceiling.py b/tests/unit/tasks/test_retry_ceiling.py index c3769966dc..502d7cd510 100644 --- a/tests/unit/tasks/test_retry_ceiling.py +++ b/tests/unit/tasks/test_retry_ceiling.py @@ -32,187 +32,12 @@ import pytest -from simplyblock_core.models.backup import Backup from simplyblock_core.models.cluster import Cluster from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.lvol_migration_group import LVolMigrationGroup from simplyblock_core.models.storage_node import StorageNode -# -------------------------------------------------------------------------- -# Backup runner: terminates instead of looping forever. -# -# The backup runner's loop is inline under ``if __name__ == '__main__'`` (it has -# no ``def main()``), so it is driven at the ``process_task`` level here. The -# other runners below are driven through their real ``main()``. -# -------------------------------------------------------------------------- - -@pytest.fixture -def runner(monkeypatch): - """Import the backup runner with its FDB writes and events neutralised. - - The module's ``while True`` loop is guarded by ``if __name__ == - '__main__'``, so importing it only defines functions and a (kv_store=None) - ``DBController`` singleton — safe under the unit tier's stubbed ``fdb``. - """ - import simplyblock_core.services.tasks_runner_backup as runner - monkeypatch.setattr(runner, "backup_events", MagicMock()) - # Model writes would otherwise dereference the None kv_store and exit(1). - monkeypatch.setattr(Backup, "write_to_db", MagicMock()) - monkeypatch.setattr(JobSchedule, "write_to_db", MagicMock()) - return runner - - -def _backup_task(retry, max_retry): - task = JobSchedule() - task.uuid = "task-1" - task.function_name = JobSchedule.FN_BACKUP - task.function_params = {"backup_id": "bk-1"} - task.retry = retry - task.max_retry = max_retry - task.canceled = False - # Recent enough that the 4h time-based timeout does not fire first. - task.date = int(__import__("time").time()) - return task - - -def _cluster(): - cl = MagicMock() - cl.backup_timeout_seconds = 14400 - return cl - - -def test_backup_fails_when_max_retry_reached(runner, monkeypatch): - """retry >= max_retry must fail the backup and finish the task, not poll.""" - backup = Backup() - backup.uuid = "bk-1" - backup.status = Backup.STATUS_IN_PROGRESS - - fake_db = MagicMock() - fake_db.get_backup_by_id.return_value = backup - monkeypatch.setattr(runner, "db", fake_db) - # If the ceiling is missing this would run and (in the incident) re-issue - # the crash-inducing RPC; assert it is never reached. - run_backup = MagicMock() - monkeypatch.setattr(runner, "_run_backup", run_backup) - - task = _backup_task(retry=10, max_retry=10) - runner.process_task(task, _cluster()) - - assert backup.status == Backup.STATUS_FAILED - assert task.status == JobSchedule.STATUS_DONE - assert "max retry" in task.function_result - run_backup.assert_not_called() - - -def test_backup_runs_below_max_retry(runner, monkeypatch): - """Below the ceiling the task still advances (dispatches its step).""" - monkeypatch.setattr(runner, "db", MagicMock()) - run_backup = MagicMock() - monkeypatch.setattr(runner, "_run_backup", run_backup) - - task = _backup_task(retry=9, max_retry=10) - runner.process_task(task, _cluster()) - - run_backup.assert_called_once_with(task) - - -def test_no_process_poll_counts_as_a_retry(runner, monkeypatch): - """The 'No process' re-issue branch must advance task.retry, otherwise the - ceiling can never bind to that path and the backup re-issues forever.""" - backup = Backup() - backup.uuid = "bk-1" - backup.status = Backup.STATUS_IN_PROGRESS - backup.snapshot_id = "snap-1" - - snode = MagicMock() - snode.status = StorageNode.STATUS_ONLINE - rpc = MagicMock() - rpc.bdev_lvol_transfer_stat.return_value = {"transfer_state": "No process"} - snode.rpc_client.return_value = rpc - - snapshot = MagicMock() - snapshot.snap_bdev = "lvs/snap0" - - fake_db = MagicMock() - fake_db.get_backup_by_id.return_value = backup - fake_db.get_storage_node_by_id.return_value = snode - fake_db.get_snapshot_by_id.return_value = snapshot - monkeypatch.setattr(runner, "db", fake_db) - - task = _backup_task(retry=3, max_retry=10) - runner._run_backup(task) - - assert task.retry == 4, "No-process re-issue must count toward the ceiling" - assert backup.status == Backup.STATUS_PENDING - rpc.bdev_lvol_s3_backup.assert_not_called() # this poll only resets state - - -# -------------------------------------------------------------------------- -# FDB backup runner: terminates instead of looping forever. -# -# Same shape as the S3 backup runner above — its loop is also inline under -# ``if __name__ == '__main__'`` (no ``def main()``), so it is driven at the -# ``process_fdb_backup_task`` level here. -# -------------------------------------------------------------------------- - -@pytest.fixture -def fdb_backup_runner(monkeypatch): - import simplyblock_core.services.tasks_runner_fdb_backup as runner - monkeypatch.setattr(runner, "fdb_backup_events", MagicMock()) - monkeypatch.setattr(JobSchedule, "write_to_db", MagicMock()) - return runner - - -def _fdb_backup_task(retry, max_retry): - task = JobSchedule() - task.uuid = "task-1" - task.cluster_id = "cl-1" - task.function_name = JobSchedule.FN_FDB_BACKUP - task.function_params = {} - task.retry = retry - task.max_retry = max_retry - task.canceled = False - task.status = JobSchedule.STATUS_NEW - return task - - -def test_fdb_backup_fails_when_max_retry_reached(fdb_backup_runner, monkeypatch): - """retry >= max_retry must finish the task instead of re-issuing the backup.""" - fake_db = MagicMock() - task = _fdb_backup_task(retry=10, max_retry=10) - fake_db.get_task_by_id.return_value = task - monkeypatch.setattr(fdb_backup_runner, "db", fake_db) - create_backup = MagicMock() - monkeypatch.setattr( - fdb_backup_runner.fdb_backup_controller, "create_backup", create_backup) - - fdb_backup_runner.process_fdb_backup_task(task) - - assert task.status == JobSchedule.STATUS_DONE - assert "max retry" in task.function_result - create_backup.assert_not_called() - fdb_backup_runner.fdb_backup_events.fdb_backup_failed.assert_called_once_with( - task.cluster_id, task.uuid) - - -def test_fdb_backup_runs_below_max_retry(fdb_backup_runner, monkeypatch): - """Below the ceiling the task still advances (dispatches its step).""" - fake_db = MagicMock() - task = _fdb_backup_task(retry=9, max_retry=10) - fake_db.get_task_by_id.return_value = task - monkeypatch.setattr(fdb_backup_runner, "db", fake_db) - create_backup = MagicMock(return_value=False) - monkeypatch.setattr( - fdb_backup_runner.fdb_backup_controller, "create_backup", create_backup) - - fdb_backup_runner.process_fdb_backup_task(task) - - create_backup.assert_called_once_with(task.cluster_id) - assert task.retry == 10 - assert task.status == JobSchedule.STATUS_SUSPENDED - - # -------------------------------------------------------------------------- # Drive-main harness: exercise each runner's real retry loop end-to-end. # -------------------------------------------------------------------------- @@ -352,6 +177,9 @@ def _wire_base(runner, monkeypatch, task): monkeypatch.setattr(runner, "db", db) monkeypatch.setattr(JobSchedule, "write_to_db", MagicMock()) _patch_clock(monkeypatch, runner, task) + # Every runner claims the lease before running a task; grant it so the loop + # reaches the work under test instead of skipping the task as foreign-owned. + monkeypatch.setattr(runner.tasks_controller, "claim_task", lambda *a, **k: True) return db, cluster, node @@ -364,77 +192,6 @@ def _wire_base(runner, monkeypatch, task): # tree, so a new retry-driven runner shows up as a failing case until a spec is # added here. -def _spec_cluster_expand(runner, monkeypatch): - task = _make_task(JobSchedule.FN_CLUSTER_EXPAND, new_node_id="new-1") - _wire_base(runner, monkeypatch, task) - monkeypatch.setattr(runner.tasks_controller, "claim_task", - lambda *a, **k: True) - # The actual expansion work fails every cycle. - monkeypatch.setattr( - runner, "integrate_new_node_into_cluster", - MagicMock(side_effect=RuntimeError("expand boom"))) - return task - - -def _spec_node_add(runner, monkeypatch): - task = _make_task(JobSchedule.FN_NODE_ADD) - _wire_base(runner, monkeypatch, task) - monkeypatch.setattr(runner, "ThreadPoolExecutor", _InlineExecutor) - monkeypatch.setattr(runner, "_inflight", set()) - monkeypatch.setattr(runner.tasks_controller, "claim_task", - lambda *a, **k: True) - # add_node fails (returns falsy) every cycle. - monkeypatch.setattr(runner.storage_node_ops, "add_node", - MagicMock(return_value=False)) - return task - - -def _spec_replication_final(runner, monkeypatch): - task = _make_task( - JobSchedule.FN_REPLICATION_FINAL, - lvol_id="lv-1", tgt_node_id="tgt-1", src_node_id="src-1") - db, _cluster, node = _wire_base(runner, monkeypatch, task) - # Target node never comes online -> cutover cannot proceed, retry each poll. - node.status = StorageNode.STATUS_OFFLINE - db.get_lvol_by_id.return_value = MagicMock() - return task - - -def _spec_jc_comp(runner, monkeypatch): - task = _make_task(JobSchedule.FN_JC_COMP_RESUME) - _wire_base(runner, monkeypatch, task) - # A task is always active on the same node -> resume is deferred, retry each - # poll (this is the branch that increments task.retry). - monkeypatch.setattr(runner.tasks_controller, "get_active_node_tasks", - lambda *a, **k: [MagicMock()]) - return task - - -def _spec_restart(runner, monkeypatch): - task = _make_task(JobSchedule.FN_NODE_RESTART) - _db, _cluster, node = _wire_base(runner, monkeypatch, task) - # Node is offline and stays unreachable -> restart keeps failing, retry - # each poll. - node.status = StorageNode.STATUS_OFFLINE - monkeypatch.setattr(runner, "_restart_pool", _InlineExecutor()) - monkeypatch.setattr(runner, "_restart_next_attempt", {}) - monkeypatch.setattr(runner, "_restart_inflight", {}) - monkeypatch.setattr(runner, "_node_inflight", {}) - monkeypatch.setattr(runner.tasks_controller, "claim_task", - lambda *a, **k: True) - monkeypatch.setattr(runner.tasks_controller, "is_auto_restart_paused", - lambda *a, **k: False) - monkeypatch.setattr(runner.tasks_controller, "add_node_to_auto_restart", - MagicMock()) - monkeypatch.setattr(runner.storage_node_ops, "set_node_status", MagicMock()) - # Node never reachable -> the reachability check fails and retry advances. - monkeypatch.setattr(runner.health_controller, "_check_node_ping", - lambda *a, **k: False) - monkeypatch.setattr(runner.health_controller, "_check_node_api", - lambda *a, **k: False) - return task - - def _spec_batch_migration(runner, monkeypatch): task = _make_task(JobSchedule.FN_LVOL_BATCH_MIG, group_id="grp-1") db, _cluster, _ = _wire_base(runner, monkeypatch, task) @@ -487,11 +244,6 @@ def _get_node(node_id): # name -> spec for the runners driven through their real main() loop. _MAIN_DRIVEN_SPECS = { - "tasks_runner_cluster_expand.py": _spec_cluster_expand, - "tasks_runner_node_add.py": _spec_node_add, - "tasks_runner_replication_final.py": _spec_replication_final, - "tasks_runner_jc_comp.py": _spec_jc_comp, - "tasks_runner_restart.py": _spec_restart, "tasks_runner_batch_migration.py": _spec_batch_migration, } @@ -499,10 +251,29 @@ def _get_node(node_id): # generic drive-main harness. The backup runner's loop is inline under # ``if __name__ == '__main__'`` (no ``def main()``), so it is exercised at the # ``process_task`` level by ``test_backup_*`` above. -_COVERED_ELSEWHERE = { - "tasks_runner_backup.py": "driven via process_task in test_backup_* above", - "tasks_runner_fdb_backup.py": - "driven via process_fdb_backup_task in test_fdb_backup_* above", +_COVERED_ELSEWHERE: dict = {} + +# Runners migrated onto the shared driver (``task_runner_base``). They no longer +# own a loop or a retry counter — the driver enforces the ceiling for all of +# them at once, covered by tests/unit/tasks/test_task_runner_base.py. They are +# therefore not discovered by _retry_driven_runner_files(); listing them here +# keeps that disappearance deliberate rather than silent, and +# test_migrated_runners_delegate_retry below pins that they really did hand the +# retry counter over. +_DRIVER_MIGRATED = { + "tasks_runner_fdb_backup.py", + "tasks_runner_jc_comp.py", + "tasks_runner_replication_final.py", + "tasks_runner_sync_lvol_del.py", + "tasks_runner_backup.py", + "tasks_runner_cluster_expand.py", + "tasks_runner_node_add.py", + "tasks_runner_restart.py", + "tasks_runner_migration.py", + "tasks_runner_new_dev_migration.py", + "tasks_runner_failed_migration.py", + "tasks_runner_node_removal.py", + "tasks_runner_port_allow.py", } # Runners that increment task.retry but are intentionally UNBOUNDED: the @@ -510,11 +281,7 @@ def _get_node(node_id): # recovery (see _migration_retry_allowed) rather than a fixed count. Value is # the reason, surfaced in the skip message. INTENTIONALLY_UNBOUNDED = { - "tasks_runner_migration.py": "created with max_retry=-1; retry gated on resource recovery", - "tasks_runner_failed_migration.py": "created with max_retry=-1; retry gated on resource recovery", - "tasks_runner_new_dev_migration.py": "created with max_retry=-1; retry gated on resource recovery", "tasks_runner_lvol_migration.py": "created with max_retry=-1; retry gated on resource recovery", - "tasks_runner_node_removal.py": "created with max_retry=-1; multi-hour removal gated on failure-migration completion", } _INCREMENTS_RETRY = re.compile(r"\.retry\s*\+=\s*1") @@ -576,6 +343,21 @@ def test_registries_are_not_stale(): names = {p.name for p in _runner_files()} listed = (set(_MAIN_DRIVEN_SPECS) | set(_COVERED_ELSEWHERE) - | set(INTENTIONALLY_UNBOUNDED)) + | set(INTENTIONALLY_UNBOUNDED) + | _DRIVER_MIGRATED) missing = listed - names assert not missing, f"listed runners no longer exist: {missing}" + + +@pytest.mark.parametrize("name", sorted(_DRIVER_MIGRATED)) +def test_migrated_runners_delegate_retry(name): + """A runner listed as migrated must not have kept a retry counter of its + own: the driver owns task.retry, and a runner that still increments it + would be applying two ceilings at once.""" + source = (Path(_runner_files()[0]).parent / name).read_text(encoding='utf-8') + assert not _INCREMENTS_RETRY.search(source), ( + f"{name} is listed as migrated to task_runner_base but still increments " + "task.retry itself") + assert "serve(SPEC)" in source, ( + f"{name} is listed as migrated to task_runner_base but does not run the " + "shared driver") diff --git a/tests/unit/tasks/test_runner_specs.py b/tests/unit/tasks/test_runner_specs.py new file mode 100644 index 0000000000..5ea6529341 --- /dev/null +++ b/tests/unit/tasks/test_runner_specs.py @@ -0,0 +1,557 @@ +# coding=utf-8 +"""Per-runner tests for the runners migrated onto the shared driver. + +Each migrated runner is reduced to a :class:`RunnerSpec`: a void handler that +signals through ``TaskDefer`` / ``TaskRetry`` / ``TaskAbort``, plus an +eligibility predicate. These tests pin that translation — the loop, lease and +retry mechanics themselves are covered once in ``test_task_runner_base.py``. +""" +import time +from unittest.mock import MagicMock + +import pytest + +from simplyblock_core.models.backup import Backup +from simplyblock_core.models.cluster import Cluster +from simplyblock_core.models.storage_node import StorageNode +from simplyblock_core.models.job_schedule import JobSchedule +from simplyblock_core.release_upgrades import jc_compression_upgrade +import simplyblock_core.services.task_runner_base as trb + + +def _task(**params): + task = JobSchedule() + task.uuid = "task-1" + task.cluster_id = "cl-1" + task.node_id = "node-1" + task.function_params = params + return task + + +def _cluster(status=Cluster.STATUS_ACTIVE): + cluster = MagicMock() + cluster.status = status + return cluster + + +# -- fdb_backup ------------------------------------------------------------- + +@pytest.fixture +def fdb_backup(monkeypatch): + import simplyblock_core.services.tasks_runner_fdb_backup as runner + monkeypatch.setattr(runner, "fdb_backup_controller", MagicMock()) + monkeypatch.setattr(runner, "fdb_backup_events", MagicMock()) + return runner + + +def test_fdb_backup_success_reports_result(fdb_backup): + fdb_backup.fdb_backup_controller.create_backup.return_value = True + task = _task() + + assert fdb_backup.SPEC.handler(task) is None + fdb_backup.fdb_backup_controller.create_backup.assert_called_once_with("cl-1") + assert task.function_result == "Backup created" + + +def test_fdb_backup_failure_is_retryable(fdb_backup): + fdb_backup.fdb_backup_controller.create_backup.return_value = False + + with pytest.raises(trb.TaskRetry): + fdb_backup.SPEC.handler(_task()) + + +def test_fdb_backup_reports_a_task_that_ran_out_of_retries(fdb_backup): + """The driver finishes the task on the ceiling without calling the handler, + so nothing but on_finish can see that the backup gave up.""" + task = _task() + task.retry = task.max_retry = 10 + + fdb_backup.SPEC.on_finish(task) + + fdb_backup.fdb_backup_events.fdb_backup_failed.assert_called_once_with( + "cl-1", "task-1") + + +def test_fdb_backup_does_not_report_a_task_that_succeeded(fdb_backup): + task = _task() + task.retry, task.max_retry = 0, 10 + + fdb_backup.SPEC.on_finish(task) + + fdb_backup.fdb_backup_events.fdb_backup_failed.assert_not_called() + + +def test_fdb_backup_skips_clusters_in_activation(fdb_backup): + task = _task() + assert fdb_backup.SPEC.is_eligible(task, _cluster()) is True + assert fdb_backup.SPEC.is_eligible(task, _cluster(Cluster.STATUS_IN_ACTIVATION)) is False + + +# -- jc_comp ---------------------------------------------------------------- + +def _node(status=StorageNode.STATUS_ONLINE, jm_vuid=7): + node = MagicMock() + node.status = status + node.jm_vuid = jm_vuid + node.cluster_id = "cl-1" + return node + + +@pytest.fixture +def jc_comp(monkeypatch): + import simplyblock_core.services.tasks_runner_jc_comp as runner + db = MagicMock() + monkeypatch.setattr(runner, "db", db) + monkeypatch.setattr(runner, "tasks_controller", MagicMock()) + runner.tasks_controller.get_active_node_tasks.return_value = [] + db.get_storage_nodes_by_cluster_id.return_value = [_node()] + # An unstubbed MagicMock attribute auto-creates, so `release_upgrade_state` + # would answer .get() with a truthy mock and hold every resume. + db.get_cluster_by_id.return_value.release_upgrade_state = {} + return runner + + +def test_jc_comp_resumes_compression(jc_comp): + node = _node() + jc_comp.db.get_storage_node_by_id.return_value = node + node.rpc_client.return_value.jc_suspend_compression.return_value = (True, None) + task = _task() + + assert jc_comp.SPEC.handler(task) is None + node.rpc_client.return_value.jc_suspend_compression.assert_called_once_with( + jm_vuid=7, suspend=False) + assert task.function_result == "JC 7 compression resumed on node" + + +def test_jc_comp_prefers_the_task_jm_vuid(jc_comp): + node = _node() + jc_comp.db.get_storage_node_by_id.return_value = node + node.rpc_client.return_value.jc_suspend_compression.return_value = (True, None) + + jc_comp.SPEC.handler(_task(jm_vuid=42)) + node.rpc_client.return_value.jc_suspend_compression.assert_called_once_with( + jm_vuid=42, suspend=False) + + +def test_jc_comp_aborts_on_missing_node(jc_comp): + jc_comp.db.get_storage_node_by_id.side_effect = KeyError("node-1") + with pytest.raises(trb.TaskAbort): + jc_comp.SPEC.handler(_task()) + + +def test_jc_comp_defers_while_the_release_upgrade_holds_resumes(jc_comp): + node = _node() + jc_comp.db.get_storage_node_by_id.return_value = node + jc_comp.db.get_cluster_by_id.return_value.release_upgrade_state = { + jc_compression_upgrade.STATE_KEY: True, + } + + with pytest.raises(trb.TaskDefer): + jc_comp.SPEC.handler(_task()) + node.rpc_client.assert_not_called() + + +def test_jc_comp_defers_while_node_is_offline(jc_comp): + jc_comp.db.get_storage_node_by_id.return_value = _node(StorageNode.STATUS_OFFLINE) + with pytest.raises(trb.TaskDefer): + jc_comp.SPEC.handler(_task()) + + +def test_jc_comp_defers_while_another_task_runs_on_the_node(jc_comp): + jc_comp.db.get_storage_node_by_id.return_value = _node() + jc_comp.tasks_controller.get_active_node_tasks.return_value = [_task()] + with pytest.raises(trb.TaskDefer): + jc_comp.SPEC.handler(_task()) + + +def test_jc_comp_defers_unless_every_cluster_node_is_online(jc_comp): + node = _node() + jc_comp.db.get_storage_node_by_id.return_value = node + jc_comp.db.get_storage_nodes_by_cluster_id.return_value = [ + _node(StorageNode.STATUS_OFFLINE), node, + ] + + with pytest.raises(trb.TaskDefer): + jc_comp.SPEC.handler(_task()) + node.rpc_client.assert_not_called() + + +def test_jc_comp_aborts_when_compression_is_not_needed(jc_comp): + node = _node() + jc_comp.db.get_storage_node_by_id.return_value = node + node.rpc_client.return_value.jc_suspend_compression.return_value = (False, "not needed") + + with pytest.raises(trb.TaskAbort): + jc_comp.SPEC.handler(_task()) + + +def test_jc_comp_retries_when_resume_fails(jc_comp): + node = _node() + jc_comp.db.get_storage_node_by_id.return_value = node + node.rpc_client.return_value.jc_suspend_compression.return_value = (False, None) + + with pytest.raises(trb.TaskRetry): + jc_comp.SPEC.handler(_task()) + + +# -- backup ----------------------------------------------------------------- + +@pytest.fixture +def backup_runner(monkeypatch): + """The backup runner with its DB, events and model writes neutralised. + + Its ceiling regression (an S3 backup whose bdev_lvol_s3_backup crashed SPDK + was re-issued every poll forever, and the backup never transitioned to + failed) now rests on two things this exercises: the "No process" poll + raising TaskRetry so the ceiling can bind, and finalize_resource failing the + backup on whichever terminal path the task ends up taking. + """ + import simplyblock_core.services.tasks_runner_backup as runner + monkeypatch.setattr(runner, "db", MagicMock()) + monkeypatch.setattr(runner, "backup_events", MagicMock()) + monkeypatch.setattr(Backup, "write_to_db", MagicMock()) + runner.db.get_cluster_by_id.return_value.backup_timeout_seconds = 14400 + return runner + + +def _backup_task(function_name=JobSchedule.FN_BACKUP, **params): + task = _task(**params) + task.function_name = function_name + task.date = int(time.time()) + return task + + +def _backup(status=Backup.STATUS_IN_PROGRESS): + backup = Backup() + backup.uuid = "bk-1" + backup.status = status + backup.snapshot_id = "snap-1" + return backup + + +def test_backup_no_process_poll_is_a_retry(backup_runner): + """The re-issue branch must consume a retry — without it the ceiling can + never bind to this path and the backup re-issues forever.""" + backup = _backup() + backup_runner.db.get_backup_by_id.return_value = backup + + snode = _node() + rpc = snode.rpc_client.return_value + rpc.bdev_lvol_transfer_stat.return_value = {"transfer_state": "No process"} + backup_runner.db.get_storage_node_by_id.return_value = snode + + with pytest.raises(trb.TaskRetry): + backup_runner.SPEC.handler(_backup_task(backup_id="bk-1")) + + assert backup.status == Backup.STATUS_PENDING + rpc.bdev_lvol_s3_backup.assert_not_called() # this poll only resets state + + +def test_backup_completes_on_done(backup_runner): + backup = _backup() + backup_runner.db.get_backup_by_id.return_value = backup + snode = _node() + snode.rpc_client.return_value.bdev_lvol_transfer_stat.return_value = { + "transfer_state": "Done"} + backup_runner.db.get_storage_node_by_id.return_value = snode + + task = _backup_task(backup_id="bk-1") + assert backup_runner.SPEC.handler(task) is None + assert backup.status == Backup.STATUS_COMPLETED + assert task.function_result == "Backup completed" + + +def test_backup_times_out(backup_runner): + backup_runner.db.get_cluster_by_id.return_value = _cluster() + backup_runner.db.get_cluster_by_id.return_value.backup_timeout_seconds = 10 + + task = _backup_task(backup_id="bk-1") + task.date = int(time.time()) - 3600 + + with pytest.raises(trb.TaskAbort, match="timeout"): + backup_runner.SPEC.handler(task) + + +def test_unfinished_backup_is_failed_when_the_task_ends(backup_runner): + """Whichever way the task ended — ceiling, timeout, abort, cancellation — + the backup must not be left sitting in a pending state forever.""" + backup = _backup() + backup_runner.db.get_backup_by_id.return_value = backup + + task = _backup_task(backup_id="bk-1") + task.function_result = "max retry reached (10/10)" + backup_runner.SPEC.on_finish(task) + + assert backup.status == Backup.STATUS_FAILED + assert backup.error_message == "max retry reached (10/10)" + backup_runner.backup_events.backup_failed.assert_called_once() + + +def test_completed_backup_survives_the_task_ending(backup_runner): + backup = _backup(Backup.STATUS_COMPLETED) + backup_runner.db.get_backup_by_id.return_value = backup + + backup_runner.SPEC.on_finish(_backup_task(backup_id="bk-1")) + + assert backup.status == Backup.STATUS_COMPLETED + backup_runner.backup_events.backup_failed.assert_not_called() + + +def test_unfinished_merge_leaves_the_old_backup_intact(backup_runner): + old = _backup(Backup.STATUS_MERGING) + backup_runner.db.get_backup_by_id.return_value = old + + backup_runner.SPEC.on_finish( + _backup_task(JobSchedule.FN_BACKUP_MERGE, old_backup_id="bk-0")) + + assert old.status == Backup.STATUS_COMPLETED + + +# -- cluster_expand --------------------------------------------------------- + +@pytest.fixture +def cluster_expand(monkeypatch): + import simplyblock_core.services.tasks_runner_cluster_expand as runner + monkeypatch.setattr(runner, "db", MagicMock()) + monkeypatch.setattr(runner, "tasks_controller", MagicMock()) + monkeypatch.setattr(runner, "integrate_new_node_into_cluster", MagicMock()) + return runner + + +def _expanded_cluster(runner, phase): + cluster = MagicMock() + cluster.expand_state = {"phase": phase} + runner.db.get_cluster_by_id.return_value = cluster + return cluster + + +def test_cluster_expand_completes_and_queues_device_migration(cluster_expand): + from simplyblock_core.controllers.cluster_expansion.planner import EXPAND_PHASE_COMPLETED + from simplyblock_core.models.nvme_device import NVMeDevice + + _expanded_cluster(cluster_expand, EXPAND_PHASE_COMPLETED) + device = MagicMock() + device.status = NVMeDevice.STATUS_ONLINE + device.get_id.return_value = "dev-1" + cluster_expand.db.get_storage_node_by_id.return_value.nvme_devices = [device] + + task = _task(new_node_id="new-1") + assert cluster_expand.SPEC.handler(task) is None + cluster_expand.tasks_controller.add_new_device_mig_task.assert_called_once_with("dev-1") + assert task.function_result == "expansion complete: new-1" + + +def test_cluster_expand_rearms_an_aborted_plan(cluster_expand): + from simplyblock_core.controllers.cluster_expansion.planner import ( + EXPAND_PHASE_ABORTED, + EXPAND_PHASE_COMPLETED, + ) + + cluster = MagicMock() + cluster.expand_state = {"phase": EXPAND_PHASE_ABORTED} + completed = MagicMock() + completed.expand_state = {"phase": EXPAND_PHASE_COMPLETED} + cluster_expand.db.get_cluster_by_id.side_effect = [cluster, completed] + cluster_expand.db.get_storage_node_by_id.return_value.nvme_devices = [] + + cluster_expand.SPEC.handler(_task(new_node_id="new-1")) + assert cluster.expand_state["phase"] != EXPAND_PHASE_ABORTED + cluster.write_to_db.assert_called_once() + + +def test_cluster_expand_retries_an_incomplete_phase(cluster_expand): + _expanded_cluster(cluster_expand, "in_progress") + with pytest.raises(trb.TaskRetry, match="unexpected phase"): + cluster_expand.SPEC.handler(_task(new_node_id="new-1")) + + +def test_cluster_expand_aborts_without_a_node(cluster_expand): + with pytest.raises(trb.TaskAbort): + cluster_expand.SPEC.handler(_task()) + + +# -- migration family ------------------------------------------------------- + +@pytest.fixture +def mig(monkeypatch): + import simplyblock_core.services.migration_task_common as common + monkeypatch.setattr(common, "db", MagicMock()) + monkeypatch.setattr(common, "tasks_controller", MagicMock()) + common.tasks_controller.get_active_node_mig_task.return_value = False + return common + + +def _status(state, error=0, progress=50): + return [{"status": state, "error": error, "progress": progress}] + + +def test_migration_completion_finishes_the_task(mig): + task = _task(distr_name="distr-1") + assert mig.report_migration_status(task, _status("completed")) is None + assert task.function_result == "Done" + + +def test_migration_in_progress_keeps_the_task_running(mig): + """Not a defer: suspending between polls would drop the RUNNING status + that the family's mutual exclusion is keyed on.""" + with pytest.raises(trb.TaskProgress): + mig.report_migration_status(_task(), _status("in_progress")) + + +def test_migration_failure_is_terminal(mig): + with pytest.raises(trb.TaskAbort): + mig.report_migration_status(_task(), _status("failed")) + + +def test_migration_error_restarts_from_scratch(mig): + """A disallowed error code drops the marker so the next attempt issues a + fresh migration instead of polling the one that errored.""" + task = _task(migration={"name": "distr-1"}) + with pytest.raises(trb.TaskRetry): + mig.report_migration_status(task, _status("completed", error=7)) + assert "migration" not in task.function_params + + +def test_migration_error_tolerated_when_devices_are_degraded(mig): + task = _task(migration={"name": "distr-1"}) + assert mig.report_migration_status(task, _status("completed", error=7), + allow_all_errors=True) is None + assert task.function_params["migration"] == {"name": "distr-1"} + + +def test_missing_migration_restarts_from_scratch(mig): + task = _task(migration={"name": "distr-1"}) + with pytest.raises(trb.TaskRetry): + mig.report_migration_status(task, _status("none")) + assert "migration" not in task.function_params + + +def test_empty_status_response_is_retryable(mig): + with pytest.raises(trb.TaskRetry): + mig.report_migration_status(_task(), None) + + +# -- migration recovery gate ------------------------------------------------ + +def test_recovery_gate_passes_a_whole_cluster(mig): + task = _task(**{mig.MIGRATION_WAIT_UNAVAILABLE_KEY: ["node:n1"]}) + assert mig.require_recovery_progress(task, []) is None + assert mig.MIGRATION_WAIT_UNAVAILABLE_KEY not in task.function_params + + +def test_recovery_gate_defers_while_nothing_changes(mig): + """Retrying against an unchanged outage would burn the budget and + terminate the migration before the cluster came back.""" + task = _task(**{mig.MIGRATION_WAIT_UNAVAILABLE_KEY: ["node:n1"]}) + with pytest.raises(trb.TaskDefer): + mig.require_recovery_progress(task, ["node:n1"]) + + +def test_recovery_gate_releases_on_a_recovery_event(mig): + task = _task(**{mig.MIGRATION_WAIT_UNAVAILABLE_KEY: ["node:n1", "dev:d1"]}) + assert mig.require_recovery_progress(task, ["dev:d1"]) is None + assert task.function_params[mig.MIGRATION_WAIT_UNAVAILABLE_KEY] == ["dev:d1"] + + +def test_recovery_gate_defers_on_a_first_outage(mig): + with pytest.raises(trb.TaskDefer): + mig.require_recovery_progress(_task(), ["node:n1"]) + + +# -- migration-family eligibility ------------------------------------------- + +def test_sibling_gate_blocks_a_task_that_has_not_started(mig): + mig.tasks_controller.get_active_node_mig_task.return_value = "other-task" + assert mig.no_sibling_migration(_task(distr_name="distr-1")) is False + + +def test_sibling_gate_releases_a_task_already_migrating(mig): + """Once its own migration is running the task IS the sibling others wait + on — gating it here would stall it forever.""" + mig.tasks_controller.get_active_node_mig_task.return_value = "other-task" + task = _task(distr_name="distr-1", migration={"name": "distr-1"}) + assert mig.no_sibling_migration(task) is True + + +# -- failed migration ------------------------------------------------------- + +@pytest.fixture +def failed_migration(monkeypatch): + import simplyblock_core.services.tasks_runner_failed_migration as runner + monkeypatch.setattr(runner, "db", MagicMock()) + monkeypatch.setattr(runner, "tasks_controller", MagicMock()) + monkeypatch.setattr(runner, "device_controller", MagicMock()) + return runner + + +def test_failed_migration_tags_the_device_once_finished(failed_migration): + failed_migration.tasks_controller.get_failed_device_mig_task.return_value = False + task = _task(distr_name="distr-1", migration={"name": "distr-1"}) + task.device_id = "dev-1" + + failed_migration.SPEC.on_finish(task) + + failed_migration.device_controller.device_set_failed_and_migrated.assert_called_once_with("dev-1") + + +def test_failed_migration_leaves_the_device_alone_if_never_started(failed_migration): + failed_migration.tasks_controller.get_failed_device_mig_task.return_value = False + task = _task(distr_name="distr-1") + task.device_id = "dev-1" + + failed_migration.SPEC.on_finish(task) + + failed_migration.device_controller.device_set_failed_and_migrated.assert_not_called() + + +def test_failed_migration_waits_for_the_last_task_on_the_device(failed_migration): + failed_migration.tasks_controller.get_failed_device_mig_task.return_value = "other-task" + task = _task(distr_name="distr-1", migration={"name": "distr-1"}) + task.device_id = "dev-1" + + failed_migration.SPEC.on_finish(task) + + failed_migration.device_controller.device_set_failed_and_migrated.assert_not_called() + + +# -- every migrated runner -------------------------------------------------- + +MIGRATED_RUNNERS = [ + "fdb_backup", "jc_comp", "replication_final", "sync_lvol_del", "backup", + "cluster_expand", "node_add", "restart", "migration", "new_dev_migration", + "failed_migration", "node_removal", "port_allow", +] + + +@pytest.mark.parametrize("name", MIGRATED_RUNNERS) +def test_runner_module_defines_a_usable_spec(name): + """Import-smoke plus spec sanity: the module-level definitions still load + under the stubbed-fdb unit env, and what serve() will be handed is + actually runnable.""" + import importlib + module = importlib.import_module(f"simplyblock_core.services.tasks_runner_{name}") + + spec = module.SPEC + assert spec.function_names, f"{name}: no function names" + assert callable(spec.handler) + assert callable(spec.is_eligible) + assert spec.concurrency >= 1 + assert spec.interval > 0 + for optional in (spec.on_finish, spec.on_cycle, spec.serialize, + spec.exclusion_key, spec.backoff): + assert optional is None or callable(optional) + assert callable(module.main) + + +def test_every_task_function_has_exactly_one_runner(): + """Two runners claiming the same function name would both drive it, and + the lease only guards against a second host, not a second spec.""" + import importlib + owners = {} + for name in MIGRATED_RUNNERS: + module = importlib.import_module(f"simplyblock_core.services.tasks_runner_{name}") + for function_name in module.SPEC.function_names: + owners.setdefault(function_name, []).append(name) + + duplicates = {fn: names for fn, names in owners.items() if len(names) > 1} + assert not duplicates, f"function names claimed by more than one runner: {duplicates}" diff --git a/tests/unit/tasks/test_task_runner_base.py b/tests/unit/tasks/test_task_runner_base.py new file mode 100644 index 0000000000..c04a416c2e --- /dev/null +++ b/tests/unit/tasks/test_task_runner_base.py @@ -0,0 +1,732 @@ +# coding=utf-8 +"""Unit tests for the shared task-runner driver (``task_runner_base``). + +These exercise the per-task lifecycle (``_process``) and the dispatch loop +(``run``) directly, against a fake store that models FoundationDB's two write +paths faithfully. They pin the handler contract: return → DONE, TaskDefer → +suspend without a retry, TaskRetry / unexpected Exception → suspend + retry + +backoff, TaskAbort → DONE; the pre-run skip-gates (eligibility, lease); that a +handler exception never escapes the loop while a DB error does; and — the part +that needs a real store to observe — that no transition can revert what another +actor committed while the handler was running. + +Assertions read the committed row (``store.row()``), not the caller's copy: the +driver's writes are compare-and-set against the current row and deliberately do +not mutate the stale object it was holding. +""" +import copy +import threading +import time +from unittest.mock import MagicMock + +import pytest + +from simplyblock_core.models.job_schedule import JobSchedule +import simplyblock_core.services.task_runner_base as trb + + +def _task(status=JobSchedule.STATUS_NEW, retry=0, max_retry=8, canceled=False): + task = JobSchedule() + task.uuid = "task-1" + task.function_name = "fn" + task.cluster_id = "cl-1" + task.node_id = "node-1" + task.status = status + task.retry = retry + task.max_retry = max_retry + task.canceled = canceled + task.function_params = {} + return task + + +class _Store: + """The task row plus the two ways it gets written, modelled faithfully. + + ``atomic_update`` follows DBController's contract: the mutator runs against + the row as it exists *in the store*, not against the caller's copy; a + mutator returning False aborts the write; the return is the fresh object, + or None when the row is gone. ``full_write`` is the ``write_to_db`` path — + it replaces the row wholesale from the caller's (possibly stale) copy. + """ + + def __init__(self, task): + self.kv_store = "KV" + self._rows = {task.uuid: copy.deepcopy(task)} + + # DBController surface used by the driver + def get_task_by_id(self, uuid): + row = self._rows.get(uuid) + return copy.deepcopy(row) if row is not None else None + + def atomic_update(self, obj, mutate): + row = self._rows.get(obj.uuid) + if row is None: + return None + fresh = copy.deepcopy(row) + if mutate(fresh) is False: + return copy.deepcopy(row) + self._rows[obj.uuid] = fresh + return copy.deepcopy(fresh) + + # test surface + def full_write(self, obj): + self._rows[obj.uuid] = copy.deepcopy(obj) + + def row(self, uuid="task-1"): + return self._rows[uuid] + + def concurrently(self, uuid="task-1", **fields): + """Another actor commits to the row while the handler is running.""" + for name, value in fields.items(): + setattr(self._rows[uuid], name, value) + + +def _wire(monkeypatch, task, claim=True): + """Point the driver at a fake store, grant the lease, and neutralise the + heartbeat (a no-op context manager, so no real thread is spawned).""" + store = _Store(task) + monkeypatch.setattr(trb, "db", store) + monkeypatch.setattr(JobSchedule, "write_to_db", + lambda self, kv=None: store.full_write(self)) + monkeypatch.setattr(trb.tasks_controller, "claim_task", lambda *a, **k: claim) + monkeypatch.setattr(trb.tasks_controller, "task_lease_heartbeat", MagicMock()) + return store + + +def _runner(handler, **spec_kw): + spec = trb.RunnerSpec(function_names=("fn",), handler=handler, **spec_kw) + return trb.TaskRunner(spec) + + +# -- handler outcome vocabulary -------------------------------------------- + +def test_void_return_marks_done(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + handler = MagicMock(return_value=None) + + _runner(handler)._process(task, MagicMock()) + + assert handler.call_args[0][0].uuid == "task-1" + assert store.row().status == JobSchedule.STATUS_DONE + + +def test_handler_runs_under_lease_heartbeat(monkeypatch): + task = _task() + _wire(monkeypatch, task) + hb = MagicMock() + monkeypatch.setattr(trb.tasks_controller, "task_lease_heartbeat", hb) + + _runner(MagicMock(return_value=None))._process(task, MagicMock()) + + assert hb.call_args[0][0].uuid == "task-1" + hb.return_value.__enter__.assert_called_once() + hb.return_value.__exit__.assert_called_once() + + +def test_defer_suspends_without_consuming_retry(monkeypatch): + task = _task(retry=2) + store = _wire(monkeypatch, task) + + def handler(_task): + raise trb.TaskDefer("node not online") + + runner = _runner(handler) + runner._process(task, MagicMock()) + + assert store.row().status == JobSchedule.STATUS_SUSPENDED + assert store.row().retry == 2 + assert store.row().function_result == "node not online" + assert "task-1" not in runner._next_attempt # no backoff for a defer + + +def test_retry_suspends_consumes_retry_and_backs_off(monkeypatch): + task = _task(retry=1) + store = _wire(monkeypatch, task) + + def handler(_task): + raise trb.TaskRetry("rpc failed") + + runner = _runner(handler) + runner._process(task, MagicMock()) + + assert store.row().status == JobSchedule.STATUS_SUSPENDED + assert store.row().retry == 2 + assert runner._next_attempt["task-1"] > time.time() + + +def test_unexpected_exception_is_treated_as_retry(monkeypatch): + task = _task(retry=0) + store = _wire(monkeypatch, task) + + def handler(_task): + raise RuntimeError("boom") + + _runner(handler)._process(task, MagicMock()) # must not raise + + assert store.row().status == JobSchedule.STATUS_SUSPENDED + assert store.row().retry == 1 + + +def test_success_message_comes_from_the_handler(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + + def handler(t): + t.function_result = "Backup created" + + _runner(handler)._process(task, MagicMock()) + + assert store.row().status == JobSchedule.STATUS_DONE + assert store.row().function_result == "Backup created" + + +def test_previous_failure_result_does_not_survive_a_later_success(monkeypatch): + task = _task(status=JobSchedule.STATUS_SUSPENDED, retry=1) + task.function_result = "rpc failed" + store = _wire(monkeypatch, task) + + _runner(MagicMock(return_value=None))._process(task, MagicMock()) + + assert store.row().status == JobSchedule.STATUS_DONE + assert store.row().function_result == "completed" + + +def test_abort_marks_done_with_reason(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + + def handler(_task): + raise trb.TaskAbort("missing param") + + _runner(handler)._process(task, MagicMock()) + + assert store.row().status == JobSchedule.STATUS_DONE + assert store.row().function_result == "missing param" + + +# -- terminal cleanup hook -------------------------------------------------- + +@pytest.mark.parametrize("handler,expected", [ + (MagicMock(return_value=None), "completed"), + (MagicMock(side_effect=trb.TaskAbort("gone")), "gone"), +]) +def test_on_finish_runs_for_every_terminal_outcome(monkeypatch, handler, expected): + task = _task() + store = _wire(monkeypatch, task) + on_finish = MagicMock() + + _runner(handler, on_finish=on_finish)._process(task, MagicMock()) + + assert on_finish.call_args[0][0].uuid == "task-1" + assert store.row().status == JobSchedule.STATUS_DONE + assert store.row().function_result == expected + + +def test_on_finish_runs_when_the_handler_is_never_reached(monkeypatch): + task = _task(canceled=True) + _wire(monkeypatch, task) + on_finish = MagicMock() + + _runner(MagicMock(), on_finish=on_finish)._process(task, MagicMock()) + + on_finish.assert_called_once() + + +def test_on_finish_does_not_run_for_a_suspended_task(monkeypatch): + task = _task() + _wire(monkeypatch, task) + on_finish = MagicMock() + + _runner(MagicMock(side_effect=trb.TaskDefer("later")), + on_finish=on_finish)._process(task, MagicMock()) + + on_finish.assert_not_called() + + +def test_failing_on_finish_does_not_break_the_task(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + + runner = _runner(MagicMock(return_value=None), + on_finish=MagicMock(side_effect=RuntimeError("cleanup boom"))) + runner._process(task, MagicMock()) # must not raise + + assert store.row().status == JobSchedule.STATUS_DONE + + +# -- on_failure alerts ------------------------------------------------------ + +def test_on_failure_reports_a_failed_attempt(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + on_failure = MagicMock() + + _runner(MagicMock(side_effect=trb.TaskRetry("boom")), + on_failure=on_failure)._process(task, MagicMock()) + + reported_task, reason = on_failure.call_args[0] + assert reported_task.uuid == "task-1" + assert reason == "boom" + assert store.row().status == JobSchedule.STATUS_SUSPENDED + + +def test_on_failure_is_not_repeated_for_the_same_message(monkeypatch): + """A cause that persists for hours must write ONE alert, not one per + attempt. The handler cannot tell — it is handed a cleared function_result — + so the driver compares against the message the row still carries.""" + task = _task(status=JobSchedule.STATUS_SUSPENDED) + task.function_result = "boom" + store = _wire(monkeypatch, task) + on_failure = MagicMock() + + _runner(MagicMock(side_effect=trb.TaskRetry("boom")), + on_failure=on_failure)._process(task, MagicMock()) + + on_failure.assert_not_called() + assert store.row().retry == 1 + + +def test_on_failure_reports_a_changed_message(monkeypatch): + task = _task(status=JobSchedule.STATUS_SUSPENDED) + task.function_result = "connection refused" + _wire(monkeypatch, task) + on_failure = MagicMock() + + _runner(MagicMock(side_effect=trb.TaskRetry("timeout")), + on_failure=on_failure)._process(task, MagicMock()) + + assert on_failure.call_args[0][1] == "timeout" + + +def test_on_failure_does_not_run_for_a_deferred_task(monkeypatch): + task = _task() + _wire(monkeypatch, task) + on_failure = MagicMock() + + _runner(MagicMock(side_effect=trb.TaskDefer("waiting")), + on_failure=on_failure)._process(task, MagicMock()) + + on_failure.assert_not_called() + + +def test_on_failure_does_not_run_when_another_actor_owns_the_outcome(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + on_failure = MagicMock() + + def _handler(t): + store.concurrently(canceled=True) + raise trb.TaskRetry("boom") + + _runner(_handler, on_failure=on_failure)._process(task, MagicMock()) + + on_failure.assert_not_called() + + +def test_failing_on_failure_does_not_break_the_task(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + + runner = _runner(MagicMock(side_effect=trb.TaskRetry("boom")), + on_failure=MagicMock(side_effect=RuntimeError("alert boom"))) + runner._process(task, MagicMock()) # must not raise + + assert store.row().status == JobSchedule.STATUS_SUSPENDED + assert store.row().retry == 1 + + +# -- pre-run skip-gates ----------------------------------------------------- + +def test_ineligible_skips_without_claim_or_write(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + claim = MagicMock(return_value=True) + monkeypatch.setattr(trb.tasks_controller, "claim_task", claim) + handler = MagicMock() + + _runner(handler, is_eligible=lambda t, c: False)._process(task, MagicMock()) + + handler.assert_not_called() + claim.assert_not_called() + assert store.row().status == JobSchedule.STATUS_NEW + + +def test_default_eligible_runs(monkeypatch): + task = _task() + _wire(monkeypatch, task) + handler = MagicMock(return_value=None) + + _runner(handler)._process(task, MagicMock()) + + handler.assert_called_once() + + +def test_lease_denied_skips(monkeypatch): + task = _task() + store = _wire(monkeypatch, task, claim=False) + handler = MagicMock() + + _runner(handler)._process(task, MagicMock()) + + handler.assert_not_called() + assert store.row().status == JobSchedule.STATUS_NEW + + +# -- terminal pre-handler checks ------------------------------------------- + +def test_canceled_marks_done_without_handler(monkeypatch): + task = _task(canceled=True) + store = _wire(monkeypatch, task) + handler = MagicMock() + + _runner(handler)._process(task, MagicMock()) + + handler.assert_not_called() + assert store.row().status == JobSchedule.STATUS_DONE + assert store.row().function_result == "canceled" + + +def test_max_retry_marks_done_without_handler(monkeypatch): + task = _task(retry=8, max_retry=8) + store = _wire(monkeypatch, task) + handler = MagicMock() + + _runner(handler)._process(task, MagicMock()) + + handler.assert_not_called() + assert store.row().status == JobSchedule.STATUS_DONE + assert "max retry" in store.row().function_result + + +def test_negative_max_retry_is_unbounded(monkeypatch): + task = _task(retry=100, max_retry=-1) + store = _wire(monkeypatch, task) + handler = MagicMock(return_value=None) + + _runner(handler)._process(task, MagicMock()) + + handler.assert_called_once() # ceiling never binds for max_retry < 0 + assert store.row().status == JobSchedule.STATUS_DONE + + +# -- concurrent-writer safety ---------------------------------------------- +# +# A handler runs for minutes (node add, restart, migration) while the driver +# holds the task copy it fetched beforehand. Other actors write that row in the +# meantime — set_node_status(ONLINE) cancels restart tasks +# (tasks_controller.cancel_pending_node_restart_tasks), an operator cancels a +# task, another host's lease heartbeat stamps it. A full-object write of the +# stale copy silently reverts all of that: it is what un-canceled a task and +# wiped its owner lease in the 2026-07-29 double-restart incident, and it is +# why upstream converted the restart runner's task writes to atomic CAS. + +def test_defer_does_not_resurrect_a_concurrently_canceled_task(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + + def handler(_task): + store.concurrently(canceled=True, status=JobSchedule.STATUS_DONE, + function_result="canceled: node back online") + raise trb.TaskDefer("peer is restarting") + + _runner(handler)._process(task, MagicMock()) + + row = store.row() + assert row.canceled is True + assert row.status == JobSchedule.STATUS_DONE + assert row.function_result == "canceled: node back online" + + +def test_failure_does_not_resurrect_a_concurrently_canceled_task(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + + def handler(_task): + store.concurrently(canceled=True, status=JobSchedule.STATUS_DONE) + raise trb.TaskRetry("rpc failed") + + _runner(handler)._process(task, MagicMock()) + + assert store.row().canceled is True + assert store.row().status == JobSchedule.STATUS_DONE + + +def test_write_does_not_steal_a_lease_taken_during_the_handler(monkeypatch): + task = _task() + task.owner = "this-host" + store = _wire(monkeypatch, task) + + def handler(_task): + store.concurrently(owner="other-host") + + _runner(handler)._process(task, MagicMock()) + + assert store.row().owner == "other-host" + + +def test_retry_is_counted_on_the_fresh_row(monkeypatch): + task = _task(retry=1) + store = _wire(monkeypatch, task) + + def handler(_task): + store.concurrently(retry=5) + raise trb.TaskRetry("rpc failed") + + _runner(handler)._process(task, MagicMock()) + + assert store.row().retry == 6 + + +def test_handler_progress_is_carried_onto_the_fresh_row(monkeypatch): + """Handlers record progress in function_params (recovery_started, + merge_started, fail_count) — a CAS that only wrote the lifecycle fields + would drop it and the next attempt would re-issue the RPC.""" + task = _task() + store = _wire(monkeypatch, task) + + def handler(t): + t.function_params["recovery_started"] = True + raise trb.TaskDefer("Restore started") + + _runner(handler)._process(task, MagicMock()) + + assert store.row().function_params["recovery_started"] is True + + +def test_on_finish_is_skipped_when_another_actor_finished_the_task(monkeypatch): + """Cleanup is a side effect of winning the terminal transition. Running it + off a lost CAS means two hosts both release the resource.""" + task = _task() + store = _wire(monkeypatch, task) + on_finish = MagicMock() + + def handler(_task): + store.concurrently(status=JobSchedule.STATUS_DONE, + function_result="canceled: node back online") + + _runner(handler, on_finish=on_finish)._process(task, MagicMock()) + + on_finish.assert_not_called() + assert store.row().function_result == "canceled: node back online" + + +# -- single dispatch path --------------------------------------------------- +# +# The 2026-07-29 incident's other half: restart had a parallel branch that +# consulted the inflight map and an inline branch that consulted neither, so a +# dispatch-mode flip mid-restart re-entered a task still running on the pool. +# Serial execution must therefore register in-flight exactly like parallel +# execution does. + +def test_a_running_task_is_not_re_entered_by_a_second_dispatch(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + calls, started, release = [], threading.Event(), threading.Event() + + def handler(_task): + calls.append(1) + started.set() + release.wait(5) + + runner = _runner(handler) + cluster = MagicMock() + + worker = threading.Thread(target=runner._dispatch, args=(task, cluster)) + worker.start() + try: + assert started.wait(5) + # The dispatch loop comes round again while the task is still running. + runner._dispatch(store.get_task_by_id("task-1"), cluster) + finally: + release.set() + worker.join(5) + + assert calls == [1] + + +def test_serialized_dispatch_waits_for_the_task(monkeypatch): + """Serialized mode must submit and wait, not run inline — that is what + makes a mode flip harmless in both directions.""" + task = _task() + store = _wire(monkeypatch, task) + + runner = _runner(MagicMock(return_value=None), concurrency=4, + serialize=lambda t, c: True) + runner._dispatch(task, MagicMock()) + + assert store.row().status == JobSchedule.STATUS_DONE + + +def test_parallel_dispatch_does_not_wait(monkeypatch): + task = _task() + _wire(monkeypatch, task) + started, release = threading.Event(), threading.Event() + + def handler(_task): + started.set() + release.wait(5) + + runner = _runner(handler, concurrency=4) + try: + runner._dispatch(task, MagicMock()) # returns while the handler runs + assert started.wait(5) + finally: + release.set() + + +# -- dispatch loop ---------------------------------------------------------- + +class _StopLoop(Exception): + pass + + +def test_run_dispatches_only_matching_non_done(monkeypatch): + match = _task() + other = _task() + other.uuid = "other" + other.function_name = "nope" + done = _task() + done.uuid = "done" + done.status = JobSchedule.STATUS_DONE + + cluster = MagicMock() + cluster.get_id.return_value = "cl-1" + db = MagicMock() + db.get_clusters.return_value = [cluster] + db.get_job_tasks.return_value = [match, other, done] + monkeypatch.setattr(trb, "db", db) + monkeypatch.setattr(trb.time, "sleep", MagicMock(side_effect=_StopLoop)) + + runner = _runner(MagicMock()) + dispatched = [] + monkeypatch.setattr(runner, "_dispatch", lambda t, c: dispatched.append(t.uuid)) + + with pytest.raises(_StopLoop): + runner.run() + assert dispatched == ["task-1"] + + +def test_run_propagates_db_error(monkeypatch): + db = MagicMock() + db.get_clusters.side_effect = RuntimeError("fdb down") + monkeypatch.setattr(trb, "db", db) + with pytest.raises(RuntimeError, match="fdb down"): + _runner(MagicMock()).run() + + +# -- spec validation -------------------------------------------------------- + +def test_spec_rejects_zero_concurrency(): + with pytest.raises(ValueError): + trb.RunnerSpec(function_names=("fn",), handler=MagicMock(), concurrency=0) + + +# -- handler progress checkpoints ------------------------------------------- + +def test_checkpoint_persists_progress_immediately(monkeypatch): + """A destructive step records itself the moment it succeeds, so a crash + before the handler returns does not repeat it.""" + task = _task() + store = _wire(monkeypatch, task) + + fresh = trb.checkpoint(store.get_task_by_id("task-1"), cleanup_shutdown_done=True) + + assert fresh.function_params["cleanup_shutdown_done"] is True + assert store.row().function_params["cleanup_shutdown_done"] is True + + +def test_checkpoint_keeps_existing_params(monkeypatch): + task = _task() + task.function_params = {"node_addr": "1.2.3.4:5000"} + store = _wire(monkeypatch, task) + + trb.checkpoint(store.get_task_by_id("task-1"), cleanup_shutdown_done=True) + + assert store.row().function_params == { + "node_addr": "1.2.3.4:5000", "cleanup_shutdown_done": True} + + +def test_checkpoint_reports_a_cancellation_under_the_handler(monkeypatch): + """The handler's cancellation probe: a task canceled mid-handler must not + go on to the next destructive step.""" + task = _task() + store = _wire(monkeypatch, task) + store.concurrently(canceled=True) + + assert trb.checkpoint(store.get_task_by_id("task-1"), step_done=True) is None + assert "step_done" not in store.row().function_params + + +def test_checkpoint_reports_a_task_finished_under_the_handler(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + store.concurrently(status=JobSchedule.STATUS_DONE) + + assert trb.checkpoint(store.get_task_by_id("task-1"), step_done=True) is None + + +# -- per-cluster cycle hook ------------------------------------------------- + +def test_cycle_hook_runs_once_per_cluster(monkeypatch): + cluster = MagicMock() + cluster.get_id.return_value = "cl-1" + db = MagicMock() + db.get_clusters.return_value = [cluster] + db.get_job_tasks.return_value = [] + monkeypatch.setattr(trb, "db", db) + monkeypatch.setattr(trb.time, "sleep", MagicMock(side_effect=_StopLoop)) + on_cycle = MagicMock() + + with pytest.raises(_StopLoop): + _runner(MagicMock(), on_cycle=on_cycle).run() + + on_cycle.assert_called_once_with(cluster) + + +def test_failing_cycle_hook_does_not_stop_the_loop(monkeypatch): + cluster = MagicMock() + cluster.get_id.return_value = "cl-1" + db = MagicMock() + db.get_clusters.return_value = [cluster] + db.get_job_tasks.return_value = [] + monkeypatch.setattr(trb, "db", db) + monkeypatch.setattr(trb.time, "sleep", MagicMock(side_effect=_StopLoop)) + + runner = _runner(MagicMock(), on_cycle=MagicMock(side_effect=RuntimeError("watchdog boom"))) + with pytest.raises(_StopLoop): # reached the sleep, i.e. the cycle completed + runner.run() + + +# -- in-progress polling ---------------------------------------------------- + +def test_progress_keeps_the_task_running(monkeypatch): + """A polled long-running operation must not be suspended between polls: + the migration family gates mutual exclusion on a sibling being RUNNING.""" + task = _task(retry=2) + store = _wire(monkeypatch, task) + + def handler(_task): + raise trb.TaskProgress("Status: in_progress, progress:42") + + runner = _runner(handler) + runner._process(task, MagicMock()) + + assert store.row().status == JobSchedule.STATUS_RUNNING + assert store.row().retry == 2 + assert store.row().function_result == "Status: in_progress, progress:42" + assert "task-1" not in runner._next_attempt + + +def test_progress_does_not_resurrect_a_concurrently_canceled_task(monkeypatch): + task = _task() + store = _wire(monkeypatch, task) + + def handler(_task): + store.concurrently(canceled=True, status=JobSchedule.STATUS_DONE) + raise trb.TaskProgress("still going") + + _runner(handler)._process(task, MagicMock()) + + assert store.row().canceled is True + assert store.row().status == JobSchedule.STATUS_DONE diff --git a/tests/unit/test_lifecycle_alert_events.py b/tests/unit/test_lifecycle_alert_events.py index af3d3b72a9..a55f258b18 100644 --- a/tests/unit/test_lifecycle_alert_events.py +++ b/tests/unit/test_lifecycle_alert_events.py @@ -5,7 +5,7 @@ 1. A sync delete failing after the async delete already succeeded. The data is going away, but a node still holds its replica bdev and the volume is pinned in_deletion until the deferred task drains -- invisible from the volume list - alone. The runner retries every 3 seconds, so the event fires only when the + alone. The runner declares the alert; the task runner fires it only when the failure message CHANGES, not per retry. 2. An lvs journal accumulating more than JM_COMPRESSION_BACKLOG_ALERT_RECORDS @@ -18,6 +18,7 @@ from unittest.mock import MagicMock, patch from simplyblock_core import constants +from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.services import main_distr_event_collector as collector from simplyblock_core.services import tasks_runner_sync_lvol_del as sync_del @@ -27,50 +28,42 @@ class TestSyncDeleteFailureEvent(unittest.TestCase): + """The runner's alert hook, which the task runner calls on a failed attempt. + That it fires once per DISTINCT failure rather than once per retry, and that + a broken event log cannot take the runner down, are the driver's contracts — + see tests/unit/tasks/test_task_runner_base.py.""" - def _task(self, previous_result=""): + def _task(self, function_name=JobSchedule.FN_LVOL_SYNC_DEL, + bdev="LVS_1/LVOL_5"): task = MagicMock() - task.function_result = previous_result + task.uuid = "task-1" + task.cluster_id = "cl-1" + task.node_id = "node-1" + task.function_name = function_name + task.function_params = {"lvol_bdev_name": bdev} return task - def _node(self): - node = MagicMock() - node.cluster_id = "cl-1" - node.get_id.return_value = "node-1" - return node - - def test_first_failure_is_logged_as_an_error(self): + def test_a_failure_is_reported_as_an_error(self): with patch.object(sync_del.events_controller, "log_event_cluster") as log: - sync_del._log_sync_delete_failure( - self._task(), self._node(), "LVS_1/LVOL_5", "connection refused") + sync_del.alert_sync_delete_failure(self._task(), "connection refused") log.assert_called_once() kwargs = log.call_args.kwargs self.assertEqual(kwargs["event_level"], "Error") self.assertEqual(kwargs["event"], "SYNC_DELETE_FAILED") self.assertEqual(kwargs["node_id"], "node-1") + self.assertEqual(kwargs["cluster_id"], "cl-1") self.assertIn("in_deletion", kwargs["message"]) self.assertIn("LVS_1/LVOL_5", kwargs["message"]) + self.assertIn("connection refused", kwargs["message"]) - def test_a_repeat_of_the_same_failure_is_not_logged_again(self): - """The runner retries every 3s; identical failures must not flood.""" - task = self._task(previous_result="boom") + def test_a_sync_op_failure_is_not_a_sync_delete_alert(self): + """Both task families share this runner, but only a delete leaves a + volume pinned in_deletion.""" with patch.object(sync_del.events_controller, "log_event_cluster") as log: - sync_del._log_sync_delete_failure(task, self._node(), "b", "boom") + sync_del.alert_sync_delete_failure( + self._task(function_name=JobSchedule.FN_LVOL_SYNC_OP), "boom") log.assert_not_called() - def test_a_different_failure_is_logged(self): - task = self._task(previous_result="connection refused") - with patch.object(sync_del.events_controller, "log_event_cluster") as log: - sync_del._log_sync_delete_failure(task, self._node(), "b", "timeout") - log.assert_called_once() - - def test_event_log_trouble_does_not_break_the_runner(self): - with patch.object(sync_del.events_controller, "log_event_cluster", - side_effect=RuntimeError("db gone")), \ - patch.object(sync_del, "logger"): - sync_del._log_sync_delete_failure( - self._task(), self._node(), "b", "boom") # must not raise - class TestCompressionBacklogEvent(unittest.TestCase): diff --git a/tests/unit/test_lvol_sync_op_task.py b/tests/unit/test_lvol_sync_op_task.py index 2e2e57191f..821d2adee0 100644 --- a/tests/unit/test_lvol_sync_op_task.py +++ b/tests/unit/test_lvol_sync_op_task.py @@ -5,6 +5,11 @@ ``_restart_op_queues`` deferral (incident 2026-07-10: a create-registration queued in the webappapi's in-memory queue was never drained; the volume's tertiary subsystem was never created). + +The runner sits on the shared driver, so its handler is void: it returns on +success and raises ``TaskDefer`` (retry later) / ``TaskAbort`` (permanently +obsolete). Task status transitions belong to the driver and are covered by +tests/unit/tasks/test_task_runner_base.py. """ import unittest @@ -14,6 +19,8 @@ from simplyblock_core.models.job_schedule import JobSchedule from simplyblock_core.models.lvol_model import LVol from simplyblock_core.models.storage_node import StorageNode +from simplyblock_core.services import tasks_runner_sync_lvol_del as runner +from simplyblock_core.services.task_runner_base import TaskAbort, TaskDefer def _task(op="register", lvol_id="lv-1", node_id="node-2", canceled=False, @@ -50,7 +57,7 @@ def _node(status=StorageNode.STATUS_ONLINE, node_id="node-2"): class TestRunLvolSyncOpTask(unittest.TestCase): def setUp(self): - patcher = patch.object(tasks_controller, "db") + patcher = patch.object(runner, "db") self.db = patcher.start() self.addCleanup(patcher.stop) @@ -62,7 +69,7 @@ def _run(self, task, lvol=None, node=None, phase=""): patch("simplyblock_core.storage_node_ops." "repair_lvol_registration_on_non_leader", return_value=(True, None)) as repair: - tasks_controller.run_lvol_sync_op_task(task) + runner.SPEC.handler(task) return repair def test_register_repairs_and_completes(self): @@ -70,47 +77,71 @@ def test_register_repairs_and_completes(self): repair = self._run(task) repair.assert_called_once() self.assertEqual(repair.call_args[0][2], 1) # secondary_index - self.assertEqual(task.status, JobSchedule.STATUS_DONE) + self.assertIn("registered lvol", task.function_result) - def test_deleted_lvol_completes_without_action(self): + def test_deleted_lvol_aborts_without_action(self): task = _task() self.db.get_lvol_by_id.side_effect = KeyError() - tasks_controller.run_lvol_sync_op_task(task) - self.assertEqual(task.status, JobSchedule.STATUS_DONE) - self.assertIn("no longer exists", task.function_result) + with self.assertRaises(TaskAbort) as ctx: + runner.SPEC.handler(task) + self.assertIn("no longer exists", str(ctx.exception)) def test_offline_node_defers(self): - task = _task() - self._run(task, node=_node(StorageNode.STATUS_OFFLINE)) - self.assertEqual(task.status, JobSchedule.STATUS_SUSPENDED) + with self.assertRaises(TaskDefer): + self._run(_task(), node=_node(StorageNode.STATUS_OFFLINE)) def test_owned_lvs_defers(self): - task = _task() - self._run(task, phase=StorageNode.RESTART_PHASE_BLOCKED) - self.assertEqual(task.status, JobSchedule.STATUS_SUSPENDED) + with self.assertRaises(TaskDefer): + self._run(_task(), phase=StorageNode.RESTART_PHASE_BLOCKED) - def test_node_dropped_from_topology_completes(self): - task = _task(node_id="node-9") - self._run(task, node=_node(node_id="node-9")) - self.assertEqual(task.status, JobSchedule.STATUS_DONE) - self.assertIn("no longer hosts", task.function_result) + def test_node_dropped_from_topology_aborts(self): + with self.assertRaises(TaskAbort) as ctx: + self._run(_task(node_id="node-9"), node=_node(node_id="node-9")) + self.assertIn("no longer hosts", str(ctx.exception)) def test_resize_converges_to_current_db_size(self): task = _task(op="resize") - lvol = _lvol() node = _node() rpc = node.rpc_client.return_value rpc.bdev_lvol_resize.return_value = True - self.db.get_lvol_by_id.return_value = lvol + self.db.get_lvol_by_id.return_value = _lvol() self.db.get_storage_node_by_id.return_value = node with patch("simplyblock_core.storage_node_ops.get_restart_phase", return_value=""): - tasks_controller.run_lvol_sync_op_task(task) + runner.SPEC.handler(task) rpc.bdev_lvol_resize.assert_called_once() args = rpc.bdev_lvol_resize.call_args[0] self.assertEqual(args[0], "LVS_6/LVOL_1") self.assertEqual(args[1], 4096) # 4 GiB in MiB - self.assertEqual(task.status, JobSchedule.STATUS_DONE) + self.assertIn("resized lvol", task.function_result) + + def test_unknown_op_aborts(self): + with self.assertRaises(TaskAbort): + self._run(_task(op="nonsense")) + + +class TestDelSyncLockRelease(unittest.TestCase): + """The primary's del-sync lock is held for the lifetime of the delete task, + so it must be released on every terminal path — including the ones the + handler never reaches (cancel, retry ceiling).""" + + def test_lock_released_for_a_finished_delete_task(self): + task = MagicMock(spec=JobSchedule) + task.function_name = JobSchedule.FN_LVOL_SYNC_DEL + task.function_params = {"primary_node": "node-1"} + task.node_id = "node-2" + + with patch.object(runner, "db") as db: + primary = _node(node_id="node-1") + db.get_storage_node_by_id.return_value = primary + runner.SPEC.on_finish(task) + + primary.lvol_del_sync_lock_reset.assert_called_once() + + def test_sync_op_task_does_not_touch_the_lock(self): + with patch.object(runner, "db") as db: + runner.SPEC.on_finish(_task()) + db.get_storage_node_by_id.assert_not_called() class TestAddTaskDedup(unittest.TestCase): diff --git a/tests/unit/test_node_removal.py b/tests/unit/test_node_removal.py index 8abeeae0b8..3f57d22ea2 100644 --- a/tests/unit/test_node_removal.py +++ b/tests/unit/test_node_removal.py @@ -2149,28 +2149,36 @@ class TestShrinkStatusDoesNotDeadlockRemoval(unittest.TestCase): def _run_gate(self, cluster_status): """Drive the runner's cluster-status gate; True == it proceeded.""" from simplyblock_core.services import tasks_runner_failed_migration as runner + from simplyblock_core.services import migration_task_common as mig + from simplyblock_core.services.task_runner_base import TaskRetry from simplyblock_core.models.job_schedule import JobSchedule task = MagicMock(spec=JobSchedule) task.node_id, task.cluster_id, task.retry = "n1", "cl-1", 0 task.status = JobSchedule.STATUS_RUNNING + task.function_params = {} - cluster = MagicMock(spec=Cluster) + # A real Cluster, not a mock: the gate goes through Cluster's own + # status predicates, which a spec'd mock would answer truthily for + # every status and so pass the gate it is meant to test. + cluster = Cluster() cluster.status = cluster_status db = MagicMock() db.get_cluster_by_id.return_value = cluster db.get_storage_node_by_id.return_value = _node("n1", n_devices=1, with_jm=False) - with patch.object(runner, "db", db): + with patch.object(runner, "db", db), patch.object(mig, "db", db): try: runner.task_runner(task) + except TaskRetry as e: + # The gate refuses by raising; the driver, not the handler, is + # what turns that into STATUS_SUSPENDED. Any other TaskRetry + # comes from the real device/RPC work past the gate, which this + # test deliberately does not mock. + return "cluster is not active" not in str(e) except Exception: - # Admitted by the gate, then reached real device/DB work this - # test deliberately does not mock. The gate's decision is - # already recorded on task.status by that point, and it is the - # only thing under test here. pass - return task.status != JobSchedule.STATUS_SUSPENDED + return True def test_failed_migration_runner_admits_in_shrink(self): self.assertTrue( diff --git a/tests/unit/test_restart_claim.py b/tests/unit/test_restart_claim.py index 5a3481cd61..6597325498 100644 --- a/tests/unit/test_restart_claim.py +++ b/tests/unit/test_restart_claim.py @@ -22,8 +22,10 @@ - the wrapper's failure cleanup (_kill_spdk_until_dead + OFFLINE flip) runs only when THIS call actually holds the claim — a refused attempt must not destroy the rightful owner's in-flight restart; -- task_runner_node defers (no retry consumed) on a live foreign claim. +- task_runner_node raises TaskDefer (no retry consumed) on a live foreign + claim. """ +import contextlib import datetime import json from unittest.mock import MagicMock @@ -37,6 +39,8 @@ from simplyblock_core.models.storage_node import StorageNode import simplyblock_core.services.tasks_runner_restart as restart_runner +from simplyblock_core.services import task_runner_base +from simplyblock_core.services.task_runner_base import TaskAbort, TaskDefer, TaskRetry def _now(): @@ -312,7 +316,11 @@ def _runner_task(retry=0, max_retry=5, status=JobSchedule.STATUS_RUNNING): def test_runner_defers_without_retry_on_live_foreign_claim(monkeypatch): - claimed = _node(status=StorageNode.STATUS_RESTARTING, + # IN_SHUTDOWN, not RESTARTING: a node found RESTARTING without this task's + # own restart_issued marker is short-circuited earlier as redundant and + # never reaches the claim check. IN_SHUTDOWN is the shape the check was + # written for anyway — a CLI `sn restart` mid-shutdown holding the claim. + claimed = _node(status=StorageNode.STATUS_IN_SHUTDOWN, claim_owner="cli:100:aa", claim_age_sec=3) task = _runner_task() @@ -325,6 +333,9 @@ def test_runner_defers_without_retry_on_live_foreign_claim(monkeypatch): fake_db.get_storage_nodes_by_cluster_id.return_value = [] fake_db.atomic_update.side_effect = lambda obj, fn: (fn(obj), obj)[1] monkeypatch.setattr(restart_runner, "db", fake_db) + # The handler's checkpoint/CAS writes go through task_runner_base's own db + # handle, not the runner module's. + monkeypatch.setattr(task_runner_base, "db", fake_db) monkeypatch.setattr(JobSchedule, "write_to_db", MagicMock()) sops = MagicMock() @@ -336,17 +347,18 @@ def test_runner_defers_without_retry_on_live_foreign_claim(monkeypatch): hc._check_ping_from_node.return_value = True monkeypatch.setattr(restart_runner, "health_controller", hc) - res = restart_runner.task_runner_node(task) + # TaskDefer is the driver's no-retry-consumed signal; the driver, not the + # handler, records the reason and suspends the task. + with pytest.raises(TaskDefer, match="claim held by cli:100:aa"): + restart_runner.task_runner_node(task) - assert res is False # defer: outer loop backoff, retried later assert task.retry == 0 # no retry budget consumed - assert "claim held by cli:100:aa" in task.function_result sops.shutdown_storage_node.assert_not_called() sops.restart_storage_node.assert_not_called() def test_runner_proceeds_when_claim_stale(monkeypatch): - stale = _node(status=StorageNode.STATUS_RESTARTING, + stale = _node(status=StorageNode.STATUS_IN_SHUTDOWN, claim_owner="cli:100:aa", claim_age_sec=constants.RESTART_CLAIM_TTL_SEC + 30) task = _runner_task() @@ -360,6 +372,9 @@ def test_runner_proceeds_when_claim_stale(monkeypatch): fake_db.get_storage_nodes_by_cluster_id.return_value = [] fake_db.atomic_update.side_effect = lambda obj, fn: (fn(obj), obj)[1] monkeypatch.setattr(restart_runner, "db", fake_db) + # The handler's checkpoint/CAS writes go through task_runner_base's own db + # handle, not the runner module's. + monkeypatch.setattr(task_runner_base, "db", fake_db) monkeypatch.setattr(JobSchedule, "write_to_db", MagicMock()) sops = MagicMock() @@ -373,7 +388,12 @@ def test_runner_proceeds_when_claim_stale(monkeypatch): hc._check_ping_from_node.return_value = True monkeypatch.setattr(restart_runner, "health_controller", hc) - restart_runner.task_runner_node(task) + # Past the claim check the handler runs the full restart and ends on + # whatever the un-mocked completion check decides — irrelevant here, and + # suppressing it cannot mask a blocked claim: a blocked one raises + # TaskDefer before reaching the shutdown the assertion below requires. + with contextlib.suppress(TaskAbort, TaskDefer, TaskRetry): + restart_runner.task_runner_node(task) # The stale claim (dead driver) did not block the resume path: the # runner reached its cleanup shutdown step. diff --git a/tests/unit/test_service_entrypoints.py b/tests/unit/test_service_entrypoints.py index cd74ab4a97..2236947d88 100644 --- a/tests/unit/test_service_entrypoints.py +++ b/tests/unit/test_service_entrypoints.py @@ -22,20 +22,34 @@ import ast import importlib +import io import pathlib +import sys import unittest from typing import ClassVar +from unittest import mock + +import yaml from simplyblock_core.services import __main__ as dispatcher +from simplyblock_core.services import task_runners SERVICES_DIR = pathlib.Path(dispatcher.__file__).parent +REPO_ROOT = SERVICES_DIR.parent.parent def _module_path(name): return SERVICES_DIR / f"{name.replace('-', '_')}.py" +def _defines_main(path): + return any( + isinstance(node, ast.FunctionDef) and node.name == "main" + for node in ast.parse(path.read_text()).body + ) + + class TestServiceDispatcher(unittest.TestCase): def test_advertises_at_least_the_known_services(self): @@ -51,24 +65,29 @@ def test_names_are_hyphenated(self): for name in dispatcher._service_names(): self.assertNotIn("_", name, f"{name} should use hyphens, not underscores") - def test_excluded_modules_are_libraries_with_no_entry_point(self): + def test_excluded_libraries_have_no_entry_point(self): """The exclusion list must stay justified: these have no way to be run. If one of them ever grows a ``main()`` it is a service and belongs in the listing, so this fails rather than letting it stay hidden. """ - for name in dispatcher._NOT_SERVICES: + for name in dispatcher._LIBRARIES: with self.subTest(module=name): path = SERVICES_DIR / f"{name}.py" self.assertTrue(path.is_file()) - tree = ast.parse(path.read_text()) - has_main = any( - isinstance(node, ast.FunctionDef) and node.name == "main" - for node in tree.body - ) - self.assertFalse(has_main, f"{name} has main(); it is a service") + self.assertFalse(_defines_main(path), f"{name} has main(); it is a service") self.assertNotIn("__main__", path.read_text()) + def test_excluded_dispatchers_are_entry_points_of_their_own(self): + """The other half of the exclusion: a module kept out of the listing + because it is a command in its own right, not because it is a library. + One that lost its ``main()`` would be excluded for no reason.""" + for name in dispatcher._DISPATCHERS: + with self.subTest(module=name): + path = SERVICES_DIR / f"{name}.py" + self.assertTrue(path.is_file()) + self.assertTrue(_defines_main(path), f"{name} has no main()") + def test_importing_the_package_is_side_effect_free(self): """``runpy`` imports the parent package, which plain ``python3 `` does not necessarily do. ``__init__`` builds ServiceObject instances at @@ -90,7 +109,7 @@ class TestPathInvocationContract(unittest.TestCase): "health_check_service", "lvol_monitor", "lvol_stat_collector", "main_distr_event_collector", "mgmt_node_monitor", "snapshot_monitor", "snapshot_replication", "spdk_http_proxy_server", "storage_node_monitor", - "tasks_cluster_status", "tasks_runner_backup", "tasks_runner_backup_merge", + "tasks_cluster_status", "tasks_runner_backup", "backup_merge_service", "tasks_runner_batch_migration", "tasks_runner_cluster_expand", "tasks_runner_failed_migration", "tasks_runner_fdb_backup", "tasks_runner_jc_comp", "tasks_runner_lvol_migration", @@ -123,7 +142,7 @@ class TestServicesHaveMain(unittest.TestCase): runs it through ``runpy``, which needs no ``main()``. """ - EXEMPT: ClassVar[str] = {"spdk_http_proxy_server"} + EXEMPT: ClassVar[set] = {"spdk_http_proxy_server"} def test_every_service_defines_main(self): for name in dispatcher._service_names(): @@ -131,8 +150,112 @@ def test_every_service_defines_main(self): if module in self.EXEMPT: continue with self.subTest(service=name): - tree = ast.parse(_module_path(name).read_text()) - self.assertTrue( - any(isinstance(n, ast.FunctionDef) and n.name == "main" for n in tree.body), - f"{module} has no top-level main()", - ) + self.assertTrue(_defines_main(_module_path(name)), + f"{module} has no top-level main()") + + +class TestTaskRunnerDispatcher(unittest.TestCase): + """The task runners' own entry point, ``simplyblock-task-runner ``. + + A task runner is fully described by the ``RunnerSpec`` its module exports, so + this dispatcher serves the spec directly rather than executing a module. The + name it takes is ``RunnerSpec.name`` — the deployment surface is the runner's + identity, not a path inside the image. + """ + + def test_advertises_every_task_runner_module(self): + modules = {path.stem for path in SERVICES_DIR.glob("tasks_runner_*.py")} + self.assertEqual( + {name.replace("-", "_") for name in task_runners.runner_names()}, + modules, + ) + + def test_names_are_hyphenated(self): + for name in task_runners.runner_names(): + self.assertNotIn("_", name, f"{name} should use hyphens, not underscores") + + def test_every_advertised_name_loads_a_spec_of_that_name(self): + """``load_spec`` raises on a mismatch; this walks every runner so a + rename on either side is caught here rather than at deploy time.""" + for name in task_runners.runner_names(): + if name in task_runners._NOT_ON_DRIVER: + continue + with self.subTest(runner=name): + self.assertEqual(task_runners.load_spec(name).name, name) + + def test_runners_listed_as_not_on_the_driver_really_export_no_spec(self): + """Keeps the exemption honest: once one of these migrates it exports a + SPEC, and leaving it listed would silently keep it on its own loop.""" + for name in task_runners._NOT_ON_DRIVER: + with self.subTest(runner=name): + module = importlib.import_module( + f"simplyblock_core.services.{name.replace('-', '_')}") + self.assertFalse(hasattr(module, "SPEC"), + f"{name} has a SPEC; drop it from _NOT_ON_DRIVER") + + def test_serves_the_named_spec(self): + with mock.patch.object(task_runners, "serve") as serve, \ + mock.patch.object(sys, "argv", ["simplyblock-task-runner", "tasks-runner-fdb-backup"]): + task_runners.main() + + import simplyblock_core.services.tasks_runner_fdb_backup as runner + serve.assert_called_once_with(runner.SPEC) + + def test_a_runner_not_on_the_driver_is_started_through_its_own_main(self): + import simplyblock_core.services.tasks_runner_lvol_migration as runner + with mock.patch.object(runner, "main") as legacy_main, \ + mock.patch.object(task_runners, "serve") as serve, \ + mock.patch.object(sys, "argv", + ["simplyblock-task-runner", "tasks-runner-lvol-migration"]): + task_runners.main() + + legacy_main.assert_called_once_with() + serve.assert_not_called() + + def test_an_unknown_runner_is_rejected(self): + with mock.patch.object(sys, "argv", ["simplyblock-task-runner", "tasks-runner-nope"]), \ + mock.patch.object(sys, "stderr", io.StringIO()): + with self.assertRaises(SystemExit): + task_runners.main() + + def test_the_console_script_is_declared(self): + pyproject = (REPO_ROOT / "pyproject.toml").read_text() + self.assertIn( + 'simplyblock-task-runner = "simplyblock_core.services.task_runners:main"', + pyproject, + ) + + +class TestDeploymentCommands(unittest.TestCase): + """The consumers this repository owns must start runners by name. + + Everything else — the Helm charts, the operator, Swarm services created by an + older control plane — still uses the paths, which is why + ``TestPathInvocationContract`` above keeps them working. These two files are + the ones a change here can actually fix, so they are pinned. + """ + + COMPOSE = pathlib.Path( + SERVICES_DIR.parent / "scripts" / "docker-compose-swarm.yml") + + def _compose_commands(self): + compose = yaml.safe_load(self.COMPOSE.read_text().replace("$", "_")) + return {name: service.get("command", "") + for name, service in compose["services"].items()} + + def test_every_task_runner_service_uses_the_entry_point(self): + expected = {f"simplyblock-task-runner {name}" + for name in task_runners.runner_names()} + found = {command for command in self._compose_commands().values() + if command.startswith("simplyblock-task-runner ")} + self.assertEqual(found, expected) + + def test_no_compose_service_starts_a_runner_by_path(self): + for name, command in self._compose_commands().items(): + with self.subTest(service=name): + self.assertNotIn("services/tasks_runner_", command) + + def test_cluster_ops_creates_runner_services_by_name(self): + source = (REPO_ROOT / "simplyblock_core" / "cluster_ops.py").read_text() + self.assertNotIn("services/tasks_runner_", source) + self.assertIn('"simplyblock-task-runner"', source) diff --git a/tests/unit/test_task_cancellation.py b/tests/unit/test_task_cancellation.py new file mode 100644 index 0000000000..7946055e23 --- /dev/null +++ b/tests/unit/test_task_cancellation.py @@ -0,0 +1,267 @@ +# coding=utf-8 +"""Unit tests for task cancellation (tasks_controller.cancel_task and +cancel_pending_node_restart_tasks). + +A canceller reads the task, decides, and writes — and in between, the runner +driving that task is writing the same row: claiming its lease, moving it to +running, advancing retry, recording handler progress. A full-object +``write_to_db`` of the canceller's copy puts all of that back. The worst of it +is the owner lease: wiping it hands the task to the next runner host that polls, +which then executes it a second time. That is the same class of lost update as +the 2026-07-29 double restart, in the opposite direction. + +These use a store that models both write paths, so the assertions are about what +ends up committed rather than about the caller's copy. +""" +import copy +from unittest.mock import MagicMock + +import pytest + +from simplyblock_core.controllers import tasks_controller +from simplyblock_core.models.job_schedule import JobSchedule + + +def _task(uuid="task-1", function_name=JobSchedule.FN_NODE_RESTART, + status=JobSchedule.STATUS_NEW, node_id="node-1"): + task = JobSchedule() + task.uuid = uuid + task.cluster_id = "cl-1" + task.node_id = node_id + task.function_name = function_name + task.status = status + task.canceled = False + task.retry = 0 + task.function_params = {} + return task + + +class _Store: + """Task rows plus the two ways they get written. + + ``atomic_update`` follows DBController's contract: the mutator runs against + the row as it exists in the store, a mutator returning False aborts the + write, and the return is the fresh object (or None when the row is gone). + ``full_write`` is the ``write_to_db`` path — a wholesale replace from the + caller's copy. + """ + + def __init__(self, *tasks): + self.kv_store = "KV" + self._rows = {t.uuid: copy.deepcopy(t) for t in tasks} + self._stale = None + + def get_task_by_id(self, task_id): + if self._stale is not None: + return copy.deepcopy(self._stale[task_id]) + if task_id not in self._rows: + raise KeyError(task_id) + return copy.deepcopy(self._rows[task_id]) + + def get_job_tasks(self, cluster_id, reverse=True, limit=0): + rows = self._stale if self._stale is not None else self._rows + return [copy.deepcopy(row) for row in rows.values()] + + def atomic_update(self, obj, mutate): + row = self._rows.get(obj.uuid) + if row is None: + return None + fresh = copy.deepcopy(row) + if mutate(fresh) is False: + return copy.deepcopy(row) + self._rows[obj.uuid] = fresh + return copy.deepcopy(fresh) + + def full_write(self, obj): + self._rows[obj.uuid] = copy.deepcopy(obj) + + def row(self, uuid="task-1"): + return self._rows[uuid] + + def runner_claims(self, uuid="task-1", **fields): + """The runner drives the task after the canceller has read it: pin what + the canceller saw, then advance the row underneath it.""" + if self._stale is None: + self._stale = copy.deepcopy(self._rows) + row = self._rows[uuid] + row.owner = fields.pop("owner", "host-A") + row.status = fields.pop("status", JobSchedule.STATUS_RUNNING) + for name, value in fields.items(): + setattr(row, name, value) + + +@pytest.fixture +def store(monkeypatch): + def _install(*tasks): + s = _Store(*tasks) + monkeypatch.setattr(tasks_controller, "db", s) + monkeypatch.setattr(JobSchedule, "write_to_db", + lambda self, kv=None: s.full_write(self)) + return s + monkeypatch.setattr(tasks_controller, "tasks_events", MagicMock()) + monkeypatch.setattr(tasks_controller, "device_controller", MagicMock()) + return _install + + +# -- cancel_task ------------------------------------------------------------ + +def test_cancel_task_does_not_wipe_the_lease_of_a_running_task(store): + """The lease is what stops a second host from running the task. A cancel + that clears it hands the task straight to the next poller.""" + s = store(_task()) + s.runner_claims(owner="host-A", retry=1) + + assert tasks_controller.cancel_task("task-1") is True + + row = s.row() + assert row.canceled is True + assert row.owner == "host-A" + assert row.status == JobSchedule.STATUS_RUNNING + assert row.retry == 1 + + +def test_cancel_task_does_not_revert_a_finished_task(store): + s = store(_task()) + s.runner_claims(owner="host-A", status=JobSchedule.STATUS_DONE, + function_result="Node is online") + + tasks_controller.cancel_task("task-1") + + row = s.row() + assert row.status == JobSchedule.STATUS_DONE + assert row.function_result == "Node is online" + + +def test_cancel_task_does_not_lose_handler_progress(store): + """function_params carries multi-cycle progress (recovery_started, + merge_started); reverting it makes the next attempt re-issue the RPC.""" + s = store(_task(function_name=JobSchedule.FN_BACKUP_RESTORE)) + s.runner_claims(owner="host-A", function_params={"recovery_started": True}) + + tasks_controller.cancel_task("task-1") + + assert s.row().function_params == {"recovery_started": True} + assert s.row().canceled is True + + +def test_cancel_task_flags_and_emits_the_event(store): + s = store(_task()) + + assert tasks_controller.cancel_task("task-1") is True + + assert s.row().canceled is True + tasks_controller.tasks_events.task_canceled.assert_called_once() + + +def test_cancel_task_is_idempotent(store): + s = store(_task()) + tasks_controller.cancel_task("task-1") + tasks_controller.tasks_events.task_canceled.reset_mock() + + assert tasks_controller.cancel_task("task-1") is True + + assert s.row().canceled is True + tasks_controller.tasks_events.task_canceled.assert_not_called() + + +def test_cancel_task_refuses_a_master_task(store): + task = _task() + task.sub_tasks = ["sub-1"] + s = store(task) + + assert tasks_controller.cancel_task("task-1") is False + assert s.row().canceled is False + + +# -- cancel_pending_node_restart_tasks -------------------------------------- + +def test_cancel_pending_restart_does_not_revert_concurrent_progress(store): + """Called from set_node_status the moment a node goes ONLINE, off a bulk + read — so its copies are stale by construction.""" + s = store(_task()) + s.runner_claims(owner="host-A", retry=3) + + assert tasks_controller.cancel_pending_node_restart_tasks("cl-1", "node-1") == 1 + + row = s.row() + assert row.canceled is True + assert row.status == JobSchedule.STATUS_DONE + assert row.function_result == "canceled: node back online" + assert row.owner == "host-A" + assert row.retry == 3 + + +def test_cancel_pending_restart_leaves_a_finished_task_alone(store): + """A task that reached its own outcome between the bulk read and the write + keeps that outcome — the cancellation is moot.""" + s = store(_task()) + s.runner_claims(status=JobSchedule.STATUS_DONE, + function_result="Node is online") + + assert tasks_controller.cancel_pending_node_restart_tasks("cl-1", "node-1") == 0 + + row = s.row() + assert row.function_result == "Node is online" + assert row.canceled is False + + +def test_cancel_pending_restart_skips_other_nodes_and_functions(store): + s = store( + _task(uuid="task-1", node_id="node-1"), + _task(uuid="task-2", node_id="node-2"), + _task(uuid="task-3", node_id="node-1", function_name=JobSchedule.FN_NODE_ADD), + ) + + assert tasks_controller.cancel_pending_node_restart_tasks("cl-1", "node-1") == 1 + + assert s.row("task-1").canceled is True + assert s.row("task-2").canceled is False + assert s.row("task-3").canceled is False + + +# -- cancel_node_tasks ------------------------------------------------------ + +def test_cancel_node_tasks_does_not_wipe_the_lease(store): + """Node shutdown cancels the migration tasks queued against the node, off + the same kind of bulk read.""" + s = store(_task(function_name=JobSchedule.FN_DEV_MIG)) + s.runner_claims(owner="host-A", retry=2) + + assert tasks_controller.cancel_node_tasks( + "cl-1", "node-1", [JobSchedule.FN_DEV_MIG]) == 1 + + row = s.row() + assert row.canceled is True + assert row.owner == "host-A" + assert row.status == JobSchedule.STATUS_RUNNING + assert row.retry == 2 + + +def test_cancel_node_tasks_leaves_a_finished_task_alone(store): + s = store(_task(function_name=JobSchedule.FN_DEV_MIG)) + s.runner_claims(status=JobSchedule.STATUS_DONE, function_result="migrated") + + assert tasks_controller.cancel_node_tasks( + "cl-1", "node-1", [JobSchedule.FN_DEV_MIG]) == 0 + + assert s.row().canceled is False + assert s.row().function_result == "migrated" + + +def test_cancel_node_tasks_skips_other_nodes_and_functions(store): + s = store( + _task(uuid="task-1", node_id="node-1", function_name=JobSchedule.FN_DEV_MIG), + _task(uuid="task-2", node_id="node-1", function_name=JobSchedule.FN_NEW_DEV_MIG), + _task(uuid="task-3", node_id="node-2", function_name=JobSchedule.FN_DEV_MIG), + _task(uuid="task-4", node_id="node-1", function_name=JobSchedule.FN_NODE_RESTART), + ) + + assert tasks_controller.cancel_node_tasks( + "cl-1", "node-1", + [JobSchedule.FN_DEV_MIG, JobSchedule.FN_FAILED_DEV_MIG, + JobSchedule.FN_NEW_DEV_MIG]) == 2 + + assert s.row("task-1").canceled is True + assert s.row("task-2").canceled is True + assert s.row("task-3").canceled is False + assert s.row("task-4").canceled is False