Skip to content

Modify URM Test Runner #585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Kartik Nema (kartnema) wants to merge 1 commit into
base: main
Choose a base branch
from
+360 −48
Open

Modify URM Test Runner #585

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 56 additions & 11 deletions Runner/suites/Performance/userspace-resource-manager/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ Only these binaries are executed, in this order (anything else is ignored):

## Gating policy

* **Service check (early gate):** If `urm.service` is **not active**, the test **SKIPs overall** and exits.
* **Service check (early gate):** The selected service (`SERVICE_NAME`, default `urm.service`) must be applicable and active. A genuinely absent/non-applicable service is **overall SKIP**; an applicable service that cannot be started/restarted is **overall FAIL**.
* **Required service restart:** Before the first runnable suite only, the runner restarts the selected service so it observes the staged test nodes. Restart failure or failure to become active is **overall FAIL** with service evidence retained in the log directory.
* **Per‑suite SKIP conditions (neutral):**
* Missing binary → **SKIP that suite**, continue.
* Missing base configs → **SKIP that suite**, continue.
Expand All @@ -33,9 +34,14 @@ Only these binaries are executed, in this order (anything else is ignored):
## Pre‑checks

### 1) Service
The runner uses the repo helper `check_systemd_services()` to verify **`urm.service`** is active.
- On failure: overall **SKIP** (ends early).
- Override service name: `SERVICE_NAME=your.service ./run.sh`
The runner uses the repo helper `check_systemd_services()` to verify the selected systemd unit is active. The default is **`urm.service`** and it can be overridden with `SERVICE_NAME=your.service ./run.sh`.

Runtime contract:
- If the selected service is genuinely absent/non-applicable → overall **SKIP**.
- If the selected service is applicable but cannot be started → overall **FAIL**.
- Before the first runnable suite only, the runner performs a required `systemctl restart "$SERVICE_NAME"`.
- If the required restart fails, or the service does not become active within the bounded wait → overall **FAIL**.
- Restart diagnostics are retained under the run log directory, for example `service_restart.log`, `service_restart_status.log`, and when available `service_restart_journal.log`.

### 2) Config presence
Suites that parse configs require **all** of these base config trees:
Expand Down Expand Up @@ -82,15 +88,54 @@ Usage: ./run.sh [--all] [--bin <name|absolute>] [--list] [--timeout SECS]
- `--all` (default): run all approved suites.
- `--bin NAME|PATH`: run a single approved suite.
- `--list`: print approved list and presence coverage, then exit.
- `--timeout SECS`: default per‑binary timeout **if** `run_with_timeout()` helper exists (else ignored).
- `--timeout SECS`: default per‑binary timeout for suites without a suite-specific timeout.

Per‑suite default timeouts (if helper is present):
Per‑suite default timeouts enforced by this runner's local timeout helper:
- `UrmComponentTests`: **1800s**
- `UrmIntegrationTests`: **2400s**
- others: **1200s** (default)

---

## Timeout helper contract

`run.sh` uses a local timeout helper for URM suites instead of depending on the shared `run_with_timeout()` helper. This is intentional: the local helper closes the lock FD before launching the test command, timeout watcher, and watcher `sleep`, preventing background timeout processes from retaining the flock after the parent exits.

Function contract for `run_cmd_with_timeout_no_lock_fd TIMEOUT_SECS COMMAND [ARG...]`:

- **Arguments:**
- `TIMEOUT_SECS`: positive integer enables timeout enforcement.
- Empty, zero, or non-numeric timeout runs `COMMAND` directly without spawning a watcher.
- `COMMAND [ARG...]`: binary and arguments to execute.
- **Return statuses:**
- command exit status on normal completion.
- `124` when this helper's own watcher expires the deadline.
- other signal-derived statuses are preserved when they were not caused by this helper's timeout watcher.
- **Spawned processes when timeout is enabled:**
- one command process,
- one watcher subshell,
- one watcher `sleep` process.
- **Retained files:**
- temporary watcher sleep PID and timeout marker files are created under the run log directory while the command is active.
- these files are removed before the helper returns; the shared `cleanup()` path owns removal on interruption or termination.
- **Timeout logging:**
- when the watcher expires, the suite log records `[TIMEOUT] command exceeded <seconds>s: <command>`.
- `run_one()` reports this as `TIMEOUT` / `FAIL` instead of ambiguous `UNKNOWN RC=143` or `UNKNOWN RC=137`.

---

## Locking and diagnostics

The runner keeps `flock` as the primary concurrency guard because URM tests and the service use shared system resources. Cleanup explicitly unlocks and closes FD 9. If `flock` is unavailable, a simple `mkdir` lock directory fallback is used and removed during cleanup.

When `flock` acquisition fails, diagnostics avoid broad `run.sh` process-name matching and prefer:

- `lslocks` entries for `/tmp/userspace-resource-manager.lock`,
- `fuser` output for processes with the lock file open,
- `/proc/*/fd` references to the lock file when available.

---

## Output layout

- **Overall status file:** `./userspace-resource-manager.res` → `PASS` / `FAIL` / `SKIP`
Expand All @@ -99,11 +144,9 @@ Per‑suite default timeouts (if helper is present):
- Per‑suite result markers: `SUITE.res` (`PASS`/`FAIL`/`SKIP`)
- Coverage summaries: `coverage.txt`, `missing_bins.txt`, `coverage_counts.env`
- System snapshot: `dmesg_snapshot.log`
- Service restart evidence when applicable: `service_restart.log`, `service_restart_status.log`, `service_restart_journal.log`
- **Symlink to latest:** `./logs/userspace-resource-manager-latest`

**Parsing heuristics:** a suite is considered PASS if the binary exits 0 **or** its log contains
`Run Successful`, `executed successfully`, or `Ran Successfully`. Strings like `Assertion failed`, `Terminating Suite`, `Segmentation fault`, `Backtrace`, or `fail/failed` mark **FAIL**.

---

## Environment overrides
Expand Down Expand Up @@ -159,11 +202,13 @@ The script writes the overall result to `userspace-resource-manager.res`. The **

## Troubleshooting

- **Overall SKIP immediately** → service inactive. Check `systemctl status urm.service`.
- **Overall SKIP immediately** → selected service appears absent/non-applicable, or no suite is runnable.
- **Overall FAIL during service setup/restart** → selected service is applicable but failed to start/restart; inspect `logs/.../service_*` files.
- **Suite SKIP (config)** → confirm required files exist under `common/`, `tests/configs` and `tests/nodes` (see lists above).
- **Suite SKIP (missing bin)** → verify the binary is installed and executable under `/usr/bin`.
- **Suite FAIL** → inspect `logs/.../SUITE.log` for the first failure pattern or assertion.
- **Very long runs** → a `run_with_timeout` helper (if available in your repo toolchain) will be used automatically.
- **Very long runs** → the local timeout helper enforces the configured per-suite deadline and reports helper-expired timeouts as `TIMEOUT` / rc `124`.
- **Unexpected concurrent-run SKIP** → inspect the emitted lock diagnostics (`lslocks`, `fuser`, `/proc/*/fd`) to identify any real lock holder.

## License
- SPDX-License-Identifier: BSD-3-Clause
Expand Down
Loading
Loading