campaigns.org

Campaign state and typed findings

A campaign turns a resolved package bundle into empirical build and binary evidence on one named target. The JSON state is durable, resumable, and the coordination surface for Hermes and OMP workers.

Hermes and OMP are role names rather than required orchestration software. Hermes is the single campaign owner that classifies findings, applies or coordinates repairs, and retries through the requested claim rung. OMP workers are optional concurrent participants; their only shared coordination surface is finding ownership in the campaign state.

Run and inspect

eb-stack campaign run \
  --bundle BUNDLE \
  --config examples/targets/base.toml \
  --config examples/targets/site.example.toml \
  --target NAME \
  --state /tmp/campaign.json

eb-stack campaign status --state /tmp/campaign.json

campaign run is synchronous and stays in the foreground until every recipe and verification command finishes or a typed failure is recorded. Run it under the target site’s terminal or service supervisor; campaign status remains available from another process.

The state file has one writer while a run, claim, or resolution mutates it. An OS-backed advisory guard enforces exclusivity, and the visible .lock file records host, PID, and Linux process-start identity for diagnosis. The guard is released by the operating system if the controlling process exits, so stale metadata cannot permanently block a retry. Before retrying an interrupted run, confirm that the routed container, scheduler job, or host command is no longer active; a remote workload can outlive its controller. Reuse the same state path. Package name and version must match, each run increments attempts, history and findings remain available, and build/binary claims reset until the current attempt proves them.

campaign status prints the complete audit record. Typed failures retain their exact routed command, exit status, and compact evidence, including after their findings are resolved or superseded. For a compact operator view during a long build:

eb-stack campaign status --state /tmp/campaign.json |
  jq '{status, attempts, claims, current_recipe,
       active_findings: [.findings[] |
         select(.status == "open" or .status == "in-progress") |
         {id, class, stage, recipe, summary, owner}]}'

The campaign writes status: running and current_recipe before launching EasyBuild, so another shell can observe those fields while the build command is active. Keep the unfiltered state for diagnosis and handoff evidence.

State schema

CAMPAIGN.json has schema_version = 1:

{
  "schema_version": 1,
  "package": "QMCPACK",
  "version": "4.3.0",
  "bundle": "/work/qmcpack",
  "target": "site-builder",
  "status": "failed",
  "attempts": 2,
  "claims": {
    "resolves": true,
    "builds": false,
    "binary_verified": false
  },
  "findings": [],
  "history": []
}

Status is planned, running, failed, or completed. current_recipe is present while an EasyBuild recipe is active and omitted otherwise. History entries record the attempt, the status, an optional recipe path, and a compact detail string.

Claim ladder

The three booleans are independent claims:

1. resolves: every requested profile has a Resolvo lock and emitted recipe;
2. builds: every emitted recipe succeeds through EasyBuild on this target;
3. binary_verified (prose: binary-verified): at least one verification command is declared and every declared command succeeds. See the glossary.

A completed campaign with no verification commands has builds: true and binary_verified: false. A planned Software Bill of Materials (SBOM) never implies either build claim.

Finding schema

A terminal failure appends a finding containing:

Field	Meaning
id	Stable attempt:N:finding:M identity
class	Error-domain classification
disposition	Mechanical, retryable, judgment, or target repair
stage	preflight, stage, build, or verify
recipe	Relative recipe path or verification profile
target	Routed target name
summary / evidence	Compact diagnosis and terminal evidence
command / exit_code	Exact routed program, arguments, and exit status
attempt	Attempt that observed the failure
status / owner	Queue state and exclusive worker owner
resolution	Action, evidence, and changed paths

Evidence contains the tail of routed stdout/stderr. When EasyBuild reports a separate combined-output path, the campaign interns the nested log before classification so the finding reflects the actual compiler or runtime error.

Classes, dispositions, and repair surfaces

Class	Default disposition	Start at
transport	target-repair	SSH, rsync, host reachability, or staging path
executor	target-repair	Slurm command, partition, account, or allocation syntax
runtime	target-repair	Host/container executable, image, mounts, ABI, or environment
interrupted	retryable	Abandoned running state after the exclusive controller lock becomes available
source	requires-judgment	Upstream URL, mirror, source cache, or runtime network path
checksum	mechanical	Source digest and positional source/patch checksum list
patch	requires-judgment	Patch source, order, and applicability
dependency-missing	requires-judgment	Manifest, profile, stack policy, or real dependency recipe
configure	requires-judgment	Product options, detected dependency, toolchain, or patch
compile	requires-judgment	Source compatibility, compiler, flags, dependency, or patch
link	requires-judgment	Link interface, library order, ABI, or dependency selection
test	requires-judgment	Product behavior, test environment, or test-specific dependency
install	requires-judgment	EasyBuild install logic, paths, or permissions
license	requires-judgment	The site accepts or declines the licence (--accept-eula-for); never a recipe repair
sanity	requires-judgment	Sanity paths or profile verification command
resource	retryable	CPU, memory, GPU, filesystem, or scheduler sizing
timeout	retryable	Routed timeout or scheduler wall time
unknown	requires-judgment	Unclassified evidence; inspect the exact command and nested log

The class narrows the first repair surface; it does not replace diagnosis. For example, a SAT-compatible dependency that fails to compile remains a compile finding until evidence supports a stack exclusion or package patch. A compiler killed by the kernel, virtual memory exhausted, or equivalent allocation evidence is instead a resource finding. Preserve the selected dependency and recipe while correcting EASYBUILD_PARALLEL or the scheduler allocation; only source-level compiler diagnostics justify a compile repair.

Ownership protocol

Claim an open finding before changing state shared with another worker:

eb-stack campaign finding claim \
  --state /tmp/campaign.json \
  --id attempt:2:finding:1 \
  --owner omp-worker-1

Claims are exclusive. Reclaiming with the same owner is idempotent; a different owner receives an ownership error.

Record the repair and its evidence:

eb-stack campaign finding resolve \
  --state /tmp/campaign.json \
  --id attempt:2:finding:1 \
  --owner omp-worker-1 \
  --action "corrected the container runtime environment" \
  --evidence "target doctor and the isolated command exit successfully" \
  --change examples/targets/site.example.toml

Run the campaign again with the same state. A successful retry marks matching open or in-progress findings as superseded while retaining their evidence. Resolved findings remain part of the audit trail.

Repair loop

The task-oriented campaign repair guide provides copyable commands and a class-to-artifact troubleshooting table. The protocol is:

1. Run target doctor and the campaign.
2. Read the finding’s stage, class, exact command, evidence, and nested log.
3. Reproduce the smallest failing command on the same target/runtime.
4. Claim the finding.
5. Change the layer supported by the evidence.
6. Run the nearest mechanical check: recipe lint/check, target doctor, or the isolated build command.
7. Resolve the finding with evidence and changed paths.
8. Rerun the full campaign state until every requested recipe and verification command succeeds.

Hermes owns this whole loop. OMP workers use finding ownership to divide residual diagnosis without racing on the state file. The executable procedures are new package and annual bump.