docs: add inventory file for multi-server-devnet skill

.claude/skills/multi-server-devnet/SKILL.md

@@ -40,22 +40,48 @@ B owns `k+1..N-1`). Everything below still applies with three changes, and
 
 Nothing about the servers is hardcoded. Establish these from the operator:
 
-- `SERVERS` — the SSH targets (any count, any names), one devnet each.
-- `SSH_USER` — login user. `docker` is invoked with `sudo`.
+- `SERVERS`: the SSH targets for the operation at hand (any count, any names),
+  one devnet each. A working set taken FROM the inventory below, not a second
+  list of hosts to keep in step with it.
+- `SSH_USER`: login user. `docker` is invoked with `sudo`.
 - Per-devnet `NODES` (validators on that server) and `SUBNETS`
   (`ATTESTATION_COMMITTEE_COUNT`); these can differ between servers.
 - A central host for Grafana + the federating Prometheus (often one of the
   servers); each server's per-host Prometheus remote-writes to it.
 
-Put those values in `scripts/devnet.env` (copy `scripts/devnet.env.example`;
-gitignored) instead of retyping them: the operator-side scripts source it via
+`SERVERS`/`SSH_USER` are the caller's to pass per command: no script reads them, so
+which hosts a command runs against is decided at the call rather than stored. The
+rest goes in `scripts/devnet.env` (copy `scripts/devnet.env.example`; gitignored)
+instead of being retyped: the operator-side scripts source it via
 `scripts/devnet-env.sh`, and an env var exported in the shell still wins over the
-file. It is the one place a deployment's hosts, urls, and Grafana ids live.
-Per-devnet `NODES`/`SUBNETS` are recorded there too, but as the operator's
-inventory — the scripts take them as positional args, and the authority on a
-running devnet is always its own `genesis/config.yaml`
-(`ATTESTATION_COMMITTEE_COUNT`), which `start-devnet.sh` and `convert.sh` check
-against.
+file. It is the one place a deployment's urls and Grafana ids live.
+
+**The servers themselves live in `scripts/devnet.inventory`** (copy
+`scripts/devnet.inventory.example`; gitignored): name, ip, tags, and per-devnet
+`NODES`/`SUBNETS`, one row per host. Query it with `scripts/inventory.sh` rather
+than reading it by hand:
+
+```bash
+bash scripts/inventory.sh --tag devnet-ab --field ip        # ips, for a loop
+bash scripts/inventory.sh --tag devnet-ab --tag aggregator  # AND across tags
+bash scripts/inventory.sh --tag validator                   # derived, see below
+SERVERS=$(bash scripts/inventory.sh --field name) || exit   # exit 2 = typo'd tag
+SERVERS=${SERVERS//$'\n'/ }                                 # newlines -> spaces
+```
+
+Tag conventions: a `devnet-*` tag names the chain a host's nodes belong to (two
+hosts sharing one means the split-chain model, *not* two like-named devnets), and
+`aggregator` is a whole-server role. `validator` is **derived**: "has a `devnet-*`
+tag and is not tagged `aggregator`", so it can never disagree with the aggregator
+tag, and writing it literally in the file is an error rather than a second opinion.
+An unknown or empty `--tag` exits 2 rather than returning nothing (or, worse,
+everything): a typo that yields an empty loop reports success while doing nothing,
+and `--tag "$UNSET"` would act on hosts the caller never named.
+
+`NODES`/`SUBNETS` there are still the operator's inventory: the scripts take them
+as positional args, and the authority on a running devnet is always its own
+`genesis/config.yaml` (`ATTESTATION_COMMITTEE_COUNT`), which `start-devnet.sh` and
+`convert.sh` check against.
 
 Per devnet, node `n` (0 ≤ n < NODES) on its host:
 
@@ -130,7 +156,8 @@ redirect must run under sudo).
 ## Workflows
 
 Examples assume `SSH_USER` is set and you iterate over `SERVERS`. Per server you
-pass its own `NODES`/`SUBNETS`.
+pass its own `NODES`/`SUBNETS`. `for h in $SERVERS` splits under bash; zsh does not
+split parameter expansions, so there write `for h in $(echo $SERVERS)`.
 
 ### Pull the latest images on all servers
 ```bash
@@ -473,7 +500,8 @@ Swap (persistent): `fallocate -l 16G /swapfile && chmod 600 && mkswap && swapon`
 | `promtail-config.sh` | operator | `NETWORK HOST_IP LOKI_PUSH_URL [N:client ...]` | Emit a per-host promtail.yml (docker_sd → central Loki, labels mirror prometheus, backlog guard) |
 | `sweep.sh` | operator | `CENTRAL_PROM_URL` | Cross-devnet audit: head/justified/finalized + client mix from the central Prometheus |
 | `deploy-finality-alert.sh` | operator | `[WEBHOOK_FILE]`; `METRICS_HOST`, `GRAFANA_*`, `PROM_DS_UID`; `DRY_RUN` | Render + ship the "lost finality" Slack alert to the central Grafana |
-| `devnet-env.sh` / `devnet.env.example` | operator | `$DEVNET_ENV`, `./devnet.env`, script dir | Load this deployment's hosts/urls/Grafana ids as defaults; exported vars win. Copy the example to `devnet.env` (gitignored) once |
+| `devnet-env.sh` / `devnet.env.example` | operator | `$DEVNET_ENV`, `./devnet.env`, script dir | Load this deployment's urls/Grafana ids as defaults; exported vars win. Copy the example to `devnet.env` (gitignored) once |
+| `inventory.sh` / `devnet.inventory.example` | operator | `[--tag T]... [--not-tag T]... [--field F] [--file P] [--count]`; `$DEVNET_INVENTORY`, `./devnet.inventory`, script dir | Select servers by tag from the fleet inventory (name, ip, tags, NODES, SUBNETS). `devnet-*` tags name the chain, `aggregator` is a server role, `validator` is derived (`devnet-*` and not `aggregator`). Unknown tag → exit 2 + known-tag list, so a typo can't masquerade as an empty fleet. Copy the example to `devnet.inventory` (gitignored) once |
 
 **Grafana dashboards** (copy into the central Grafana's dashboards dir —
 `GRAFANA_DASHBOARDS_DIR`, *not* the provisioning tree; they auto-load in ~30s and

.claude/skills/multi-server-devnet/scripts/devnet-env.sh

@@ -4,6 +4,9 @@
 #
 #   . "$SCRIPT_DIR/devnet-env.sh"; devnet_load_env
 #
+# Also provides devnet_find_file, the file-lookup ladder both this and
+# inventory.sh use.
+#
 # Lookup order, first hit wins: $DEVNET_ENV, ./devnet.env, <scripts dir>/devnet.env.
 # Copy devnet.env.example -> devnet.env and fill it in (devnet.env is gitignored,
 # since it names your hosts and may point at a webhook file).
@@ -17,21 +20,32 @@
 # fine. A line that looks like an assignment but whose name isn't usable is
 # reported on stderr rather than dropped in silence, because a config line that
 # goes unread is how you deploy against the wrong deployment.
-devnet_load_env() {
-  local dir file line key val
+
+# Resolve one of the deployment's config files. First hit wins: the path in $2 (an
+# env var NAME), ./<name>, <scripts dir>/<name>. Prints the path on stdout.
+# Returns 1 when nothing exists, and 2 when the env var names a path that doesn't
+# -- an explicit path that isn't there is a typo, not a licence to fall back to
+# another deployment's file. Shared with inventory.sh so devnet.env and
+# devnet.inventory can't end up with two ideas of where they live.
+devnet_find_file() {
+  local name=$1 envvar=$2 dir explicit candidate
   dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-  file=""
-  if [ -n "${DEVNET_ENV:-}" ]; then
-    # An explicit path that doesn't exist is a typo, not a reason to silently fall
-    # back to some other devnet.env and deploy against the wrong deployment.
-    [ -f "$DEVNET_ENV" ] || { echo "DEVNET_ENV=$DEVNET_ENV does not exist" >&2; return 1; }
-    file=$DEVNET_ENV
-  else
-    for candidate in "./devnet.env" "$dir/devnet.env"; do
-      [ -f "$candidate" ] && { file=$candidate; break; }
-    done
+  explicit=${!envvar:-}
+  if [ -n "$explicit" ]; then
+    [ -f "$explicit" ] || { echo "$envvar=$explicit does not exist" >&2; return 2; }
+    printf '%s\n' "$explicit"; return 0
   fi
-  [ -n "$file" ] || return 0
+  for candidate in "./$name" "$dir/$name"; do
+    [ -f "$candidate" ] && { printf '%s\n' "$candidate"; return 0; }
+  done
+  return 1
+}
+
+devnet_load_env() {
+  local file line key val rc
+  file=$(devnet_find_file devnet.env DEVNET_ENV); rc=$?
+  [ "$rc" -eq 2 ] && return 1        # DEVNET_ENV names a missing file: reported
+  [ "$rc" -eq 0 ] || return 0        # no env file at all is not an error
 
   while IFS= read -r line || [ -n "$line" ]; do
     line=${line#"${line%%[![:space:]]*}"}          # ltrim, so an indented line is read

.claude/skills/multi-server-devnet/scripts/devnet.env.example

@@ -4,15 +4,9 @@
 # Loaded by the operator-side scripts via devnet-env.sh; an env var already set in
 # your shell overrides the value here.
 
-# --- fleet (one independent devnet per server) -------------------------------
-SERVERS="host-a host-b"                 # ssh targets, any count/names
-SSH_USER=user                           # login user; docker is invoked with sudo
-
-# Per-devnet size, as INVENTORY: the scripts take NODES/SUBNETS as positional
-# args, and the authority for a running devnet is its own genesis/config.yaml
-# (ATTESTATION_COMMITTEE_COUNT). Recorded here so a restart doesn't have to guess.
-#   host-a: NODES=32 SUBNETS=4
-#   host-b: NODES=16 SUBNETS=2
+# Only what a script actually reads lives here. SERVERS/SSH_USER don't: which hosts
+# a command runs against is the caller's to pass per invocation, and which hosts
+# EXIST is devnet.inventory's record, queried with scripts/inventory.sh.
 
 # --- central metrics / logs stack -------------------------------------------
 # sweep.sh reads CENTRAL_PROM_URL; the other two are args you pass to

.claude/skills/multi-server-devnet/scripts/devnet.inventory.example

@@ -0,0 +1,52 @@
+# The deployment's servers: what each one is called, where it is, what it belongs
+# to, and how big its devnet is. Copy to devnet.inventory (gitignored) and fill in.
+#
+# This is the operator's inventory, in the same sense devnet.env's values are: the
+# authority on a RUNNING devnet is always its own genesis/config.yaml
+# (ATTESTATION_COMMITTEE_COUNT), and the scripts still take NODES/SUBNETS as
+# positional args. Recorded here so a restart doesn't have to guess, and so
+# "which hosts are in devnet-c" has one answer instead of living in someone's head.
+#
+# Read it with inventory.sh rather than by hand:
+#
+#   bash scripts/inventory.sh --tag devnet-ab --field ip        # ips, for a loop
+#   bash scripts/inventory.sh --tag devnet-ab --tag aggregator  # AND: both needed
+#   bash scripts/inventory.sh --tag validator                   # derived, see TAGS
+#   SERVERS=$(bash scripts/inventory.sh --field name) || exit   # exit 2 = typo'd tag
+#   SERVERS=${SERVERS//$'\n'/ }                                 # newlines -> spaces
+#
+# FORMAT: whitespace-separated columns, aligned however you like. Blank lines and
+# lines whose first non-space character is '#' are ignored. Five columns:
+#
+#   name      ssh target / display name. Need not equal the machine's hostname,
+#             and often doesn't after a host is repurposed -- inventory.sh never
+#             resolves it, so a stale name here misroutes every ssh that uses it.
+#   ip        address. The one field worth re-deriving from the fleet rather than
+#             trusting after a rebuild.
+#   tags      COMMA-separated, NO spaces (a space would start the next column;
+#             inventory.sh rejects the file rather than answer from shifted rows).
+#   nodes     NODES on this server, or '-' if it runs no devnet.
+#   subnets   ATTESTATION_COMMITTEE_COUNT, or '-'.
+#
+# TAGS: free-form, with two conventions inventory.sh knows about:
+#
+#   devnet-*    names the chain this server's nodes belong to. Two servers sharing
+#               one tag means they share one genesis, i.e. the split-chain model
+#               (nodes 0..N on one, N+1.. on the other) -- NOT two devnets that
+#               happen to be named alike. Also the `network` Prometheus label.
+#   aggregator  this server runs the aggregator duty. A whole-server role, not a
+#               per-node one.
+#
+#   `validator` is DERIVED, never written here: it means "has a devnet-* tag and
+#   is not tagged aggregator". Writing it as a literal tag would let it disagree
+#   with the aggregator column, so inventory.sh computes it instead.
+#
+# The example below shows both supported topologies at once: host-a + host-b share
+# the chain `devnet-ab` (64 nodes total, 32 each), host-c runs its own independent
+# `devnet-c`, and metrics-1 carries the Grafana/Prometheus/Loki stack and no nodes.
+
+# name        ip              tags                    nodes  subnets
+host-a        10.0.0.1        devnet-ab,aggregator    32     8
+host-b        10.0.0.2        devnet-ab               32     8
+host-c        10.0.0.3        devnet-c                16     2
+metrics-1     10.0.0.4        tooling                 -      -