From 904774f2a3a58e47de4f9b482323ebad58b412b6 Mon Sep 17 00:00:00 2001 From: Andreas Date: Wed, 2 Sep 2026 06:28:01 +1200 Subject: [PATCH 1/3] add the assay_env role: fingerprint capture to interchange file Nothing wrote /etc/assay/environment.json, so every machine this repo built was unfingerprinted as far as assay was concerned - env@none on every run, and results with no identity for the conditions that produced them. The capture existed (scripts/vm-fingerprint.sh) and the contract existed (assay environments/README.md); the two were not connected. assay_env is the layer-2 capstone. It installs nothing. It runs the existing capture script from the CONTROL node (delegate_to: localhost, become: false - the script SSHes in, so re-implementing it on the guest would fork it), parses the DIGESTS block, and writes the interchange file the contract specifies: schema as the number 1, core_digest and provider_digest as sha256:<64 hex>, plus provider and definition as context. Three things it does not do, deliberately: - It never holds the capture as content. Given an outfile the script prints only `fingerprint written to ` plus tail -4 on stderr, and that tail is what is parsed - the body carries the guest's authorized_keys and sshd drop-ins. The role refuses to run at all if assay_env_capture_path is empty, because without an outfile the whole capture would arrive on stdout and land in ansible's registered output. - It never writes a partial file. The parse keeps only lines matching ^(core|provider|combined)\s+sha256:[0-9a-f]{64}$, so ssh noise cannot be mistaken for a digest and a truncated hash is a non-match rather than a match on a truncated hash. Missing any of the three fails before the first write: a file naming a digest that was never computed is worse than no file, because assay believes it. - It never moves the digest it records. CORE reads enumerated /etc paths (os-release, three apt.conf.d files, the Types/Suites/Components/Snapshot lines of sources.list.d/*.sources, sshd_config.d/*.conf, docker/daemon.json) and the ~/.local and ~/.bun trees. It does not walk /etc, so /etc/assay is outside it. file and copy are the only modules touching the target, and they touch nothing else - no package, no unit, no user, no hostname. provider is hardcoded to proxmox-ve with a comment rather than read from a hostvar: inventory/tofu.py builds hostvars from the vm-pve module's output and that is the only provider in this tree. When a second one lands it belongs in the tofu output and this default becomes the fallback. Wired into the inventory example as the last role, with the reason: anything listed after it is software the recorded identity does not describe. README's "docker - the only role using become" was true and no longer is. Verified on the control node: ansible-lint clean at the production profile; site.yaml syntax-check clean; a stubbed capture with a truncated core digest refuses at "Refuse to write a partial environment file" with nothing written, a well-formed one passes every guard, and `grep -c SECRET` over a -vvv run is 0 with the capture task's stdout empty. Not yet run against a live guest - that is the acceptance run. Refs the-metafactory/crucible#25 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_018PtJarXm8d8bSHDbhmskVm --- README.md | 24 +++- ansible/roles/assay_env/defaults/main.yaml | 36 ++++++ ansible/roles/assay_env/tasks/main.yaml | 142 +++++++++++++++++++++ inventory-example.yaml | 7 +- 4 files changed, 202 insertions(+), 7 deletions(-) create mode 100644 ansible/roles/assay_env/defaults/main.yaml create mode 100644 ansible/roles/assay_env/tasks/main.yaml diff --git a/README.md b/README.md index 453f727..3ff82e1 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,8 @@ overall testing workflow. false. Third-party repos (e.g. docker) are covered by explicit version pins in the ansible role defaults instead. - **Composable post-boot software.** A spec lists ansible roles, for example, - (`ansible_roles: [nats_server, bun, claude, docker, metafactory_arc]`) and + (`ansible_roles: [nats_server, bun, claude, docker, metafactory_arc, + assay_env]`) and `ansible-playbook ansible/site.yaml` applies them — idempotently, with every download verified and every version pinned. A dynamic inventory reads the tofu state, so there is no hosts file to maintain, and changing ansible @@ -73,7 +74,8 @@ overall testing workflow. │ ├── site.yaml # base, then each VM's declared roles │ ├── inventory/tofu.py # dynamic inventory from tofu output │ └── roles// # base (every VM), then nats_server, bun, -│ # claude, docker, metafactory_arc +│ # claude, docker, metafactory_arc, +│ # assay_env (last: fingerprint + interchange) ├── inventory/ # one YAML file per VM [EDIT] ├── scripts/ # build-image.sh, check-ansible.sh, │ # install-collections.sh, vm-fingerprint.sh @@ -322,10 +324,20 @@ declarative grammar as the rest of the repo: `claude` (vendor installer, binary verified against the release manifest first), `metafactory_arc` (pinned git tag + `bun install`/`bun link`; needs `git` in the VM's packages, and `bun` earlier in the role list — spec order - is application order), and `docker` — the only role using `become` — which - pins the signing key by full GPG fingerprint, writes a deb822 source, - installs version-pinned packages, and manages `daemon.json` with a restart - handler. + is application order), and `docker` — which pins the signing key by full GPG + fingerprint, writes a deb822 source, installs version-pinned packages, and + manages `daemon.json` with a restart handler. +- **`assay_env` is the capstone, and installs nothing.** Listed last, it runs + `scripts/vm-fingerprint.sh` from the control node (`delegate_to: localhost`), + parses the capture's `DIGESTS` block, and writes + `/etc/assay/environment.json` — the interchange + [assay](https://github.com/the-metafactory/assay/blob/main/environments/README.md) + reads to learn which environment a result was produced under. It and `docker` + are the only roles using `become`; a missing or malformed digest fails the + role before anything is written, because a file naming a digest that was + never computed is worse than no file. The capture lands under gitignored + `fingerprints/` and is never registered as content — it carries the guest's + `authorized_keys`. - **Host-key checking is off** in `ansible.cfg` (same stance as `vm-fingerprint.sh`): host keys are per-instance noise in a fleet where rebuilds are routine, and `accept-new` would poison `known_hosts` on first diff --git a/ansible/roles/assay_env/defaults/main.yaml b/ansible/roles/assay_env/defaults/main.yaml new file mode 100644 index 0000000..5393210 --- /dev/null +++ b/ansible/roles/assay_env/defaults/main.yaml @@ -0,0 +1,36 @@ +# The layer-2 capstone's knobs. Everything here is context for the file the +# role writes; none of it is identity - the identity is the digest, computed +# from the machine by scripts/vm-fingerprint.sh and never supplied here. + +# Control-node path to the capture script. playbook_dir is ansible/, so its +# dirname is the repo root - the same checkout the playbook was launched +# from, which is what keeps this role from carrying its own copy of the +# script. +assay_env_fingerprint_script: "{{ playbook_dir | dirname }}/scripts/vm-fingerprint.sh" + +# Where the capture lands on the CONTROL node, not the guest. Under +# fingerprints/, which .gitignore excludes wholesale (`fingerprints/**`, with +# only .gitkeep re-admitted): a capture carries the login user's +# authorized_keys and the sshd drop-ins, so it is the operator's private +# overlay and is never committed. Must be non-empty - see the refusal in +# tasks/main.yaml for why. +assay_env_capture_path: "{{ playbook_dir | dirname }}/fingerprints/{{ inventory_hostname }}.txt" + +# Which factory backend built this machine. Context for a human reading a +# drift report; assay never compares it (assay environments/README.md). +# +# Hardcoded here rather than read from a hostvar because nothing upstream +# carries it yet: inventory/tofu.py builds hostvars from the vm-pve module's +# output, and that module is the only provider in this tree. When a second +# provider module lands, this belongs in the tofu output next to ansible_host +# and this default becomes the fallback, not the answer. +assay_env_provider: "proxmox-ve" + +# Path, in the factory's repo, of the file that declared this environment - +# the per-VM spec inventory/.yaml that tofu reads. Context, not +# identity. inventory_hostname is the VM name, which is the spec's filename. +assay_env_definition: "inventory/{{ inventory_hostname }}.yaml" + +# Where assay looks. ASSAY_ENVIRONMENT_FILE can override it at run time, but +# the fleet writes the contract's default path. +assay_env_file: /etc/assay/environment.json diff --git a/ansible/roles/assay_env/tasks/main.yaml b/ansible/roles/assay_env/tasks/main.yaml new file mode 100644 index 0000000..ea8ad7d --- /dev/null +++ b/ansible/roles/assay_env/tasks/main.yaml @@ -0,0 +1,142 @@ +# Writes /etc/assay/environment.json - the interchange assay reads to learn +# which environment a corpus ran on (crucible spec 4a; the contract is assay's +# environments/README.md). This is the layer-2 CAPSTONE: it fingerprints the +# layers already installed, so it belongs LAST in a VM spec's ansible_roles, +# after every tool role and before anything installs the software under test. +# +# The capture runs on the CONTROL node, not the guest: scripts/vm-fingerprint.sh +# SSHes in and digests what it finds. So the first task is delegate_to: +# localhost with become: false, and nothing is re-implemented on the VM. +# +# THE INVARIANT this role is written around: writing the environment file must +# not move the core digest it records. It holds because of what the capture +# does and does not look at - CORE covers os-release, hostname, timezone, the +# dpkg set, three named /etc/apt/apt.conf.d files, the Types/Suites/Components/ +# Snapshot lines of /etc/apt/sources.list.d/*.sources, enabled units, the login +# user, /etc/ssh/sshd_config.d/*.conf, /etc/docker/daemon.json, and the ~/.local +# and ~/.bun file trees. /etc is never walked, only those enumerated paths are +# read, and /etc/assay is not among them. The two modules below (file, copy) +# touch nothing else: no package, no unit, no user, no group, no hostname, and +# nothing under ~/.local or ~/.bun. Adding a task that does is how this role +# starts lying about the machine it describes. +# +# The capture itself is never registered as content. It carries the login +# user's authorized_keys and the sshd drop-ins; only the four-line DIGESTS +# tail the script prints to stderr is read back. + +- name: Refuse to capture without a capture path + ansible.builtin.fail: + msg: >- + assay_env_capture_path is empty. Given no outfile, + scripts/vm-fingerprint.sh prints the whole capture on stdout, which + would put the guest's authorized_keys and sshd drop-ins into ansible's + registered output. Point it at a path under fingerprints/ (gitignored). + when: assay_env_capture_path | default('', true) | length == 0 + +- name: Probe for the capture script on the control node + ansible.builtin.stat: + path: "{{ assay_env_fingerprint_script }}" + delegate_to: localhost + become: false + register: assay_env_script + +- name: Refuse to continue without the capture script + ansible.builtin.fail: + msg: >- + The assay_env role runs {{ assay_env_fingerprint_script }} from the + control node and it is not executable there. Run ansible-playbook from + a full checkout of this repo, or point assay_env_fingerprint_script at + one. + when: not (assay_env_script.stat.exists and assay_env_script.stat.executable) + +- name: Refuse to continue without an address and login user for the guest + ansible.builtin.fail: + msg: >- + The assay_env role addresses the guest as @ + from the control node; one of those hostvars is missing. The dynamic + inventory (ansible/inventory/tofu.py) serves both - a static inventory + has to set them too. + when: ansible_user is not defined or ansible_host is not defined + +# changed_when: false because this reports nothing about the guest changing. +# It only reads the guest, and writes on the control node - into the +# operator's private overlay, which is not the fleet's state. The one task +# here allowed to report changed is the copy at the end, which is the file +# assay actually reads. +- name: Capture the guest's fingerprint from the control node + ansible.builtin.command: + argv: + - "{{ assay_env_fingerprint_script }}" + - "{{ ansible_user }}@{{ ansible_host }}" + - "{{ assay_env_capture_path }}" + delegate_to: localhost + become: false + changed_when: false + register: assay_env_capture + +# With an outfile given, the script writes the capture to that file and +# prints only `fingerprint written to ` plus `tail -4` of it on stderr +# - the ##### DIGESTS ##### header and its three lines. That tail is what is +# parsed, so the capture body is never held in a variable. select() keeps +# only well-formed digest lines, so ssh noise on the same stream cannot be +# mistaken for one, and a truncated hash is not a match rather than a match +# on a truncated hash. +- name: Extract the digests from the capture's DIGESTS block + ansible.builtin.set_fact: + assay_env_digests: >- + {{ + dict(assay_env_capture.stderr_lines + | select('match', '^(core|provider|combined)\s+sha256:[0-9a-f]{64}$') + | map('split')) + }} + +- name: Refuse to write a partial environment file + ansible.builtin.fail: + msg: >- + The capture did not yield all three digests + (found: {{ assay_env_digests.keys() | list | sort | join(', ') | default('none', true) }}). + Nothing has been written to {{ assay_env_file }} - a file naming a + digest that was never computed is worse than no file, because assay + believes it. Inspect {{ assay_env_capture_path }}. + when: >- + 'core' not in assay_env_digests + or 'provider' not in assay_env_digests + or 'combined' not in assay_env_digests + +- name: Create the interchange directory + ansible.builtin.file: + path: "{{ assay_env_file | dirname }}" + state: directory + owner: root + group: root + mode: "0755" + become: true + +# schema is the number 1, not the string "1": assay refuses a schema it does +# not know, and "1" is not 1. Unquoted here, so it stays an int through +# to_nice_json and lands in the file as a JSON number. +# +# combined is deliberately absent. It is a capture-side convenience for +# diffing whole captures; the contract's fields are schema, core_digest, +# provider_digest, provider and definition, and an extra key buys nothing. +- name: Assemble the interchange document + ansible.builtin.set_fact: + assay_env_document: + schema: 1 + core_digest: "{{ assay_env_digests.core }}" + provider_digest: "{{ assay_env_digests.provider }}" + provider: "{{ assay_env_provider }}" + definition: "{{ assay_env_definition }}" + +# copy reports changed only when the bytes differ, so a re-run on a machine +# that has not moved is green and unchanged without a changed_when of its +# own. That is the idempotence claim: the file follows the machine, not the +# run. +- name: Write the assay environment file + ansible.builtin.copy: + dest: "{{ assay_env_file }}" + content: "{{ assay_env_document | to_nice_json }}\n" + owner: root + group: root + mode: "0644" + become: true diff --git a/inventory-example.yaml b/inventory-example.yaml index bde1301..74e9bb5 100644 --- a/inventory-example.yaml +++ b/inventory-example.yaml @@ -33,7 +33,12 @@ ipv4: dhcp # because an archive_snapshot cannot cover download.docker.com). Roles apply # in list order - metafactory_arc needs bun before it. Cross-layer deps: the # bun role needs 'unzip' and metafactory_arc needs 'git' in packages. +# +# assay_env goes LAST. It installs nothing: it fingerprints whatever the roles +# before it put on the machine and writes that identity to +# /etc/assay/environment.json, so anything listed after it would be software +# the recorded identity does not describe. # packages: [unzip, git] -# ansible_roles: [nats_server, bun, claude, docker, metafactory_arc] +# ansible_roles: [nats_server, bun, claude, docker, metafactory_arc, assay_env] tags: ["ubuntu"] From 40a164db6a5c2ca292476def65a609d7dd2f3daa Mon Sep 17 00:00:00 2001 From: Andreas Date: Wed, 2 Sep 2026 06:42:34 +1200 Subject: [PATCH 2/3] assay_env: fail closed, and stop overstating who uses become Review follow-up on 904774f. Three fixes. Fail closed on a failed re-capture. "Nothing has been written" was true and misleading: on a second run it is the file from the FIRST run that matters, still sitting on a machine that may have moved since, and that file is what assay believes. Refusing to write leaves a stale identity standing on an unverified machine - the exact failure this role exists to prevent, one run later. Capture, parse and the digest guard now sit in a block whose rescue removes the environment file before failing, so an identity nothing has just confirmed is withdrawn rather than inherited. assay reporting env@none for a machine with no file is a true statement; assay reporting a digest from last week is not. The rescue is one fail task with an inline conditional, not two guarded by `when: ... .changed` - that idiom trips ansible-lint's no-handler, and a handler is what this must not be: it has to fail the play, now, on this host. check_mode: false on the two control-node tasks. Neither writes to the guest: the stat reads the operator's own checkout, the capture reads the guest over SSH and writes only into the gitignored overlay. Skipped under --check they left the parse with nothing and the run died on an empty digest set, instead of showing the diff for the file actually at stake. README said assay_env and docker "are the only roles using become". False: base uses it throughout (ansible/roles/base/tasks/main.yaml:13 and handlers). Scoped to spec-declared roles, which is the distinction the README already draws two bullets up. Verified as root on Linux (python:3.13-slim, container as both control node and guest, stubbed capture): - happy path writes /etc/assay/environment.json 0644 root:root; core_digest and provider_digest equal the capture's DIGESTS block, schema is int 1, keys are exactly the five the contract names - second run on an unchanged machine: changed=0 - --check over a correct machine: changed=0, no crash - malformed capture with the file already standing: rescued=1, the file is gone from /etc/assay afterwards, play fails with both the original detail and what was withdrawn - --check on the fileless machine: shows the full + diff, writes nothing ansible-lint clean at the production profile, whole tree and role alone; site.yaml syntax-check clean. Refs the-metafactory/crucible#25 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_018PtJarXm8d8bSHDbhmskVm --- README.md | 9 +- ansible/roles/assay_env/tasks/main.yaml | 134 ++++++++++++++++-------- 2 files changed, 98 insertions(+), 45 deletions(-) diff --git a/README.md b/README.md index 3ff82e1..c5ecfc5 100644 --- a/README.md +++ b/README.md @@ -333,9 +333,12 @@ declarative grammar as the rest of the repo: `/etc/assay/environment.json` — the interchange [assay](https://github.com/the-metafactory/assay/blob/main/environments/README.md) reads to learn which environment a result was produced under. It and `docker` - are the only roles using `become`; a missing or malformed digest fails the - role before anything is written, because a file naming a digest that was - never computed is worse than no file. The capture lands under gitignored + are the only *spec-declared* roles using `become` — the implicit `base` role + uses it throughout. A missing or malformed digest fails the role before + anything is written, and revokes any environment file an earlier run left + behind: a file naming a digest that was never computed is worse than no + file, and so is one naming a digest this run could not confirm. The capture + lands under gitignored `fingerprints/` and is never registered as content — it carries the guest's `authorized_keys`. - **Host-key checking is off** in `ansible.cfg` (same stance as diff --git a/ansible/roles/assay_env/tasks/main.yaml b/ansible/roles/assay_env/tasks/main.yaml index ea8ad7d..6425dfb 100644 --- a/ansible/roles/assay_env/tasks/main.yaml +++ b/ansible/roles/assay_env/tasks/main.yaml @@ -23,6 +23,12 @@ # The capture itself is never registered as content. It carries the login # user's authorized_keys and the sshd drop-ins; only the four-line DIGESTS # tail the script prints to stderr is read back. +# +# FAIL CLOSED. Capture, parse and the digest guard run inside a block whose +# rescue REMOVES {{ assay_env_file }} before failing. Refusing to write is not +# enough on a re-run: a file from an earlier run is still sitting on a machine +# that may have moved since, and that file is what assay believes. An identity +# this run could not confirm is withdrawn rather than left standing. - name: Refuse to capture without a capture path ansible.builtin.fail: @@ -33,11 +39,18 @@ registered output. Point it at a path under fingerprints/ (gitignored). when: assay_env_capture_path | default('', true) | length == 0 +# check_mode: false on both control-node tasks - this one and the capture. +# Neither writes anything to the guest: the stat reads the operator's own +# checkout, and the capture reads the guest over SSH and writes only into the +# operator's gitignored overlay. Skipped under --check they would leave the +# parse with nothing, and the run would die on an empty digest set instead of +# showing the honest diff for the file that is actually at stake. - name: Probe for the capture script on the control node ansible.builtin.stat: path: "{{ assay_env_fingerprint_script }}" delegate_to: localhost become: false + check_mode: false register: assay_env_script - name: Refuse to continue without the capture script @@ -58,50 +71,87 @@ has to set them too. when: ansible_user is not defined or ansible_host is not defined -# changed_when: false because this reports nothing about the guest changing. -# It only reads the guest, and writes on the control node - into the -# operator's private overlay, which is not the fleet's state. The one task -# here allowed to report changed is the copy at the end, which is the file -# assay actually reads. -- name: Capture the guest's fingerprint from the control node - ansible.builtin.command: - argv: - - "{{ assay_env_fingerprint_script }}" - - "{{ ansible_user }}@{{ ansible_host }}" - - "{{ assay_env_capture_path }}" - delegate_to: localhost - become: false - changed_when: false - register: assay_env_capture +- name: Establish the machine's identity, or withdraw the one on file + block: + # changed_when: false because this reports nothing about the guest + # changing. It only reads the guest, and writes on the control node - + # into the operator's private overlay, which is not the fleet's state. + # The one task here allowed to report changed is the copy at the end, + # which is the file assay actually reads. + - name: Capture the guest's fingerprint from the control node + ansible.builtin.command: + argv: + - "{{ assay_env_fingerprint_script }}" + - "{{ ansible_user }}@{{ ansible_host }}" + - "{{ assay_env_capture_path }}" + delegate_to: localhost + become: false + changed_when: false + check_mode: false + register: assay_env_capture -# With an outfile given, the script writes the capture to that file and -# prints only `fingerprint written to ` plus `tail -4` of it on stderr -# - the ##### DIGESTS ##### header and its three lines. That tail is what is -# parsed, so the capture body is never held in a variable. select() keeps -# only well-formed digest lines, so ssh noise on the same stream cannot be -# mistaken for one, and a truncated hash is not a match rather than a match -# on a truncated hash. -- name: Extract the digests from the capture's DIGESTS block - ansible.builtin.set_fact: - assay_env_digests: >- - {{ - dict(assay_env_capture.stderr_lines - | select('match', '^(core|provider|combined)\s+sha256:[0-9a-f]{64}$') - | map('split')) - }} + # With an outfile given, the script writes the capture to that file and + # prints only `fingerprint written to ` plus `tail -4` of it on + # stderr - the ##### DIGESTS ##### header and its three lines. That tail + # is what is parsed, so the capture body is never held in a variable. + # select() keeps only well-formed digest lines, so ssh noise on the same + # stream cannot be mistaken for one, and a truncated hash is not a match + # rather than a match on a truncated hash. + - name: Extract the digests from the capture's DIGESTS block + ansible.builtin.set_fact: + assay_env_digests: >- + {{ + dict(assay_env_capture.stderr_lines + | select('match', '^(core|provider|combined)\s+sha256:[0-9a-f]{64}$') + | map('split')) + }} -- name: Refuse to write a partial environment file - ansible.builtin.fail: - msg: >- - The capture did not yield all three digests - (found: {{ assay_env_digests.keys() | list | sort | join(', ') | default('none', true) }}). - Nothing has been written to {{ assay_env_file }} - a file naming a - digest that was never computed is worse than no file, because assay - believes it. Inspect {{ assay_env_capture_path }}. - when: >- - 'core' not in assay_env_digests - or 'provider' not in assay_env_digests - or 'combined' not in assay_env_digests + - name: Refuse to write a partial environment file + ansible.builtin.fail: + msg: >- + The capture did not yield all three digests + (found: {{ assay_env_digests.keys() | list | sort | join(', ') | default('none', true) }}). + A file naming a digest that was never computed is worse than no + file, because assay believes it. Inspect + {{ assay_env_capture_path }}. + when: >- + 'core' not in assay_env_digests + or 'provider' not in assay_env_digests + or 'combined' not in assay_env_digests + + rescue: + # The machine keeps no identity this run could not confirm. Absent when + # there was nothing to remove, so `changed` below distinguishes "an + # identity was withdrawn" from "there was never one". + - name: Withdraw any environment file an earlier run left behind + ansible.builtin.file: + path: "{{ assay_env_file }}" + state: absent + become: true + register: assay_env_withdrawn + + # One task, not two guarded by `when: ... .changed` - that idiom reads as + # a missing handler to ansible-lint (no-handler), and a handler is exactly + # what this must not be: it has to fail the play, now, on this host. + - name: Refuse to leave the machine holding an unconfirmed identity + ansible.builtin.fail: + msg: >- + Fingerprinting failed at "{{ ansible_failed_task.name }}": + {{ ansible_failed_result.msg | default('no message given') }} + {{ + 'The environment file an earlier run wrote at ' ~ assay_env_file ~ + ' has been REMOVED. It described a machine this run could not + confirm, and a stale identity is worse than none: assay reports + env@none for a machine with no file, which is a true statement, + where it would otherwise have reported a digest nothing had just + verified.' + if assay_env_withdrawn.changed else + 'No environment file was present at ' ~ assay_env_file ~ ' and + none has been written, so the machine presents no identity at all + and assay will report env@none for it.' + }} + Fix the capture and re-run to restore it; the capture is at + {{ assay_env_capture_path }}. - name: Create the interchange directory ansible.builtin.file: From 101e27e5754265b9b895dc825b8120fa25b84094 Mon Sep 17 00:00:00 2001 From: Andreas Date: Wed, 2 Sep 2026 12:13:18 +1200 Subject: [PATCH 3/3] assay_env: the write is inside the fail-closed block, and the withdrawal is verified Review follow-up on 40a164d, from vpzed-dev-lux on vpzed-dev/smithy#26. The block ended before the interchange directory and the copy, so the fail-closed guarantee covered only half the ways this role can fail. Capture succeeds, the write dies - full disk, a permission change, a dropped connection - and the environment file from an earlier run is still sitting on a machine that may have moved since. Same stale identity the block already existed to prevent, reached from the other end. Directory creation, document assembly and the copy now sit inside the same block, so any unsuccessful refresh withdraws the file that is no longer true. That widened the rescue's own failure mode. Whatever breaks the write is frequently what breaks the removal: on a read-only filesystem the copy fails with EROFS and the unlink fails too, and the old rescue would have failed the play with the removal's errno and no word about the stale file left standing. So the withdrawal now runs with failed_when: false, a stat confirms the outcome, and the message is written from the filesystem rather than from the module's report. If the file survived, the run says so in those words - COULD NOT BE REMOVED, still on the machine, treat the digest as UNVERIFIED. A withdrawal this role only attempted is never reported as one that happened. Fourth branch while the message was open: under --check nothing is ever removed, and saying "has been REMOVED" there was the cosmetic dishonesty recorded as known-outstanding on the PR. It now says the file is still there and would have been withdrawn on a real run. Pre-flight refusals stay OUTSIDE the block, declining the review's softer ask, and the tasks now carry a comment saying why. An empty capture path, a missing capture script, an inventory with no ansible_host/ansible_user are all control-node misconfigurations: the role has not run, nothing has touched the guest, and the file already on it still describes the machine the last successful run confirmed. Withdrawing an identity because the operator launched the playbook from the wrong directory would revoke a good machine's identity over a fault on this side of the SSH connection, and would make one bad checkout's blast radius the whole fleet. Verified as root on Linux (python:3.13-slim, container as both control node and guest, real /etc/assay, stubbed capture), 38 assertions, 0 failures: - write fails ENOSPC (tmpfs sized to 64k and filled) with a stale file present: rescued=1, failure names "Write the assay environment file", /etc/assay/environment.json is gone afterwards, message claims REMOVED and absence confirmed. Same case against the pre-fix role: rescued=0, the stale file SURVIVES, nothing in the output mentions it - red observed before green - write fails EROFS (read-only bind mount) with a stale file present: rescued=1, the removal fails, the run says COULD NOT BE REMOVED / STILL ON THE MACHINE / UNVERIFIED, and the stale file is still byte-identical. Pre-fix: rescued=0, no mention of the file at all - --check with a failing capture and a file on disk: the message names check mode, claims no removal, and the file is untouched - regressions all hold: happy path writes 0644 root:root with schema int 1, the capture's two digests and no combined key; re-run on an unchanged machine changed=0; malformed digests still refuse at the partial-file guard before any write, with the file withdrawn; --check on a correct machine is changed=0 and writes nothing (mtime, size and bytes identical) The harness is scratch and deliberately uncommitted: it needs a privileged Linux container to mount the failing filesystems, which is not something this repo's CI has today. ansible-lint clean at the production profile, whole tree (25 files) and role alone (3 files); site.yaml syntax-check clean. Refs the-metafactory/crucible#25 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_018PtJarXm8d8bSHDbhmskVm --- ansible/roles/assay_env/tasks/main.yaml | 166 ++++++++++++++++-------- 1 file changed, 115 insertions(+), 51 deletions(-) diff --git a/ansible/roles/assay_env/tasks/main.yaml b/ansible/roles/assay_env/tasks/main.yaml index 6425dfb..965a10b 100644 --- a/ansible/roles/assay_env/tasks/main.yaml +++ b/ansible/roles/assay_env/tasks/main.yaml @@ -24,12 +24,38 @@ # user's authorized_keys and the sshd drop-ins; only the four-line DIGESTS # tail the script prints to stderr is read back. # -# FAIL CLOSED. Capture, parse and the digest guard run inside a block whose -# rescue REMOVES {{ assay_env_file }} before failing. Refusing to write is not -# enough on a re-run: a file from an earlier run is still sitting on a machine -# that may have moved since, and that file is what assay believes. An identity -# this run could not confirm is withdrawn rather than left standing. +# FAIL CLOSED. Everything that decides or writes the machine's identity - +# capture, parse, the digest guard, the interchange directory, the document +# and the write itself - runs inside ONE block whose rescue WITHDRAWS +# {{ assay_env_file }} before failing. Refusing to write is not enough on a +# re-run: a file from an earlier run is still sitting on a machine that may +# have moved since, and that file is what assay believes. An identity this run +# could not confirm is withdrawn rather than left standing. +# +# The write is inside the block for exactly the reason the capture is +# (vpzed-dev/smithy#26 review): a copy that dies on a full disk, a permission +# change or a dropped connection leaves the previous run's file describing a +# machine nothing just verified - the same stale identity, reached from the +# other end. +# +# And the withdrawal is VERIFIED, never assumed. Whatever breaks the write - +# no space, a read-only filesystem, a lost connection - is frequently the same +# thing that breaks the removal, so the rescue stats the path afterwards and +# says plainly when the stale file is still there. A withdrawal this role only +# attempted is never reported as one that happened. +# The three refusals below sit OUTSIDE the fail-closed block DELIBERATELY, and +# the review's softer ask - give pre-flight failures the same stale-file +# handling - is declined on purpose. Every one of them is a CONTROL-NODE +# misconfiguration: an empty capture path, no capture script in this checkout, +# an inventory serving no ansible_host/ansible_user. The role has not run, +# nothing has touched the guest, and the file already on it still describes +# the machine the last successful run confirmed. Withdrawing an identity +# because the operator launched the playbook from the wrong directory would +# revoke a good machine's identity over a fault on this side of the SSH +# connection, and would make one bad checkout's blast radius the whole fleet. +# From the capture command onward the guest is in play, and everything from +# there down is inside the block. - name: Refuse to capture without a capture path ansible.builtin.fail: msg: >- @@ -119,32 +145,108 @@ or 'provider' not in assay_env_digests or 'combined' not in assay_env_digests + # Inside the block, with the write: the interchange file is the identity, + # and every step that leads to it has to answer for a stale one. + - name: Create the interchange directory + ansible.builtin.file: + path: "{{ assay_env_file | dirname }}" + state: directory + owner: root + group: root + mode: "0755" + become: true + + # schema is the number 1, not the string "1": assay refuses a schema it + # does not know, and "1" is not 1. Unquoted here, so it stays an int + # through to_nice_json and lands in the file as a JSON number. + # + # combined is deliberately absent. It is a capture-side convenience for + # diffing whole captures; the contract's fields are schema, core_digest, + # provider_digest, provider and definition, and an extra key buys + # nothing. + - name: Assemble the interchange document + ansible.builtin.set_fact: + assay_env_document: + schema: 1 + core_digest: "{{ assay_env_digests.core }}" + provider_digest: "{{ assay_env_digests.provider }}" + provider: "{{ assay_env_provider }}" + definition: "{{ assay_env_definition }}" + + # copy reports changed only when the bytes differ, so a re-run on a + # machine that has not moved is green and unchanged without a + # changed_when of its own. That is the idempotence claim: the file + # follows the machine, not the run. + - name: Write the assay environment file + ansible.builtin.copy: + dest: "{{ assay_env_file }}" + content: "{{ assay_env_document | to_nice_json }}\n" + owner: root + group: root + mode: "0644" + become: true + rescue: # The machine keeps no identity this run could not confirm. Absent when # there was nothing to remove, so `changed` below distinguishes "an # identity was withdrawn" from "there was never one". + # + # failed_when: false so a removal that cannot succeed does not abort the + # rescue before it can SAY so. Whatever broke the write - a full disk, a + # read-only remount, a permission change - can break the unlink too, and + # a rescue that died here would fail the play with the removal's errno + # and not one word about the stale file left standing. - name: Withdraw any environment file an earlier run left behind ansible.builtin.file: path: "{{ assay_env_file }}" state: absent become: true register: assay_env_withdrawn + failed_when: false - # One task, not two guarded by `when: ... .changed` - that idiom reads as - # a missing handler to ansible-lint (no-handler), and a handler is exactly - # what this must not be: it has to fail the play, now, on this host. + # Trust the filesystem, not the module's report. This stat is the only + # evidence that the withdrawal actually happened, and the message below + # is written from it: claiming a removal that did not occur is the same + # class of lie as an environment file naming a digest nobody computed. + - name: Confirm the withdrawal by looking for the file + ansible.builtin.stat: + path: "{{ assay_env_file }}" + become: true + register: assay_env_after_withdrawal + + # One task, not several guarded by `when: ... .changed` - that idiom + # reads as a missing handler to ansible-lint (no-handler), and a handler + # is exactly what this must not be: it has to fail the play, now, on this + # host. Four branches, because there are four honest endings: check mode + # (nothing is ever removed under --check), the file survived the removal, + # the file was removed, and there was never a file. - name: Refuse to leave the machine holding an unconfirmed identity ansible.builtin.fail: msg: >- Fingerprinting failed at "{{ ansible_failed_task.name }}": {{ ansible_failed_result.msg | default('no message given') }} {{ + 'This is a --check run, so nothing was removed: the environment + file at ' ~ assay_env_file ~ ' is still there and WOULD HAVE BEEN + withdrawn on a real run. Nothing on the machine has been changed + either way.' + if ansible_check_mode and assay_env_withdrawn.changed else + 'The environment file an earlier run wrote at ' ~ assay_env_file ~ + ' COULD NOT BE REMOVED (' ~ + (assay_env_withdrawn.msg | default('no message given')) ~ + '), and it is STILL ON THE MACHINE. It describes a machine this + run could not confirm, so assay will keep reporting a digest + nothing has verified: treat that digest as UNVERIFIED and this + host as untrusted until the file is gone. Whatever broke the write + is the likeliest thing to have broken the removal, so fix that + first, then delete the file by hand or re-run this role.' + if assay_env_after_withdrawal.stat.exists else 'The environment file an earlier run wrote at ' ~ assay_env_file ~ - ' has been REMOVED. It described a machine this run could not - confirm, and a stale identity is worse than none: assay reports - env@none for a machine with no file, which is a true statement, - where it would otherwise have reported a digest nothing had just - verified.' + ' has been REMOVED, and its absence confirmed. It described a + machine this run could not confirm, and a stale identity is worse + than none: assay reports env@none for a machine with no file, + which is a true statement, where it would otherwise have reported + a digest nothing had just verified.' if assay_env_withdrawn.changed else 'No environment file was present at ' ~ assay_env_file ~ ' and none has been written, so the machine presents no identity at all @@ -152,41 +254,3 @@ }} Fix the capture and re-run to restore it; the capture is at {{ assay_env_capture_path }}. - -- name: Create the interchange directory - ansible.builtin.file: - path: "{{ assay_env_file | dirname }}" - state: directory - owner: root - group: root - mode: "0755" - become: true - -# schema is the number 1, not the string "1": assay refuses a schema it does -# not know, and "1" is not 1. Unquoted here, so it stays an int through -# to_nice_json and lands in the file as a JSON number. -# -# combined is deliberately absent. It is a capture-side convenience for -# diffing whole captures; the contract's fields are schema, core_digest, -# provider_digest, provider and definition, and an extra key buys nothing. -- name: Assemble the interchange document - ansible.builtin.set_fact: - assay_env_document: - schema: 1 - core_digest: "{{ assay_env_digests.core }}" - provider_digest: "{{ assay_env_digests.provider }}" - provider: "{{ assay_env_provider }}" - definition: "{{ assay_env_definition }}" - -# copy reports changed only when the bytes differ, so a re-run on a machine -# that has not moved is green and unchanged without a changed_when of its -# own. That is the idempotence claim: the file follows the machine, not the -# run. -- name: Write the assay environment file - ansible.builtin.copy: - dest: "{{ assay_env_file }}" - content: "{{ assay_env_document | to_nice_json }}\n" - owner: root - group: root - mode: "0644" - become: true