diff --git a/.github/workflows/ci_test.yml b/.github/workflows/ci_test.yml index 9e1bf369..ecaffb3b 100644 --- a/.github/workflows/ci_test.yml +++ b/.github/workflows/ci_test.yml @@ -32,6 +32,16 @@ jobs: - name: Cargo source package run: cargo package --locked + ledger: + name: ledger proofs and writer exclusion + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + - name: Records, proofs, and POSIX writer locks + run: cargo test --locked --test ledger --test ledger_posix_lock --test ledger_lock_aliases --test ledger_retry --test ledger_durability + msrv: name: Rust 1.88 minimum runs-on: ubuntu-latest @@ -88,6 +98,7 @@ jobs: --test eon_foss_2026_1 \ --test qmcpack_foss_2026_1 \ --test eon_packaging \ + --test bundle_component_checksums \ --test goal_scenario \ -- --nocapture diff --git a/.gitignore b/.gitignore index 747abc12..db2f28e0 100644 --- a/.gitignore +++ b/.gitignore @@ -17,9 +17,9 @@ target # rustc ICE dumps rustc-ice-*.txt -# Machine-local agent configuration -.claude/ -.codex/ +# Hidden working directories are private; repository CI remains visible. +.*/ +!/.github/ # Sphinx build (docs/orgmode/*.org is the source of truth; docs/source/*.rst # and graphviz SVGs under docs/orgmode/images are regenerated by diff --git a/AGENTS.md b/AGENTS.md index a1e115b1..4122cab2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -77,6 +77,7 @@ stack solve | sbom | order | easystack | sections | from-commit | gap | provi target list | doctor campaign run | status | finding claim | finding resolve repro summary +ledger append | head | show | verify easyblock-reach pr-overlap rebuild-set @@ -86,7 +87,7 @@ mcp The Model Context Protocol (MCP) catalog mirrors these commands, including pr-overlap, rebuild-set, component-reach, easyblock-reach, -and repro summary, with `eb_`-prefixed tools. Recipe file tools +repro summary, and the ledger commands, with `eb_`-prefixed tools. Recipe file tools (`eb_recipe_read`, `eb_recipe_list`, `eb_recipe_edit`, `eb_recipe_apply_suggestion`, `eb_recipe_diff`, `eb_recipe_write`, `eb_recipe_grep`, and `eb_dep_versions`), easyblock dry-run, module diff --git a/CHANGELOG.md b/CHANGELOG.md index fab084d9..19625814 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,22 @@ All notable changes to this project are documented here. ### Fixed +- `stack shard` resolves a build-list line with `--from-commit` or + `--from-pr` against that commit's or pull request's easyconfigs first, + then the robot path, the order EasyBuild uses. A dependency such a line + shares with another shard goes to the prelude, so two shards do not + build it at once. `petsc4py-3.25.0-foss-2026.1.eb --from-commit=...` + and the `PETSc-3.25.0-foss-2026.1.eb` line it needs plan into one + shard, PETSc first. Split over two shards, the second build stops on + PETSc's install lock. A prelude line for a module only a + fetched tree provides carries that line's `--from-commit`, + `--from-pr` and `--include-easyblocks-from-*` options. A fetched line + that still cannot be resolved stays opaque; `plan.json` gives the cause + under `opaque_reason` and the top-level `opaque`, and the command + prints a `warning:` line for it and `opaque=` in its summary. + A `--commit-cache` download that fails before an HTTP status exits 1 + and writes nothing; the error names the cache, the commit, the URL, + and the transport error. An HTTP status, including 404, stays opaque. - `stack shard` plans a build-list line it cannot resolve as an opaque line when EasyBuild can still build it: the line carries `--from-pr` or `--from-commit`, or its easyconfig is on the robot path but does @@ -44,6 +60,27 @@ All notable changes to this project are documented here. ### Added +- `stack shard --commit-cache DIR` reads the trees of `--from-commit` and + `--from-pr` lines from DIR and downloads the missing ones: a commit from + `codeload.github.com/easybuilders/easybuild-easyconfigs` (its `.eb` + files only), a pull request's changed easyconfigs at its head from + `api.github.com`. `--commit-tree COMMIT=DIR` and `--pr-tree N=DIR` name + trees already on disk. `eb_stack_shard` takes `commit_cache`, + `commit_tree` and `pr_tree`. +- `ledger append`, `ledger head`, `ledger show`, and `ledger verify` keep + a per-stack software ledger: one JSON Lines file of install, rebuild, + retire, move, access, removal, finding, and correction records. Each + line is canonical JSON (keys sorted, no whitespace, integers only) with + its `index` and `id` filled in, and its leaf is the RFC 9162 leaf hash + of the line, from the `rfc9162` crate. Heads are signed with Ed25519 + under the domain `eb-stack-ledger-v1`; the key is a PEM path passed on + the command line and is refused inside the log's directory. + `ledger verify --record N --head HEAD` checks an inclusion proof and + `ledger verify --from HEAD1 --to HEAD2` a consistency proof, so a + removed record or a rewritten log is status 1. `eb_ledger_append`, + `eb_ledger_head`, `eb_ledger_show`, and `eb_ledger_verify` are the same + commands over MCP. + - `stack order --installed` stops the build order at modules already on the machine. A file is a module listing and a directory is an Lmod tree, read the same way as `stack gap --installed`. The version is the diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0b59325f..d40319b2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -66,7 +66,7 @@ Do not hand-edit generated `docs/source/**/*.rst` or `docs/source/crates/`. ## Packaging campaigns -Follow `AGENTS.md`, `skills/annual-bump/SKILL.md` (generation rebuild), and +Follow `skills/annual-bump/SKILL.md` (generation rebuild) and `skills/new-package/SKILL.md` (conda-forge/Spack: new EasyBuild recipe). Run `skills/verify-recipe/SKILL.md` over what they emit before a build or a push, and `skills/tool-repair/SKILL.md` when the emitted file needed a hand diff --git a/Cargo.lock b/Cargo.lock index 16df86d8..9a999e4e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -108,6 +108,12 @@ version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" +[[package]] +name = "base64ct" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" + [[package]] name = "bit-set" version = "0.8.0" @@ -141,6 +147,15 @@ dependencies = [ "wyz", ] +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + [[package]] name = "block-buffer" version = "0.12.1" @@ -243,6 +258,12 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" +[[package]] +name = "const-oid" +version = "0.9.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" + [[package]] name = "const-oid" version = "0.10.2" @@ -255,6 +276,15 @@ version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + [[package]] name = "cpufeatures" version = "0.3.0" @@ -279,6 +309,16 @@ version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + [[package]] name = "crypto-common" version = "0.2.2" @@ -288,6 +328,33 @@ dependencies = [ "hybrid-array", ] +[[package]] +name = "curve25519-dalek" +version = "4.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "curve25519-dalek-derive", + "digest 0.10.7", + "fiat-crypto", + "rustc_version", + "subtle", + "zeroize", +] + +[[package]] +name = "curve25519-dalek-derive" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + [[package]] name = "cyclonedx-bom" version = "0.8.1" @@ -323,21 +390,42 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "der" +version = "0.7.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" +dependencies = [ + "const-oid 0.9.6", + "pem-rfc7468", + "zeroize", +] + [[package]] name = "deranged" version = "0.5.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer 0.10.4", + "crypto-common 0.1.7", +] + [[package]] name = "digest" version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" dependencies = [ - "block-buffer", - "const-oid", - "crypto-common", + "block-buffer 0.12.1", + "const-oid 0.10.2", + "crypto-common 0.2.2", ] [[package]] @@ -359,19 +447,22 @@ dependencies = [ "chrono", "clap", "cyclonedx-bom", + "ed25519-dalek", "flate2", "fs2", "jsonschema", + "libc", "object", "petgraph 0.7.1", "regex", "resolvo", + "rfc9162", "rustpython-ast", "rustpython-parser", "serde", "serde_json", "serde_yaml", - "sha2", + "sha2 0.11.0", "tar", "tempfile", "thiserror", @@ -380,6 +471,30 @@ dependencies = [ "version-ranges", ] +[[package]] +name = "ed25519" +version = "2.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53" +dependencies = [ + "pkcs8", + "signature", +] + +[[package]] +name = "ed25519-dalek" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9" +dependencies = [ + "curve25519-dalek", + "ed25519", + "serde", + "sha2 0.10.9", + "subtle", + "zeroize", +] + [[package]] name = "either" version = "1.17.0" @@ -447,6 +562,12 @@ version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" +[[package]] +name = "fiat-crypto" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" + [[package]] name = "filetime" version = "0.2.29" @@ -615,6 +736,16 @@ dependencies = [ "slab", ] +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + [[package]] name = "getopts" version = "0.2.24" @@ -1086,6 +1217,15 @@ version = "2.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" +[[package]] +name = "pem-rfc7468" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" +dependencies = [ + "base64ct", +] + [[package]] name = "percent-encoding" version = "2.3.2" @@ -1158,6 +1298,16 @@ version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" +[[package]] +name = "pkcs8" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" +dependencies = [ + "der", + "spki", +] + [[package]] name = "potential_utf" version = "0.1.5" @@ -1339,6 +1489,14 @@ dependencies = [ "version-ranges", ] +[[package]] +name = "rfc9162" +version = "0.6.1" +source = "git+https://github.com/leidarljos/deedar?rev=a749a63ebd085274b998efa12541bdd6abb7e176#a749a63ebd085274b998efa12541bdd6abb7e176" +dependencies = [ + "sha2 0.10.9", +] + [[package]] name = "ring" version = "0.17.14" @@ -1359,6 +1517,15 @@ version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + [[package]] name = "rustix" version = "1.1.4" @@ -1476,6 +1643,12 @@ version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + [[package]] name = "serde" version = "1.0.229" @@ -1541,6 +1714,17 @@ dependencies = [ "unsafe-libyaml", ] +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest 0.10.7", +] + [[package]] name = "sha2" version = "0.11.0" @@ -1548,8 +1732,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" dependencies = [ "cfg-if", - "cpufeatures", - "digest", + "cpufeatures 0.3.0", + "digest 0.11.3", ] [[package]] @@ -1558,6 +1742,15 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" +[[package]] +name = "signature" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" +dependencies = [ + "rand_core", +] + [[package]] name = "simd-adler32" version = "0.3.10" @@ -1591,6 +1784,16 @@ dependencies = [ "smallvec", ] +[[package]] +name = "spki" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" +dependencies = [ + "base64ct", + "der", +] + [[package]] name = "stable_deref_trait" version = "1.2.1" diff --git a/Cargo.toml b/Cargo.toml index 81ee05d1..13357b66 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -21,8 +21,6 @@ categories = [ "config", ] exclude = [ - ".claude/", - ".codex/", ".pixi/", "docs/build*/", "target/", @@ -59,6 +57,13 @@ ureq = "2" flate2 = "1" tar = "0.4" object = { version = "0.40.0", default-features = false, features = ["std", "read"] } +ed25519-dalek = { version = "2", features = ["pkcs8", "pem"] } +# RFC 9162 Merkle log for `ledger`. A git dependency until the crate is on +# crates.io; `cargo package` needs the registry version. +rfc9162 = { version = "=0.6.1", git = "https://github.com/leidarljos/deedar", rev = "a749a63ebd085274b998efa12541bdd6abb7e176" } + +[target.'cfg(unix)'.dependencies] +libc = "0.2" [dev-dependencies] jsonschema = { version = "0.26", default-features = false } diff --git a/README.md b/README.md index 838f2e75..acfbb1cf 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ The [glossary](docs/orgmode/reference/glossary.org) defines closure, generation, ## Discovery - [llms.txt](llms.txt) indexes the manual. [llms-full.txt](llms-full.txt) repeats every page in one file. -- [AGENTS.md](AGENTS.md) lists the build, test, lint, and docs commands. +- [Contributing](CONTRIBUTING.md) describes contribution and validation requirements. - The skills for inspect, plan, bump, a stack SBOM, pull-request overlap, and the stdio server are under [skills/](skills/). - The Model Context Protocol manifest is [server.json](server.json). Client configuration is the manual page [Model Context Protocol server](https://eb-stack.rgoswami.me/reference/mcp.html). @@ -473,3 +473,18 @@ The suite covers foreign syntax adapters, conditions, CycloneDX generation, prof ## License [MIT](LICENSE) · [Code of Conduct](CODE_OF_CONDUCT.md) · [Security](SECURITY.md) · [Contributing](CONTRIBUTING.md) + +## Ledger event retries + +A producer can retain one JSON event packet with a unique `build.event_id` +and retry `ledger append` with that same packet. An identical retry +acknowledges the recorded entry and signs the current head without adding +another line. Different content under the same identifier is refused. +The identifier belongs to one event, including its operation stage. + +A failed signed-head write can occur after the record is durable. Retaining +the packet makes that failure recoverable without duplicating the event. +An acknowledged retry accepts `--expect-head` only when it matches a prefix +of the current history. A new append still requires the exact current head. +Head signature trust is checked through `ledger verify`; the expected-head +argument is an optimistic history check. diff --git a/docs/orgmode/explanation/sbom-blast-radius.org b/docs/orgmode/explanation/sbom-blast-radius.org index 6487242c..33d9354e 100644 --- a/docs/orgmode/explanation/sbom-blast-radius.org +++ b/docs/orgmode/explanation/sbom-blast-radius.org @@ -989,7 +989,7 @@ table. ~reverse=0~ again, for the same reason as the CMakeMake run. * The same document over MCP -The Model Context Protocol (MCP) catalog has 51 tools. ~tools/list~ prints +The Model Context Protocol (MCP) catalog has 55 tools. ~tools/list~ prints their names: #+begin_src bash @@ -1050,9 +1050,13 @@ eb_pr_overlap eb_rebuild_set eb_easyblock_reach eb_repro_summary +eb_ledger_append +eb_ledger_head +eb_ledger_show +eb_ledger_verify #+end_src -That is 51 tools. +That is 55 tools. ~eb_stack_sbom~ writes the CycloneDX file ~stack sbom~ writes. Its ~tools/list~ description reads: "Emit CycloneDX from an existing stack lock, reading dependency edges and source checksums from the easyconfigs the lock names. With backport_list, also write the modules whose reach class justifies a backport." ~eb_stack_sbom~ reads the Cbc lock: diff --git a/docs/orgmode/howto/stack-shard.org b/docs/orgmode/howto/stack-shard.org index 8a585c5d..9fe100ab 100644 --- a/docs/orgmode/howto/stack-shard.org +++ b/docs/orgmode/howto/stack-shard.org @@ -71,6 +71,98 @@ in that file. ~--weights~ is ~Name/VersionFull~ or the easyconfig filename, a tab, then seconds. A module the file does not list weighs ~--default-weight~, which is 1. Build the prelude, then the shards. -A line whose easyconfig is not on the robot path exits 1 and writes -nothing. ~eb_stack_shard~ is the same command. See +* Resolve lines EasyBuild fetches from GitHub + +A line with ~--from-commit=SHA~ or ~--from-pr=N~ builds an easyconfig +the robot path may not have. Pass a cache directory and ~stack shard~ +reads each such line's tree there, downloading the ones it lacks: + +#+begin_src bash +eb-stack stack shard \ + --build-list Snel-2026e.list \ + --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs \ + --commit-cache /scratch-shared/$USER/eb-stack-commit-trees \ + --shards 4 \ + --out-dir /tmp/shards +#+end_src + +The line is then closed over its tree first and the robot path second, +as EasyBuild closes it. ~petsc4py-3.25.0-foss-2026.1.eb --from-commit=...~ +needs PETSc 3.25.0, which is a line of its own, so the two lines share a +shard and PETSc builds first. A dependency it shares with another shard +goes to the prelude. ~--commit-tree SHA=DIR~ and ~--pr-tree N=DIR~ name a +tree you already have. A fetched line that still cannot be resolved is +planned on its own without a closure; the command prints a +~warning:~ line with the reason, and ~plan.json~ keeps it under +~opaque_reason~. + +A line whose easyconfig is not on the robot path, and that EasyBuild +does not fetch, exits 1 and writes nothing. A ~--commit-cache~ download +that fails before an HTTP status also exits 1 and writes nothing. The +message names the cache, the commit, the URL, and the transport error. +~eb_stack_shard~ is the same command. See [[file:../reference/cli.org][the CLI reference]]. + +* Plan a list that fetches PETSc + +~tests/stack_shard_fixtures/fetched.list~ is 34 lines whose easyconfigs +are in easybuild-easyconfigs +~0d6b80c34f3553774e93162cf89088e2423ecf89~ (the commit that adds +~petsc4py-3.25.0-foss-2026.1.eb~). Eleven lines pass ~--from-commit~ for +that commit, ten pass only +~--include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287~ +(the peeled commit of easybuild-easyblocks v5.4.1), and the rest are +plain. ~petsc4py~ depends on ~PETSc~ 3.25.0. The PETSc line is one of +the fetched lines. + +The robot path is that commit's ~easybuild/easyconfigs~ with +~p/petsc4py/petsc4py-3.25.0-foss-2026.1.eb~ removed, so the file comes +only from the commit. ~EB_STACK_COMMIT_MIRROR~, when set, is a local +copy of the codeload ~.tar.gz~ for the commit. The shard command still +downloads that commit into ~--commit-cache~. + +#+begin_src bash +SHA=0d6b80c34f3553774e93162cf89088e2423ecf89 +MIRROR=${EB_STACK_COMMIT_MIRROR:-/tmp/ec-mirror/${SHA}.tar.gz} +mkdir -p "$(dirname "$MIRROR")" +HOST=codeload.github.com +test -f "$MIRROR" || curl -fsSL -o "$MIRROR" \ + "https://${HOST}/easybuilders/easybuild-easyconfigs/tar.gz/${SHA}" +rm -rf /tmp/ec-robot +mkdir -p /tmp/ec-robot +tar -xzf "$MIRROR" -C /tmp/ec-robot +ROBOT=/tmp/ec-robot/easybuild-easyconfigs-${SHA}/easybuild/easyconfigs +rm -f "$ROBOT"/p/petsc4py/petsc4py-3.25.0-foss-2026.1.eb +eb-stack stack shard \ + --build-list tests/stack_shard_fixtures/fetched.list \ + --easyconfigs "$ROBOT" \ + --commit-cache /tmp/eb-stack-commit-trees \ + --shards 4 \ + --generated-at 2026-11-04T00:00:00Z \ + --out-dir /tmp/shards-fetched +#+end_src + +That command printed: + +#+begin_example +shards=4 prelude=173 lines=34 +#+end_example + +~/tmp/shards-fetched/shard-03.list~ holds the PETSc line and then +petsc4py. No module is listed in two shards or in a shard and the +prelude. + +#+begin_example +Hypre-3.1.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +SuperLU_DIST-9.2.1-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +PETSc-3.25.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +petsc4py-3.25.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +HeFFTe-2.4.1-foss-2026.1.eb +CheMPS2-1.8.12-foss-2026.1.eb +#+end_example + +Without ~--commit-cache~, petsc4py is planned on its own. The summary +is ~shards=4 prelude=173 lines=34 opaque=1~, and petsc4py is in a +different shard from PETSc. The same inputs with the cache put them in +one shard, PETSc first. A second run with ~--generated-at~ writes the +same files. diff --git a/docs/orgmode/index.org b/docs/orgmode/index.org index e91b9296..e8db1035 100644 --- a/docs/orgmode/index.org +++ b/docs/orgmode/index.org @@ -395,8 +395,8 @@ Rohit Goswami, eb-stack (source revision ), https://github.com/HaoZeke/e this manual. [[https://github.com/HaoZeke/eb-stack/blob/main/llms-full.txt][llms-full.txt]] repeats every page in one file. - [[https://github.com/HaoZeke/eb-stack/blob/main/AGENTS.md][AGENTS.md]] lists - the build, test, lint, and docs commands. The skills for inspect, plan, + [[https://github.com/HaoZeke/eb-stack/blob/main/CONTRIBUTING.md][Contributing]] describes + contribution and validation requirements. The skills for inspect, plan, bump, a stack SBOM, pull-request overlap, and the stdio server are under [[https://github.com/HaoZeke/eb-stack/tree/main/skills][skills/]]. The Model Context Protocol manifest is diff --git a/docs/orgmode/reference/cli.org b/docs/orgmode/reference/cli.org index 7ff01c7f..272d3365 100644 --- a/docs/orgmode/reference/cli.org +++ b/docs/orgmode/reference/cli.org @@ -1,8 +1,8 @@ #+TITLE: Command-line reference #+OPTIONS: toc:t num:nil author:nil -~eb-stack~ has six namespaced command groups (~package~, ~recipe~, -~stack~, ~target~, ~campaign~, ~repro~), four reports (~easyblock-reach~, +~eb-stack~ has seven namespaced command groups (~package~, ~recipe~, +~stack~, ~target~, ~campaign~, ~repro~, ~ledger~), four reports (~easyblock-reach~, ~pr-overlap~, ~rebuild-set~, ~component-reach~), and a Model Context Protocol (MCP) server. ~--version~ prints the crate version. All paths written by package commands are deterministic bundle artifacts. Closure, generation, baseline, @@ -17,7 +17,7 @@ reachability, toolchain family, and the claim ladder are defined in | 1 | The work was attempted and failed: a missing or unreadable file, a document that does not parse, a check that found errors, a network fetch that failed, a doctor probe, or a campaign whose build failed | | 2 | The command line was rejected before that work: a missing or unknown flag (clap), or a flag value or combination the command does not accept | -A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails ~recipe lint~ or ~recipe check~ is status 1: the files were read and the check failed. ~stack from-commit~ is status 1 when any pin is ~released~, and status 0 when every pin is ~newer~ or ~unknown~ or the files name no pin. A file name with no ~eb-X.Y.Z~, and no ~--easybuild-version~, is status 2. ~stack arch~ is status 1 when an entry is unsupported or flagged, or when ~--probe-sources~ reports ~unreachable~ or ~arch-missing~ sources. ~stack files~ is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are ~no_url~ or ~unchecked~. ~stack generation~ is status 1 on any finding, and status 2 when an easystack path names no EESSI version and ~--eessi-version~ is absent. ~stack known-failures~ is status 0 when it prints warnings: they are advice for the pull request, not a refusal; with ~--strict~ it is status 1 when a warning remains, and status 0 when every hit is remedied. ~stack oom~ is status 0 when it warns: the warnings are advice, not a failed check. A ~--node~ value that is not ~CORES:MEMORY_MIB~ or ~CORES:MEMORY_MIB:TARGET~ is status 2, and so is a command with none of ~--easystack~, ~--node~, and ~--hooks~, or a ~--parallel~ or ~--max-parallel~ of 0. A ~--hooks~ file that is not valid Python, has no module-level ~PARALLELISM_LIMITS~, or has a row the model cannot follow is status 1, and the message names the line. ~stack bot-classify~ is status 0 for every class, and status 2 when a path is a directory. ~stack batch~ is status 1 when the plan includes, or depends on, an entry the architecture cannot build. ~stack provided~ is status 1 when a build-list line names software the baseline already provides, names no package, uses a toolchain other than 2026.1 or GCCcore-15.2.0 below it, sets ~--hooks~, or belongs in a ~--family-lock~. A group-restricted warning is status 0 unless ~--strict~. Status 2 is when neither ~--installed~ nor ~--baseline-lock~ is set. ~stack shard~ is status 1 when a build-list line cannot be resolved, and writes nothing; ~--shards 0~ and a list with no lines are status 2. ~stack shadow~ is status 1 when a site-layer recipe shadows a base-stack module, or when a recipe does not parse, and status 0 when every recipe names software the base does not provide. +A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails ~recipe lint~ or ~recipe check~ is status 1: the files were read and the check failed. ~stack from-commit~ is status 1 when any pin is ~released~, and status 0 when every pin is ~newer~ or ~unknown~ or the files name no pin. A file name with no ~eb-X.Y.Z~, and no ~--easybuild-version~, is status 2. ~stack arch~ is status 1 when an entry is unsupported or flagged, or when ~--probe-sources~ reports ~unreachable~ or ~arch-missing~ sources. ~stack files~ is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are ~no_url~ or ~unchecked~. ~stack generation~ is status 1 on any finding, and status 2 when an easystack path names no EESSI version and ~--eessi-version~ is absent. ~stack known-failures~ is status 0 when it prints warnings: they are advice for the pull request, not a refusal; with ~--strict~ it is status 1 when a warning remains, and status 0 when every hit is remedied. ~stack oom~ is status 0 when it warns: the warnings are advice, not a failed check. A ~--node~ value that is not ~CORES:MEMORY_MIB~ or ~CORES:MEMORY_MIB:TARGET~ is status 2, and so is a command with none of ~--easystack~, ~--node~, and ~--hooks~, or a ~--parallel~ or ~--max-parallel~ of 0. A ~--hooks~ file that is not valid Python, has no module-level ~PARALLELISM_LIMITS~, or has a row the model cannot follow is status 1, and the message names the line. ~stack bot-classify~ is status 0 for every class, and status 2 when a path is a directory. ~stack batch~ is status 1 when the plan includes, or depends on, an entry the architecture cannot build. ~stack provided~ is status 1 when a build-list line names software the baseline already provides, names no package, uses a toolchain other than 2026.1 or GCCcore-15.2.0 below it, sets ~--hooks~, or belongs in a ~--family-lock~. A group-restricted warning is status 0 unless ~--strict~. Status 2 is when neither ~--installed~ nor ~--baseline-lock~ is set. ~stack shard~ is status 1 when a build-list line cannot be resolved and is not fetched by EasyBuild, or when a ~--commit-cache~ download fails before an HTTP status, and writes nothing in either case. ~--shards 0~, a list with no lines, and a ~--commit-tree~ or ~--pr-tree~ that is not ~COMMIT=DIR~ or ~N=DIR~ are status 2. ~stack shadow~ is status 1 when a site-layer recipe shadows a base-stack module, or when a recipe does not parse, and status 0 when every recipe names software the base does not provide. ~ledger verify~ is status 1 when a proof or a head signature does not check, and status 2 without exactly one of ~--record~ with ~--head~ and ~--from~ with ~--to~. ~ledger append~ is status 1 when the record is outside the format, the log no longer reads, or the log is not at ~--expect-head~; it then writes nothing. ~--help~ for every command has an example and a description of every flag. A default that the command applies is printed in that help. A directory that receives several artifacts is ~--out-dir~. A single output file is ~--out~ (~--lock-out~ for the lock ~stack solve~ writes). ~--installed~, ~--easyconfigs~, ~--filter-deps~, and ~--generated-at~ use those names on every command that takes them. @@ -61,6 +61,10 @@ Package commands take the toolchain as ~--toolchain name/version~, the spelling | ~campaign run~ / ~campaign status~ | Execute or inspect a persisted build campaign | | ~campaign finding claim~ | Claim an open finding for one worker | | ~campaign finding resolve~ | Resolve a claimed finding with an action, evidence, and the paths it changed | +| ~ledger append~ | Append one event record to a per-stack ledger as a canonical JSON line and sign the new head | +| ~ledger head~ | Sign a ledger's current head, for a published anchor | +| ~ledger show~ | List ledger records newest first, for one module or software name | +| ~ledger verify~ | Check that a record is in a signed head, or that one signed head extends another | | ~mcp [--workspace]~ | Serve each CLI workflow over stdio, including ~pr-overlap~, ~rebuild-set~, ~component-reach~, and ~easyblock-reach~, with recipe file tools rooted at ~--workspace~ | | ~easyblock-reach~ | Report easyconfigs and installed modules reached by an easyblock or framework change | @@ -492,6 +496,7 @@ eb-stack stack batch \ eb-stack stack shard \ --build-list NODE-TYPE.list \ --easyconfigs TREE [--easyconfigs OVERLAY]... \ + [--commit-cache DIR] [--commit-tree COMMIT=DIR]... [--pr-tree N=DIR]... \ [--installed-modules-dir MODULES/all]... [--installed INSTALLED.txt]... \ --shards 4 \ [--weights SECONDS.tsv] [--default-weight 1] \ @@ -823,6 +828,28 @@ a fraction rounds up), or ~--default-weight~ (1) when the file does not list it. ~--filter-deps~, or ~EASYBUILD_FILTER_DEPS~ when the flag is absent, drops dependencies the way EasyBuild does. +A line with ~--from-commit=SHA~ or ~--from-pr=N~ is resolved the way +EasyBuild resolves it: that commit's or pull request's easyconfigs first, +then the robot path. ~--commit-tree COMMIT=DIR~ names the easyconfigs +directory of one commit (an abbreviated commit matches) and +~--pr-tree N=DIR~ the easyconfigs one pull request changes; both repeat. +~--commit-cache DIR~ holds every other tree. A commit missing there is +downloaded over https from +~codeload.github.com/easybuilders/easybuild-easyconfigs/tar.gz/~ +plus the commit. That address also serves a fork commit a pull request +carries, and only its ~.eb~ files under ~easybuild/easyconfigs/~ are +kept, at ~DIR/SHA/easybuild/easyconfigs~. +A pull request's head commit and changed easyconfigs come from +~api.github.com~ and are kept at ~DIR/pr-N-HEAD~, so a pull request that +moves is read again. ~GITHUB_TOKEN~ or ~GH_TOKEN~ is sent to +~api.github.com~ when set. A tree is written under a temporary name and +renamed into place, so several planners can share one cache. A module the +closures of two or more shards need goes to the prelude whichever tree it +comes from; a prelude line for a module only a fetched tree provides +carries that line's ~--from-commit~, ~--from-pr~ and +~--include-easyblocks-from-*~ options, so it builds the same easyconfig +with the same easyblock. + ~--out-dir~ gets ~prelude.list~ (the prelude as easyconfig filenames in dependency order; empty when nothing is shared), ~shard-NN.list~ (the input lines verbatim, dependencies first), and ~plan.json~ (each line's closure, @@ -830,11 +857,26 @@ group and shard, the weight of every shard and of the prelude, and a SHA-256 over the build list, the weights, the installed modules and every easyconfig a closure reached). A ~shard-NN.list~ that an earlier run left and this run does not write is removed. ~--generated-at~ fixes the time in -~plan.json~, so the same inputs give identical files. A line whose -easyconfig is not on the robot path, whose first word is not an ~.eb~ -file, or whose closure has a requirement nothing satisfies exits 1, names -every such line, and writes nothing. The command prints -~shards= prelude= lines=~. +~plan.json~, so the same inputs give identical files. A +~--commit-cache~ download that fails before an HTTP status (the cache has +no tree and the client never gets a status) exits 1 and writes nothing; +the message names the cache, the commit, the URL, and the transport +error. An HTTP error status is not that case. A fetched line +that still cannot be resolved (no tree for its commit, its easyconfig in +neither its tree nor the robot path, or a requirement neither provides), +and a line whose easyconfig is on the robot path but does not parse, is +planned as an opaque line: its own group, no closure, the default weight, +written to its shard unchanged. ~plan.json~ gives the kind under each +line's ~opaque~, the cause under ~opaque_reason~, and every opaque line +under the top-level ~opaque~; ~inputs.trees~ maps each fetched source to +~null~ when its tree was read or to the reason it was not. Each opaque line +and each other caveat (a fetched line planned over the robot path only, a +module two trees resolve differently) is printed to stderr as +~warning: ...~ and kept under ~warnings~. Any other line whose easyconfig is +not on the robot path, whose first word is not an ~.eb~ file, or whose +closure has a requirement nothing satisfies exits 1, names every such line, +and writes nothing. The command prints ~shards= prelude= lines=~, +followed by ~opaque=~ when a line is opaque. ** ~stack arch~ @@ -1709,6 +1751,116 @@ independently. See [[file:campaigns.org][campaign state and typed findings]] for the serialized state, claim ladder, class/disposition table, and repair protocol. +* Ledger commands + +#+begin_src bash +eb-stack ledger append RECORD.json \ + --log ledger/2026.06.jsonl \ + [--expect-head HEAD.json] \ + [--key KEY.pem --head-out HEAD.json] + +eb-stack ledger head \ + --log ledger/2026.06.jsonl --key KEY.pem --out HEAD.json [--ts RFC3339] + +eb-stack ledger show --log ledger/2026.06.jsonl [--package NAME] [--json] + +eb-stack ledger verify \ + --log ledger/2026.06.jsonl --public-key KEY.pub \ + --record N --head HEAD.json + +eb-stack ledger verify \ + --log ledger/2026.06.jsonl --public-key KEY.pub \ + --from HEAD1.json --to HEAD2.json +#+end_src + +A ledger is one append-only JSON Lines file per stack. Each line is one +event on one installation, and every line of a file has the same ~stack~. +The file is an RFC 9162 Merkle log over the ~rfc9162~ crate: the leaf of +record ~N~ is ~SHA-256(0x00 || line N)~, over the bytes of the line without +its newline, and an interior node is ~SHA-256(0x01 || left || right)~. + +*Canonical form.* A record is a JSON object written as one line: keys +sorted by their UTF-8 bytes at every depth, no whitespace outside strings, +strings escaped as ~serde_json~ escapes them (~"~, ~\~ and control +characters; other characters as UTF-8), numbers as integers only. A record +with a fractional number is refused. Two files that differ only in key +order and spacing append the same line. + +*Fields.* ~append~ refuses a field outside this table. + +| Field | Content | +|- +| ~ts~ | UTC time, RFC 3339 (~Z~ or ~+00:00~). ~append~ fills the current time when absent | +| ~actor~ | Who wrote it: a Jenkins run URL, a bot pull request and job, or a login | +| ~stack~ | ~2026.06~ (EESSI version) or ~2025~ | +| ~target~ | CPU target and accelerator, e.g. ~x86_64/amd/zen4~ | +| ~module~ | Module name, e.g. ~PETSc/3.25.0-foss-2026.1~ | +| ~location~ | The installation path that RPATHs use | +| ~action~ | ~install~, ~rebuild-inplace~, ~rebuild-bump~, ~retire~, ~unretire~, ~move~, ~access~, ~remove~, ~finding~, or ~correction~ | +| ~stage~ | Optional: ~started~, ~finished~, or ~aborted~, for an operation that runs in steps | +| ~reason~, ~easyconfig~, ~sources~, ~sbom~, ~modulefile~, ~build~ | Optional JSON: free text or an object of references and SHA-256 sums | +| ~supersedes~, ~superseded_by~, ~moved_from~, ~moved_to~, ~corrects~ | Optional ~id~ of a related record. A ~correction~ needs ~corrects~ | +| ~index~ | Leaf number. Written by ~append~; a record that carries another value is refused | +| ~id~ | SHA-256 of the canonical record without ~id~ and ~sig~. Written by ~append~; a record that carries another value is refused | +| ~sig~ | Optional per-record signature, kept in the line and left out of ~id~ | + +The first fields are required non-empty strings: ~ts~, ~actor~, ~stack~, +~target~, ~module~, ~location~, and ~action~. A wrong record is never edited: +a ~correction~ record follows it. Licence numbers, keys and tokens do not +belong in a record. + +*Reading a log.* ~show~, ~verify~, ~head~ and ~append~ read every line and +refuse the log when a line is not canonical, its ~index~ is not its line +number, its ~id~ does not recompute, its ~stack~ differs from the first +line, or the last line has no newline. + +*Heads.* A head file is one JSON line: ~domain~ (always +~eb-stack-ledger-v1~), ~size~, ~root~ (hex), ~ts~, ~public_key~ (hex), and +~signature~ (hex). The Ed25519 signature covers +~eb-stack-ledger-v1sizerootts~. ~--key~ is an Ed25519 PKCS#8 +PEM file, as ~openssl genpkey -algorithm ed25519~ writes it. A key inside the +log's directory is refused, so the key is always an argument and never a +file beside the log. ~--public-key~ is the matching PEM (~openssl pkey +-pubout~), or the 64 hex characters of a head's ~public_key~. + +~append~ locks the log file, checks the log, refuses the record when the +log is not at the size and root of ~--expect-head~, writes the line, and, +with ~--key~ and ~--head-out~, writes the signed head after the append. It +prints ~appended record N id ID; head size S root ROOT~. ~head~ signs the +log's current head into ~--out~, at ~--ts~ or now. + +*Locking and crashes.* Writers take an exclusive POSIX record lock on +~LOG.posix-lock~ beside the log; ~head~ takes a shared lock on the same +file, so a head is the log between two appends and an older head never +replaces a newer one. ~head~ opens the log and the lock file read-only and +needs no write access to the ledger directory, but refuses a log whose +lock file no writer has created. ~append~ flushes the log and its +directory before it reports a record, including a record it acknowledges +as already recorded. A write that fails part way is cut back. Bytes after +the last newline that are not a JSON value, left by an append that never +finished, are copied to a new ~LOG.quarantine.*~ file and flushed before +the next ~append~ cuts them from the log; ~append~ then adds +~; moved N bytes of an interrupted append to PATH~ to its line. A complete +JSON value without its newline is refused and left in place. A head file is +replaced by renaming a flushed new file over it. A head path that names +the log, its lock file, a quarantine file, or the signing key, directly or +through a symbolic or hard link, is refused, and so is a head path that is +a symbolic link. A signing key with more than one hard link is refused. The +lock is a POSIX record lock: whether it excludes writers on several hosts +depends on the shared file system's lock support. ~show~ prints ~index~, +~ts~, ~action~, ~target~, ~module~ and ~actor~ per record, newest first, then +~K of N records~. ~--package~ matches a whole module name, or the software +name before its ~/~. ~--json~ prints the records as a JSON array. + +~verify --record N --head HEAD~ checks HEAD's signature against +~--public-key~, computes the inclusion proof of record N from the first +~size~ lines, and checks it against HEAD's root. A record removed from the +log, or any line changed, makes the root differ. ~verify --from HEAD1 --to +HEAD2~ checks both signatures, computes the consistency proof from HEAD1's +size to HEAD2's size, and checks that it yields both roots. A log rewritten +under HEAD1 and signed again fails, because no proof yields the old root +and the new one. + * ~repro summary~ #+begin_src bash @@ -1798,7 +1950,9 @@ plus sandboxed recipe and easyblock tools: - ~eb_pr_overlap~ (the same writer as ~pr-overlap~, including ~--kind easyconfigs~, a merge order, and ~--impact~), ~eb_rebuild_set~, ~eb_component_reach~, ~eb_easyblock_reach~; -- ~eb_repro_summary~. +- ~eb_repro_summary~; +- ~eb_ledger_append~, ~eb_ledger_head~, ~eb_ledger_show~, + ~eb_ledger_verify~. ~eb_recipe_read~, ~eb_recipe_list~, ~eb_recipe_edit~, ~eb_recipe_write~, and ~eb_recipe_grep~ resolve every path inside that root. A ~..~ component, an @@ -1887,7 +2041,14 @@ that is not an integer, and ~hook_warnings~ for a dropped sum in the ~oom_profile~, and ~oom_resources~. It refuses an unsupported plan the same way and writes known-failures and oom warnings. ~eb_repro_summary~ -requires ~scores~ and takes ~ratchet~ and ~json~. Each of these tools +requires ~scores~ and takes ~ratchet~ and ~json~. +~eb_ledger_append~ requires ~log~ and ~record~ and takes ~expect_head~, +~key~ and ~head_out~; it also returns ~index~, ~id~, ~size~, ~root~ and the +signed head. ~eb_ledger_head~ requires ~log~, ~key~ and ~out~ and takes ~ts~. +~eb_ledger_show~ requires ~log~ and takes ~package~ and ~json~, and returns +the records. ~eb_ledger_verify~ requires ~log~ and ~public_key~, and either +~record~ (an integer) with ~head~, or ~from~ with ~to~; it returns the proof +nodes, and a proof that does not check is a tool error. Each of these tools calls the same runner as the command of that name, and the text it returns is that command's stdout. diff --git a/docs/orgmode/reference/mcp.org b/docs/orgmode/reference/mcp.org index 6bf95858..64798ad3 100644 --- a/docs/orgmode/reference/mcp.org +++ b/docs/orgmode/reference/mcp.org @@ -88,7 +88,7 @@ absolute path there. * Tools -The catalog has 51 tools. ~tools/list~ returns one tool for each CLI +The catalog has 55 tools. ~tools/list~ returns one tool for each CLI command in the table, and the recipe and easyblock tools that have no command. ~mcp~ publishes that catalog and has no tool of its own. Each command's tool calls the same runner as that command. The ~text~ field @@ -133,6 +133,10 @@ name in the schema ~tools/list~ returns. The | ~campaign finding claim~ | ~eb_campaign_finding_claim~ | | ~campaign finding resolve~ | ~eb_campaign_finding_resolve~ | | ~repro summary~ | ~eb_repro_summary~ | +| ~ledger append~ | ~eb_ledger_append~ | +| ~ledger head~ | ~eb_ledger_head~ | +| ~ledger show~ | ~eb_ledger_show~ | +| ~ledger verify~ | ~eb_ledger_verify~ | | ~easyblock-reach~ | ~eb_easyblock_reach~ | | ~pr-overlap~ | ~eb_pr_overlap~ | | ~rebuild-set~ | ~eb_rebuild_set~ | diff --git a/llms-full.txt b/llms-full.txt index 87019011..4d77e146 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -163,8 +163,8 @@ Rohit Goswami, eb-stack (source revision ), https://github.com/HaoZeke/e this manual. [llms-full.txt](https://github.com/HaoZeke/eb-stack/blob/main/llms-full.txt) repeats every page in one file. - [AGENTS.md](https://github.com/HaoZeke/eb-stack/blob/main/AGENTS.md) lists - the build, test, lint, and docs commands. The skills for inspect, plan, + [Contributing](https://github.com/HaoZeke/eb-stack/blob/main/CONTRIBUTING.md) describes + contribution and validation requirements. The skills for inspect, plan, bump, a stack SBOM, pull-request overlap, and the stdio server are under [skills/](https://github.com/HaoZeke/eb-stack/tree/main/skills). The Model Context Protocol manifest is @@ -5605,10 +5605,102 @@ in that file. `--weights` is `Name/VersionFull` or the easyconfig filename, a tab, then seconds. A module the file does not list weighs `--default-weight`, which is 1. Build the prelude, then the shards. -A line whose easyconfig is not on the robot path exits 1 and writes -nothing. `eb_stack_shard` is the same command. See +# Resolve lines EasyBuild fetches from GitHub + +A line with `--from-commit=SHA` or `--from-pr=N` builds an easyconfig +the robot path may not have. Pass a cache directory and `stack shard` +reads each such line's tree there, downloading the ones it lacks: + +```bash +eb-stack stack shard \ + --build-list Snel-2026e.list \ + --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs \ + --commit-cache /scratch-shared/$USER/eb-stack-commit-trees \ + --shards 4 \ + --out-dir /tmp/shards +``` + +The line is then closed over its tree first and the robot path second, +as EasyBuild closes it. `petsc4py-3.25.0-foss-2026.1.eb --from-commit=...` +needs PETSc 3.25.0, which is a line of its own, so the two lines share a +shard and PETSc builds first. A dependency it shares with another shard +goes to the prelude. `--commit-tree SHA=DIR` and `--pr-tree N=DIR` name a +tree you already have. A fetched line that still cannot be resolved is +planned on its own without a closure; the command prints a +`warning:` line with the reason, and `plan.json` keeps it under +`opaque_reason`. + +A line whose easyconfig is not on the robot path, and that EasyBuild +does not fetch, exits 1 and writes nothing. A `--commit-cache` download +that fails before an HTTP status also exits 1 and writes nothing. The +message names the cache, the commit, the URL, and the transport error. +`eb_stack_shard` is the same command. See [the CLI reference](https://eb-stack.rgoswami.me/../reference/cli.html.md). +# Plan a list that fetches PETSc + +`tests/stack_shard_fixtures/fetched.list` is 34 lines whose easyconfigs +are in easybuild-easyconfigs +`0d6b80c34f3553774e93162cf89088e2423ecf89` (the commit that adds +`petsc4py-3.25.0-foss-2026.1.eb`). Eleven lines pass `--from-commit` for +that commit, ten pass only +`--include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287` +(the peeled commit of easybuild-easyblocks v5.4.1), and the rest are +plain. `petsc4py` depends on `PETSc` 3.25.0. The PETSc line is one of +the fetched lines. + +The robot path is that commit's `easybuild/easyconfigs` with +`p/petsc4py/petsc4py-3.25.0-foss-2026.1.eb` removed, so the file comes +only from the commit. `EB_STACK_COMMIT_MIRROR`, when set, is a local +copy of the codeload `.tar.gz` for the commit. The shard command still +downloads that commit into `--commit-cache`. + +```bash +SHA=0d6b80c34f3553774e93162cf89088e2423ecf89 +MIRROR=${EB_STACK_COMMIT_MIRROR:-/tmp/ec-mirror/${SHA}.tar.gz} +mkdir -p "$(dirname "$MIRROR")" +HOST=codeload.github.com +test -f "$MIRROR" || curl -fsSL -o "$MIRROR" \ + "https://${HOST}/easybuilders/easybuild-easyconfigs/tar.gz/${SHA}" +rm -rf /tmp/ec-robot +mkdir -p /tmp/ec-robot +tar -xzf "$MIRROR" -C /tmp/ec-robot +ROBOT=/tmp/ec-robot/easybuild-easyconfigs-${SHA}/easybuild/easyconfigs +rm -f "$ROBOT"/p/petsc4py/petsc4py-3.25.0-foss-2026.1.eb +eb-stack stack shard \ + --build-list tests/stack_shard_fixtures/fetched.list \ + --easyconfigs "$ROBOT" \ + --commit-cache /tmp/eb-stack-commit-trees \ + --shards 4 \ + --generated-at 2026-11-04T00:00:00Z \ + --out-dir /tmp/shards-fetched +``` + +That command printed: + +``` +shards=4 prelude=173 lines=34 +``` + +`/tmp/shards-fetched/shard-03.list` holds the PETSc line and then +petsc4py. No module is listed in two shards or in a shard and the +prelude. + +``` +Hypre-3.1.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +SuperLU_DIST-9.2.1-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +PETSc-3.25.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +petsc4py-3.25.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +HeFFTe-2.4.1-foss-2026.1.eb +CheMPS2-1.8.12-foss-2026.1.eb +``` + +Without `--commit-cache`, petsc4py is planned on its own. The summary +is `shards=4 prelude=173 lines=34 opaque=1`, and petsc4py is in a +different shard from PETSc. The same inputs with the cache put them in +one shard, PETSc first. A second run with `--generated-at` writes the +same files. + # Plan software-layer pull requests that do not overlap What it does: `stack batch` plans European Environment for Scientific @@ -6179,8 +6271,8 @@ Flags and exit status stay in the [command-line reference](https://eb-stack.rgos # Command-line reference -`eb-stack` has six namespaced command groups (`package`, `recipe`, -`stack`, `target`, `campaign`, `repro`), four reports (`easyblock-reach`, +`eb-stack` has seven namespaced command groups (`package`, `recipe`, +`stack`, `target`, `campaign`, `repro`, `ledger`), four reports (`easyblock-reach`, `pr-overlap`, `rebuild-set`, `component-reach`), and a Model Context Protocol (MCP) server. `--version` prints the crate version. All paths written by package commands are deterministic bundle artifacts. Closure, generation, baseline, @@ -6195,7 +6287,7 @@ reachability, toolchain family, and the claim ladder are defined in | 1 | The work was attempted and failed: a missing or unreadable file, a document that does not parse, a check that found errors, a network fetch that failed, a doctor probe, or a campaign whose build failed | | 2 | The command line was rejected before that work: a missing or unknown flag (clap), or a flag value or combination the command does not accept | -A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails `recipe lint` or `recipe check` is status 1: the files were read and the check failed. `stack from-commit` is status 1 when any pin is `released`, and status 0 when every pin is `newer` or `unknown` or the files name no pin. A file name with no `eb-X.Y.Z`, and no `--easybuild-version`, is status 2. `stack arch` is status 1 when an entry is unsupported or flagged, or when `--probe-sources` reports `unreachable` or `arch-missing` sources. `stack files` is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are `no_url` or `unchecked`. `stack generation` is status 1 on any finding, and status 2 when an easystack path names no EESSI version and `--eessi-version` is absent. `stack known-failures` is status 0 when it prints warnings: they are advice for the pull request, not a refusal; with `--strict` it is status 1 when a warning remains, and status 0 when every hit is remedied. `stack oom` is status 0 when it warns: the warnings are advice, not a failed check. A `--node` value that is not `CORES:MEMORY_MIB` or `CORES:MEMORY_MIB:TARGET` is status 2, and so is a command with none of `--easystack`, `--node`, and `--hooks`, or a `--parallel` or `--max-parallel` of 0. A `--hooks` file that is not valid Python, has no module-level `PARALLELISM_LIMITS`, or has a row the model cannot follow is status 1, and the message names the line. `stack bot-classify` is status 0 for every class, and status 2 when a path is a directory. `stack batch` is status 1 when the plan includes, or depends on, an entry the architecture cannot build. `stack provided` is status 1 when a build-list line names software the baseline already provides, names no package, uses a toolchain other than 2026.1 or GCCcore-15.2.0 below it, sets `--hooks`, or belongs in a `--family-lock`. A group-restricted warning is status 0 unless `--strict`. Status 2 is when neither `--installed` nor `--baseline-lock` is set. `stack shard` is status 1 when a build-list line cannot be resolved, and writes nothing; `--shards 0` and a list with no lines are status 2. `stack shadow` is status 1 when a site-layer recipe shadows a base-stack module, or when a recipe does not parse, and status 0 when every recipe names software the base does not provide. +A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails `recipe lint` or `recipe check` is status 1: the files were read and the check failed. `stack from-commit` is status 1 when any pin is `released`, and status 0 when every pin is `newer` or `unknown` or the files name no pin. A file name with no `eb-X.Y.Z`, and no `--easybuild-version`, is status 2. `stack arch` is status 1 when an entry is unsupported or flagged, or when `--probe-sources` reports `unreachable` or `arch-missing` sources. `stack files` is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are `no_url` or `unchecked`. `stack generation` is status 1 on any finding, and status 2 when an easystack path names no EESSI version and `--eessi-version` is absent. `stack known-failures` is status 0 when it prints warnings: they are advice for the pull request, not a refusal; with `--strict` it is status 1 when a warning remains, and status 0 when every hit is remedied. `stack oom` is status 0 when it warns: the warnings are advice, not a failed check. A `--node` value that is not `CORES:MEMORY_MIB` or `CORES:MEMORY_MIB:TARGET` is status 2, and so is a command with none of `--easystack`, `--node`, and `--hooks`, or a `--parallel` or `--max-parallel` of 0. A `--hooks` file that is not valid Python, has no module-level `PARALLELISM_LIMITS`, or has a row the model cannot follow is status 1, and the message names the line. `stack bot-classify` is status 0 for every class, and status 2 when a path is a directory. `stack batch` is status 1 when the plan includes, or depends on, an entry the architecture cannot build. `stack provided` is status 1 when a build-list line names software the baseline already provides, names no package, uses a toolchain other than 2026.1 or GCCcore-15.2.0 below it, sets `--hooks`, or belongs in a `--family-lock`. A group-restricted warning is status 0 unless `--strict`. Status 2 is when neither `--installed` nor `--baseline-lock` is set. `stack shard` is status 1 when a build-list line cannot be resolved and is not fetched by EasyBuild, or when a `--commit-cache` download fails before an HTTP status, and writes nothing in either case. `--shards 0`, a list with no lines, and a `--commit-tree` or `--pr-tree` that is not `COMMIT=DIR` or `N=DIR` are status 2. `stack shadow` is status 1 when a site-layer recipe shadows a base-stack module, or when a recipe does not parse, and status 0 when every recipe names software the base does not provide. `ledger verify` is status 1 when a proof or a head signature does not check, and status 2 without exactly one of `--record` with `--head` and `--from` with `--to`. `ledger append` is status 1 when the record is outside the format, the log no longer reads, or the log is not at `--expect-head`; it then writes nothing. `--help` for every command has an example and a description of every flag. A default that the command applies is printed in that help. A directory that receives several artifacts is `--out-dir`. A single output file is `--out` (`--lock-out` for the lock `stack solve` writes). `--installed`, `--easyconfigs`, `--filter-deps`, and `--generated-at` use those names on every command that takes them. @@ -6239,6 +6331,10 @@ Package commands take the toolchain as `--toolchain name/version`, the spelling | `campaign run` / `campaign status` | Execute or inspect a persisted build campaign | | `campaign finding claim` | Claim an open finding for one worker | | `campaign finding resolve` | Resolve a claimed finding with an action, evidence, and the paths it changed | +| `ledger append` | Append one event record to a per-stack ledger as a canonical JSON line and sign the new head | +| `ledger head` | Sign a ledger's current head, for a published anchor | +| `ledger show` | List ledger records newest first, for one module or software name | +| `ledger verify` | Check that a record is in a signed head, or that one signed head extends another | | `mcp [--workspace]` | Serve each CLI workflow over stdio, including `pr-overlap`, `rebuild-set`, `component-reach`, and `easyblock-reach`, with recipe file tools rooted at `--workspace` | | `easyblock-reach` | Report easyconfigs and installed modules reached by an easyblock or framework change | @@ -6670,6 +6766,7 @@ eb-stack stack batch \ eb-stack stack shard \ --build-list NODE-TYPE.list \ --easyconfigs TREE [--easyconfigs OVERLAY]... \ + [--commit-cache DIR] [--commit-tree COMMIT=DIR]... [--pr-tree N=DIR]... \ [--installed-modules-dir MODULES/all]... [--installed INSTALLED.txt]... \ --shards 4 \ [--weights SECONDS.tsv] [--default-weight 1] \ @@ -7001,6 +7098,28 @@ a fraction rounds up), or `--default-weight` (1) when the file does not list it. `--filter-deps`, or `EASYBUILD_FILTER_DEPS` when the flag is absent, drops dependencies the way EasyBuild does. +A line with `--from-commit=SHA` or `--from-pr=N` is resolved the way +EasyBuild resolves it: that commit's or pull request's easyconfigs first, +then the robot path. `--commit-tree COMMIT=DIR` names the easyconfigs +directory of one commit (an abbreviated commit matches) and +`--pr-tree N=DIR` the easyconfigs one pull request changes; both repeat. +`--commit-cache DIR` holds every other tree. A commit missing there is +downloaded over https from +`codeload.github.com/easybuilders/easybuild-easyconfigs/tar.gz/` +plus the commit. That address also serves a fork commit a pull request +carries, and only its `.eb` files under `easybuild/easyconfigs/` are +kept, at `DIR/SHA/easybuild/easyconfigs`. +A pull request's head commit and changed easyconfigs come from +`api.github.com` and are kept at `DIR/pr-N-HEAD`, so a pull request that +moves is read again. `GITHUB_TOKEN` or `GH_TOKEN` is sent to +`api.github.com` when set. A tree is written under a temporary name and +renamed into place, so several planners can share one cache. A module the +closures of two or more shards need goes to the prelude whichever tree it +comes from; a prelude line for a module only a fetched tree provides +carries that line's `--from-commit`, `--from-pr` and +`--include-easyblocks-from-*` options, so it builds the same easyconfig +with the same easyblock. + `--out-dir` gets `prelude.list` (the prelude as easyconfig filenames in dependency order; empty when nothing is shared), `shard-NN.list` (the input lines verbatim, dependencies first), and `plan.json` (each line's closure, @@ -7008,11 +7127,26 @@ group and shard, the weight of every shard and of the prelude, and a SHA-256 over the build list, the weights, the installed modules and every easyconfig a closure reached). A `shard-NN.list` that an earlier run left and this run does not write is removed. `--generated-at` fixes the time in -`plan.json`, so the same inputs give identical files. A line whose -easyconfig is not on the robot path, whose first word is not an `.eb` -file, or whose closure has a requirement nothing satisfies exits 1, names -every such line, and writes nothing. The command prints -`shards= prelude= lines=`. +`plan.json`, so the same inputs give identical files. A +`--commit-cache` download that fails before an HTTP status (the cache has +no tree and the client never gets a status) exits 1 and writes nothing; +the message names the cache, the commit, the URL, and the transport +error. An HTTP error status is not that case. A fetched line +that still cannot be resolved (no tree for its commit, its easyconfig in +neither its tree nor the robot path, or a requirement neither provides), +and a line whose easyconfig is on the robot path but does not parse, is +planned as an opaque line: its own group, no closure, the default weight, +written to its shard unchanged. `plan.json` gives the kind under each +line's `opaque`, the cause under `opaque_reason`, and every opaque line +under the top-level `opaque`; `inputs.trees` maps each fetched source to +`null` when its tree was read or to the reason it was not. Each opaque line +and each other caveat (a fetched line planned over the robot path only, a +module two trees resolve differently) is printed to stderr as +`warning: ...` and kept under `warnings`. Any other line whose easyconfig is +not on the robot path, whose first word is not an `.eb` file, or whose +closure has a requirement nothing satisfies exits 1, names every such line, +and writes nothing. The command prints `shards= prelude= lines=`, +followed by `opaque=` when a line is opaque. ## `stack arch` @@ -7887,6 +8021,116 @@ independently. See [campaign state and typed findings](https://eb-stack.rgoswami.me/campaigns.html.md) for the serialized state, claim ladder, class/disposition table, and repair protocol. +# Ledger commands + +```bash +eb-stack ledger append RECORD.json \ + --log ledger/2026.06.jsonl \ + [--expect-head HEAD.json] \ + [--key KEY.pem --head-out HEAD.json] + +eb-stack ledger head \ + --log ledger/2026.06.jsonl --key KEY.pem --out HEAD.json [--ts RFC3339] + +eb-stack ledger show --log ledger/2026.06.jsonl [--package NAME] [--json] + +eb-stack ledger verify \ + --log ledger/2026.06.jsonl --public-key KEY.pub \ + --record N --head HEAD.json + +eb-stack ledger verify \ + --log ledger/2026.06.jsonl --public-key KEY.pub \ + --from HEAD1.json --to HEAD2.json +``` + +A ledger is one append-only JSON Lines file per stack. Each line is one +event on one installation, and every line of a file has the same `stack`. +The file is an RFC 9162 Merkle log over the `rfc9162` crate: the leaf of +record `N` is `SHA-256(0x00 || line N)`, over the bytes of the line without +its newline, and an interior node is `SHA-256(0x01 || left || right)`. + +**Canonical form.** A record is a JSON object written as one line: keys +sorted by their UTF-8 bytes at every depth, no whitespace outside strings, +strings escaped as `serde_json` escapes them (`"`, `\` and control +characters; other characters as UTF-8), numbers as integers only. A record +with a fractional number is refused. Two files that differ only in key +order and spacing append the same line. + +**Fields.** `append` refuses a field outside this table. + +| Field | Content | +| --- | --- | +| `ts` | UTC time, RFC 3339 (`Z` or `+00:00`). `append` fills the current time when absent | +| `actor` | Who wrote it: a Jenkins run URL, a bot pull request and job, or a login | +| `stack` | `2026.06` (EESSI version) or `2025` | +| `target` | CPU target and accelerator, e.g. `x86_64/amd/zen4` | +| `module` | Module name, e.g. `PETSc/3.25.0-foss-2026.1` | +| `location` | The installation path that RPATHs use | +| `action` | `install`, `rebuild-inplace`, `rebuild-bump`, `retire`, `unretire`, `move`, `access`, `remove`, `finding`, or `correction` | +| `stage` | Optional: `started`, `finished`, or `aborted`, for an operation that runs in steps | +| `reason`, `easyconfig`, `sources`, `sbom`, `modulefile`, `build` | Optional JSON: free text or an object of references and SHA-256 sums | +| `supersedes`, `superseded_by`, `moved_from`, `moved_to`, `corrects` | Optional `id` of a related record. A `correction` needs `corrects` | +| `index` | Leaf number. Written by `append`; a record that carries another value is refused | +| `id` | SHA-256 of the canonical record without `id` and `sig`. Written by `append`; a record that carries another value is refused | +| `sig` | Optional per-record signature, kept in the line and left out of `id` | + +The first fields are required non-empty strings: `ts`, `actor`, `stack`, +`target`, `module`, `location`, and `action`. A wrong record is never edited: +a `correction` record follows it. Licence numbers, keys and tokens do not +belong in a record. + +**Reading a log.** `show`, `verify`, `head` and `append` read every line and +refuse the log when a line is not canonical, its `index` is not its line +number, its `id` does not recompute, its `stack` differs from the first +line, or the last line has no newline. + +**Heads.** A head file is one JSON line: `domain` (always +`eb-stack-ledger-v1`), `size`, `root` (hex), `ts`, `public_key` (hex), and +`signature` (hex). The Ed25519 signature covers +`eb-stack-ledger-v1sizerootts`. `--key` is an Ed25519 PKCS#8 +PEM file, as `openssl genpkey -algorithm ed25519` writes it. A key inside the +log's directory is refused, so the key is always an argument and never a +file beside the log. `--public-key` is the matching PEM (~openssl pkey +-pubout`), or the 64 hex characters of a head's `public_key~. + +`append` locks the log file, checks the log, refuses the record when the +log is not at the size and root of `--expect-head`, writes the line, and, +with `--key` and `--head-out`, writes the signed head after the append. It +prints `appended record N id ID; head size S root ROOT`. `head` signs the +log's current head into `--out`, at `--ts` or now. + +**Locking and crashes.** Writers take an exclusive POSIX record lock on +`LOG.posix-lock` beside the log; `head` takes a shared lock on the same +file, so a head is the log between two appends and an older head never +replaces a newer one. `head` opens the log and the lock file read-only and +needs no write access to the ledger directory, but refuses a log whose +lock file no writer has created. `append` flushes the log and its +directory before it reports a record, including a record it acknowledges +as already recorded. A write that fails part way is cut back. Bytes after +the last newline that are not a JSON value, left by an append that never +finished, are copied to a new `LOG.quarantine.*` file and flushed before +the next `append` cuts them from the log; `append` then adds +`; moved N bytes of an interrupted append to PATH` to its line. A complete +JSON value without its newline is refused and left in place. A head file is +replaced by renaming a flushed new file over it. A head path that names +the log, its lock file, a quarantine file, or the signing key, directly or +through a symbolic or hard link, is refused, and so is a head path that is +a symbolic link. A signing key with more than one hard link is refused. The +lock is a POSIX record lock: whether it excludes writers on several hosts +depends on the shared file system's lock support. `show` prints `index`, +`ts`, `action`, `target`, `module` and `actor` per record, newest first, then +`K of N records`. `--package` matches a whole module name, or the software +name before its `/`. `--json` prints the records as a JSON array. + +`verify --record N --head HEAD` checks HEAD's signature against +`--public-key`, computes the inclusion proof of record N from the first +`size` lines, and checks it against HEAD's root. A record removed from the +log, or any line changed, makes the root differ. ~verify --from HEAD1 --to +HEAD2~ checks both signatures, computes the consistency proof from HEAD1's +size to HEAD2's size, and checks that it yields both roots. A log rewritten +under HEAD1 and signed again fails, because no proof yields the old root +and the new one. + # `repro summary` ```bash @@ -7976,7 +8220,9 @@ plus sandboxed recipe and easyblock tools: - `eb_pr_overlap` (the same writer as `pr-overlap`, including `--kind easyconfigs`, a merge order, and `--impact`), `eb_rebuild_set`, `eb_component_reach`, `eb_easyblock_reach`; -- `eb_repro_summary`. +- `eb_repro_summary`; +- `eb_ledger_append`, `eb_ledger_head`, `eb_ledger_show`, + `eb_ledger_verify`. `eb_recipe_read`, `eb_recipe_list`, `eb_recipe_edit`, `eb_recipe_write`, and `eb_recipe_grep` resolve every path inside that root. A `..` component, an @@ -8065,7 +8311,14 @@ that is not an integer, and `hook_warnings` for a dropped sum in the `oom_profile`, and `oom_resources`. It refuses an unsupported plan the same way and writes known-failures and oom warnings. `eb_repro_summary` -requires `scores` and takes `ratchet` and `json`. Each of these tools +requires `scores` and takes `ratchet` and `json`. +`eb_ledger_append` requires `log` and `record` and takes `expect_head`, +`key` and `head_out`; it also returns `index`, `id`, `size`, `root` and the +signed head. `eb_ledger_head` requires `log`, `key` and `out` and takes `ts`. +`eb_ledger_show` requires `log` and takes `package` and `json`, and returns +the records. `eb_ledger_verify` requires `log` and `public_key`, and either +`record` (an integer) with `head`, or `from` with `to`; it returns the proof +nodes, and a proof that does not check is a tool error. Each of these tools calls the same runner as the command of that name, and the text it returns is that command's stdout. @@ -8191,7 +8444,7 @@ absolute path there. # Tools -The catalog has 51 tools. `tools/list` returns one tool for each CLI +The catalog has 55 tools. `tools/list` returns one tool for each CLI command in the table, and the recipe and easyblock tools that have no command. `mcp` publishes that catalog and has no tool of its own. Each command's tool calls the same runner as that command. The `text` field @@ -8236,6 +8489,10 @@ name in the schema `tools/list` returns. The | `campaign finding claim` | `eb_campaign_finding_claim` | | `campaign finding resolve` | `eb_campaign_finding_resolve` | | `repro summary` | `eb_repro_summary` | +| `ledger append` | `eb_ledger_append` | +| `ledger head` | `eb_ledger_head` | +| `ledger show` | `eb_ledger_show` | +| `ledger verify` | `eb_ledger_verify` | | `easyblock-reach` | `eb_easyblock_reach` | | `pr-overlap` | `eb_pr_overlap` | | `rebuild-set` | `eb_rebuild_set` | @@ -12155,7 +12412,7 @@ table. `reverse=0` again, for the same reason as the CMakeMake run. # The same document over MCP -The Model Context Protocol (MCP) catalog has 51 tools. `tools/list` prints +The Model Context Protocol (MCP) catalog has 55 tools. `tools/list` prints their names: ```bash @@ -12216,9 +12473,13 @@ eb_pr_overlap eb_rebuild_set eb_easyblock_reach eb_repro_summary +eb_ledger_append +eb_ledger_head +eb_ledger_show +eb_ledger_verify ``` -That is 51 tools. +That is 55 tools. `eb_stack_sbom` writes the CycloneDX file `stack sbom` writes. Its `tools/list` description reads: "Emit CycloneDX from an existing stack lock, reading dependency edges and source checksums from the easyconfigs the lock names. With backport_list, also write the modules whose reach class justifies a backport." `eb_stack_sbom` reads the Cbc lock: diff --git a/llms.txt b/llms.txt index 07defad7..5e8c27b6 100644 --- a/llms.txt +++ b/llms.txt @@ -2,7 +2,7 @@ > eb-stack parses conda-forge, Spack, PyPI, CRAN, Cargo, LuaRocks, Raku, and EasyBuild recipes into one package plan, a planned CycloneDX SBOM, Resolvo locks, and conventional EasyBuild recipes. -The binary is `eb-stack`. The crate is `eb_stack`. Rust 1.88 or newer builds it from source. Each link below is the markdown form of a page on https://eb-stack.rgoswami.me/. `llms-full.txt` repeats those pages in one file. Contributor instructions are in AGENTS.md. The Model Context Protocol server is `eb-stack mcp`. +The binary is `eb-stack`. The crate is `eb_stack`. Rust 1.88 or newer builds it from source. Each link below is the markdown form of a page on https://eb-stack.rgoswami.me/. `llms-full.txt` repeats those pages in one file. Contribution guidelines are in CONTRIBUTING.md. The Model Context Protocol server is `eb-stack mcp`. ## Getting started @@ -50,7 +50,7 @@ The binary is `eb-stack`. The crate is `eb_stack`. Rust 1.88 or newer builds it ## Reference - [Glossary](https://eb-stack.rgoswami.me/reference/glossary.html.md): Names shared by the tutorial, the how-to guides, and the reference. -- [Command-line reference](https://eb-stack.rgoswami.me/reference/cli.html.md): eb-stack has six namespaced command groups (package, recipe, stack, target, campaign, repro), four reports (easyblock-reach, pr-overlap, rebuild-set, component-reach), and a Model Context Protocol (MCP) server. +- [Command-line reference](https://eb-stack.rgoswami.me/reference/cli.html.md): eb-stack has seven namespaced command groups (package, recipe, stack, target, campaign, repro, ledger), four reports (easyblock-reach, pr-overlap, rebuild-set, component-reach), and a Model Context Protocol (MCP) server. - [Model Context Protocol server](https://eb-stack.rgoswami.me/reference/mcp.html.md): eb-stack mcp serves the version-one commands on stdin and stdout. - [Package bundle reference](https://eb-stack.rgoswami.me/reference/package-bundles.html.md): package inspect, package plan, and package bump write a version-one package bundle. - [Build target configuration](https://eb-stack.rgoswami.me/reference/targets.html.md): A target routes one EasyBuild workload through four independent layers: transport, executor, runtime, and EasyBuild settings. @@ -91,8 +91,8 @@ The binary is `eb-stack`. The crate is `eb_stack`. Rust 1.88 or newer builds it ## Project -- [README](https://github.com/HaoZeke/eb-stack/blob/main/README.md): Install, the claim ladder, llms.txt, AGENTS.md, the skills, and the Model Context Protocol server. -- [AGENTS.md](https://github.com/HaoZeke/eb-stack/blob/main/AGENTS.md): Build, test, lint, and documentation commands for a contributor. +- [README](https://github.com/HaoZeke/eb-stack/blob/main/README.md): Install, the claim ladder, the skills, and the Model Context Protocol server. +- [Contributing](https://github.com/HaoZeke/eb-stack/blob/main/CONTRIBUTING.md): Contribution guidelines and validation commands. - [MCP server manifest](https://github.com/HaoZeke/eb-stack/blob/main/server.json): Registry entry for the eb-stack Model Context Protocol server. ## Optional diff --git a/scripts/gen-llms-txt.py b/scripts/gen-llms-txt.py index b3509b58..0ed3147d 100644 --- a/scripts/gen-llms-txt.py +++ b/scripts/gen-llms-txt.py @@ -373,7 +373,7 @@ def render_index(pages: list[dict[str, str]]) -> str: "The binary is `eb-stack`. The crate is `eb_stack`. Rust 1.88 or " "newer builds it from source. Each link below is the markdown form " "of a page on https://eb-stack.rgoswami.me/. `llms-full.txt` repeats " - "those pages in one file. Contributor instructions are in AGENTS.md. " + "those pages in one file. Contribution guidelines are in CONTRIBUTING.md. " "The Model Context Protocol server is `eb-stack mcp`.", "", ] @@ -422,12 +422,12 @@ def project_entries() -> list[str]: ( "README", f"{REPO}/README.md", - "Install, the claim ladder, llms.txt, AGENTS.md, the skills, and the Model Context Protocol server.", + "Install, the claim ladder, the skills, and the Model Context Protocol server.", ), ( - "AGENTS.md", - f"{REPO}/AGENTS.md", - "Build, test, lint, and documentation commands for a contributor.", + "Contributing", + f"{REPO}/CONTRIBUTING.md", + "Contribution guidelines and validation commands.", ), ] if (ROOT / "server.json").is_file(): diff --git a/src/eb_parse.rs b/src/eb_parse.rs index 1e235b0e..be6a9585 100644 --- a/src/eb_parse.rs +++ b/src/eb_parse.rs @@ -131,6 +131,9 @@ pub struct ResolvedEasyconfig { /// easyblock is called. #[serde(default)] pub has_components: bool, + /// Every parsed component source, patch, and postinstall patch carries a checksum. + #[serde(default)] + pub component_checksums_complete: bool, } impl ResolvedEasyconfig { @@ -2883,6 +2886,11 @@ fn resolve_easyconfig_str_inner(src: &str, arch: &str) -> Result Result } } +fn component_checksums_complete( + env: &HashMap, + templates: &HashMap, +) -> bool { + let Some(raw) = env.get("components") else { + return false; + }; + let Ok(items) = value_list_as_slice(Some(raw)) else { + return false; + }; + let entries = sub_entries(Some(raw), templates); + if items.is_empty() || entries.len() != items.len() { + return false; + } + let defaults = match env.get("default_component_specs") { + None => Vec::new(), + Some(Value::Dict(values)) => values.clone(), + _ => return false, + }; + entries.into_iter().all(|(_, _, options)| { + let mut fields: HashMap = defaults.iter().cloned().collect(); + match options { + None => {} + Some(Value::Dict(values)) => fields.extend(values), + _ => return false, + } + let count = |key| { + value_list_as_slice(fields.get(key)) + .ok() + .map(<[Value]>::len) + }; + // EasyBuild's check_checksums_for counts sources, then patches, then + // postinstallpatches. A component dict may spell that last list + // post_install_patches, and only when postinstallpatches is absent. + let postinstall_key = if fields.contains_key("postinstallpatches") { + "postinstallpatches" + } else { + "post_install_patches" + }; + let (Some(sources), Some(patches), Some(postinstall)) = + (count("sources"), count("patches"), count(postinstall_key)) + else { + return false; + }; + let expected = sources + patches + postinstall; + let Ok(checksums) = value_list_as_slice(fields.get("checksums")) else { + return false; + }; + expected > 0 + && checksums.len() == expected + && checksums.iter().all(|entry| { + let tokens = checksum_strings_from_value(entry); + !tokens.is_empty() + && tokens.iter().all(|token| { + matches!(token.len(), 32 | 40 | 64 | 128) + && token.bytes().all(|byte| byte.is_ascii_hexdigit()) + }) + }) + }) +} + /// Packaging gate: checksums present, moduleclass set, and optional required /// configopts substrings (e.g. `-Dwith_tests=false`). /// @@ -3942,7 +4012,11 @@ pub fn packaging_gate( recipe.easyblock.as_deref(), Some("PythonBundle" | "RPackage" | "Bundle") ) && !recipe.exts_list.is_empty(); - if !extension_bundle { + let component_bundle = recipe.easyblock.as_deref() == Some("Bundle") + && recipe.sources_count == 0 + && recipe.patch_names.is_empty() + && recipe.component_checksums_complete; + if !extension_bundle && !component_bundle { errs.push("missing checksums".into()); } } @@ -5146,6 +5220,7 @@ dependencies = [('Java', '21', '', SYSTEM)]\nmoduleclass = 'devel'\n"; checksum_entry_keys: vec![], checksums_by_filename: Default::default(), has_components: false, + component_checksums_complete: false, }; let c = resolved.to_candidate(); let cuda = c.dependencies.iter().find(|d| d.name == "CudaLib").unwrap(); @@ -5857,6 +5932,7 @@ builddependencies = [ checksum_entry_keys: vec![], checksums_by_filename: Default::default(), has_components: false, + component_checksums_complete: false, } } diff --git a/src/eb_style/mod.rs b/src/eb_style/mod.rs index 482044f1..212d0e30 100644 --- a/src/eb_style/mod.rs +++ b/src/eb_style/mod.rs @@ -121,8 +121,7 @@ pub fn valid_easyconfig_parameters(easyblock: Option<&str>) -> BTreeSet /// edits (pre-existing) and no review line or file-wide scope covers it; /// `error_free` still reports it. pub fn blocks_clean(finding: &StyleFinding) -> bool { - (finding.severity == Severity::Error && !finding.info) - || (is_formatting_code(&finding.code) && !finding.info) + (!finding.info && (finding.severity == Severity::Error || is_formatting_code(&finding.code))) || (finding.in_review && REVIEW_COUNTED.contains(&finding.code.as_str())) || (finding.code == "EB_DIFF_SOURCES_CHANGED" && !finding.info) || (finding.code == "EB_REVIEW_ITEM_OPEN" && !finding.info) @@ -801,6 +800,16 @@ impl fmt::Display for StyleFinding { mod tests { use super::*; + #[test] + fn build_parallelism_limit_is_a_generic_parameter() { + let findings = lint_style("easyblock = 'MakeCp'\nmax_parallel = 1\n"); + assert!(findings_clean(&findings), "{findings:?}"); + let typo = lint_style("easyblock = 'MakeCp'\nmax_paralell = 1\n"); + assert!(typo + .iter() + .any(|finding| finding.code == "EB_LOCAL_VAR_NAMING")); + } + #[test] fn lint_empty_ok() { assert!(lint_style("").is_empty()); diff --git a/src/eb_style/vocab.rs b/src/eb_style/vocab.rs index 16e9370a..6f4aa99a 100644 --- a/src/eb_style/vocab.rs +++ b/src/eb_style/vocab.rs @@ -65,6 +65,7 @@ pub(super) const DEFAULT_PARAMS: &[&str] = &[ "license_file", "license_server", "license_server_port", + "max_parallel", "maxparallel", "modaliases", "modaltsoftname", diff --git a/src/ledger.rs b/src/ledger.rs new file mode 100644 index 00000000..64e62567 --- /dev/null +++ b/src/ledger.rs @@ -0,0 +1,1630 @@ +//! Per-stack software ledger: one append-only JSON Lines file of install, +//! rebuild, retire, move, access, removal, finding and correction events, +//! hashed as an RFC 9162 Merkle log. +//! +//! Each line is the canonical JSON of one record: keys sorted by their UTF-8 +//! bytes, no whitespace outside strings, integers only. The leaf of record +//! `N` is [`rfc9162::leaf_hash`] over the bytes of line `N` without its +//! newline. A head is the size and root over the first `size` leaves, signed +//! with Ed25519 under [`DOMAIN`]. An inclusion proof shows one record is in a +//! signed head; a consistency proof shows a later head extends an earlier +//! one, which a rewritten log cannot. +//! +//! The signing key is a path the writer passes on each call. A key inside +//! the log's directory is refused, so the key never travels with the log. + +use anyhow::{anyhow, bail, Context, Result}; +use chrono::{DateTime, SecondsFormat, Utc}; +use ed25519_dalek::pkcs8::{DecodePrivateKey, DecodePublicKey}; +use ed25519_dalek::{Signature, Signer, SigningKey, VerifyingKey}; +use fs2::FileExt; +use serde::{Deserialize, Serialize}; +use serde_json::{Map, Value}; +use sha2::{Digest, Sha256}; +use std::ffi::OsString; +use std::fmt::Write as _; +use std::fs::{File, OpenOptions}; +use std::io::{Read, Seek, SeekFrom, Write}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; + +/// The name a head signature covers, so a head of this log is not a head of +/// any other log that signs RFC 9162 heads. +pub const DOMAIN: &str = "eb-stack-ledger-v1"; + +/// The events a record can carry. +pub const ACTIONS: &[&str] = &[ + "install", + "rebuild-inplace", + "rebuild-bump", + "retire", + "unretire", + "move", + "access", + "remove", + "finding", + "correction", +]; + +/// Stages of an operation that runs in steps. +pub const STAGES: &[&str] = &["started", "finished", "aborted"]; + +/// Fields every record carries, each a non-empty string. +const REQUIRED: &[&str] = &[ + "ts", "actor", "stack", "target", "module", "location", "action", +]; + +/// Fields that name another record by its `id`. +const LINKS: &[&str] = &[ + "supersedes", + "superseded_by", + "moved_from", + "moved_to", + "corrects", +]; + +/// Fields with structured content: free text or an object of references. +const STRUCTURED: &[&str] = &[ + "reason", + "easyconfig", + "sources", + "sbom", + "modulefile", + "build", +]; + +/// Fields the ledger writes: `id` and `index`. `sig` is an optional record +/// signature, kept in the line and left out of `id`. +const DERIVED: &[&str] = &["id", "index", "sig"]; + +/// The canonical JSON of `value`: object keys sorted by their UTF-8 bytes, +/// no whitespace outside strings, strings escaped as `serde_json` writes +/// them. A number that is not an integer is refused, because two writers +/// can print the same float two ways. +pub fn canonical(value: &Value) -> Result { + let mut out = String::new(); + write_canonical(value, &mut out)?; + Ok(out) +} + +fn write_canonical(value: &Value, out: &mut String) -> Result<()> { + match value { + Value::Null => out.push_str("null"), + Value::Bool(flag) => out.push_str(if *flag { "true" } else { "false" }), + Value::Number(number) => { + if number.is_f64() { + bail!("ledger records hold integers only, not {number}"); + } + out.push_str(&number.to_string()); + } + Value::String(text) => out.push_str(&serde_json::to_string(text)?), + Value::Array(items) => { + out.push('['); + for (at, item) in items.iter().enumerate() { + if at > 0 { + out.push(','); + } + write_canonical(item, out)?; + } + out.push(']'); + } + Value::Object(map) => { + let mut keys = map.keys().collect::>(); + keys.sort(); + out.push('{'); + for (at, key) in keys.into_iter().enumerate() { + if at > 0 { + out.push(','); + } + out.push_str(&serde_json::to_string(key)?); + out.push(':'); + write_canonical(&map[key], out)?; + } + out.push('}'); + } + } + Ok(()) +} + +fn sha256_hex(bytes: &[u8]) -> String { + let digest = Sha256::digest(bytes); + let mut text = String::with_capacity(64); + for byte in digest { + write!(&mut text, "{byte:02x}").expect("writing to a String cannot fail"); + } + text +} + +/// The record `id`: SHA-256 of the canonical JSON of the record without +/// `id` and `sig`. `index` is inside it, so a record moved to another +/// position has another id. +pub fn record_id(record: &Map) -> Result { + let mut body = record.clone(); + body.remove("id"); + body.remove("sig"); + Ok(sha256_hex(canonical(&Value::Object(body))?.as_bytes())) +} + +fn event_id(record: &Map) -> Result> { + match record.get("build").and_then(|build| build.get("event_id")) { + Some(Value::String(id)) if !id.trim().is_empty() => Ok(Some(id)), + Some(_) => bail!("ledger build.event_id must be a non-empty string"), + None => Ok(None), + } +} + +/// Check a record's fields against the ledger format. `id` and `index` are +/// not checked here. +pub fn check_fields(record: &Map) -> Result<()> { + for key in record.keys() { + let known = REQUIRED.contains(&key.as_str()) + || LINKS.contains(&key.as_str()) + || STRUCTURED.contains(&key.as_str()) + || DERIVED.contains(&key.as_str()) + || key == "stage"; + if !known { + bail!("unknown ledger field {key:?}"); + } + } + for field in REQUIRED { + match record.get(*field) { + Some(Value::String(text)) if !text.is_empty() => {} + Some(_) => bail!("ledger field {field} must be a non-empty string"), + None => bail!("ledger record has no {field}"), + } + } + let action = record["action"].as_str().unwrap_or_default(); + if !ACTIONS.contains(&action) { + bail!( + "ledger action {action:?} is not one of {}", + ACTIONS.join(", ") + ); + } + let ts = record["ts"].as_str().unwrap_or_default(); + let parsed = DateTime::parse_from_rfc3339(ts) + .with_context(|| format!("ledger ts {ts:?} is not RFC 3339"))?; + if parsed.offset().local_minus_utc() != 0 { + bail!("ledger ts {ts:?} is not UTC"); + } + for field in LINKS.iter().copied().chain(["stage", "sig"]) { + if let Some(value) = record.get(field) { + if value.as_str().is_none_or(|text| text.is_empty()) { + bail!("ledger field {field} must be a non-empty string"); + } + } + } + if let Some(stage) = record.get("stage").and_then(Value::as_str) { + if !STAGES.contains(&stage) { + bail!("ledger stage {stage:?} is not one of {}", STAGES.join(", ")); + } + } + if action == "correction" && !record.contains_key("corrects") { + bail!("a correction record names the record it corrects in corrects"); + } + event_id(record)?; + Ok(()) +} + +/// One line of the log. +#[derive(Debug, Clone)] +pub struct Entry { + /// Leaf number, which is also the record's `index`. + pub index: usize, + /// The record's `id`. + pub id: String, + /// The canonical line, without its newline. Its bytes are the leaf. + pub line: String, + /// The parsed record. + pub record: Map, +} + +impl Entry { + /// The RFC 9162 leaf hash of this record. + #[must_use] + pub fn leaf(&self) -> [u8; 32] { + rfc9162::leaf_hash(self.line.as_bytes()) + } + + /// The record's `module`. + #[must_use] + pub fn module(&self) -> &str { + self.record["module"].as_str().unwrap_or_default() + } +} + +/// Parse log text. Every line is checked: it is canonical, its `index` is its +/// line number, its `id` recomputes, its fields are the ledger's, and its +/// `stack` is the stack of the first line. +pub fn parse_log(text: &str) -> Result> { + if text.is_empty() { + return Ok(Vec::new()); + } + let Some(body) = text.strip_suffix('\n') else { + bail!("the last ledger line has no newline; the log ends in a partial write"); + }; + let mut entries = Vec::new(); + for (index, line) in body.split('\n').enumerate() { + let entry = parse_line(line, index).with_context(|| format!("ledger line {index}"))?; + if let Some(first) = entries.first() { + let first: &Entry = first; + if first.record["stack"] != entry.record["stack"] { + bail!( + "ledger line {index} is stack {} in a log of stack {}", + entry.record["stack"], + first.record["stack"] + ); + } + } + entries.push(entry); + } + Ok(entries) +} + +fn parse_line(line: &str, index: usize) -> Result { + let value: Value = serde_json::from_str(line).context("not JSON")?; + let Value::Object(record) = value else { + bail!("not a JSON object"); + }; + check_fields(&record)?; + if record.get("index").and_then(Value::as_u64) != Some(index as u64) { + bail!("index is {:?}, not {index}", record.get("index")); + } + let id = record_id(&record)?; + if record.get("id").and_then(Value::as_str) != Some(id.as_str()) { + bail!("id is {:?}; the record hashes to {id}", record.get("id")); + } + if canonical(&Value::Object(record.clone()))? != line { + bail!("not in canonical form"); + } + Ok(Entry { + index, + id, + line: line.to_string(), + record, + }) +} + +/// Read and check a log file. +pub fn read_log(path: &Path) -> Result> { + let text = + std::fs::read_to_string(path).with_context(|| format!("read ledger {}", path.display()))?; + parse_log(&text).with_context(|| format!("ledger {}", path.display())) +} + +/// The leaves of `entries`. +#[must_use] +pub fn leaves(entries: &[Entry]) -> Vec<[u8; 32]> { + entries.iter().map(Entry::leaf).collect() +} + +/// Turn a record into the log line for position `index`. A record that +/// already carries `index` or `id` must carry the values the ledger would +/// write; `ts` is filled with the current UTC time when absent. +pub fn seal_record(record: &Value, index: usize, now: &str) -> Result { + let Value::Object(record) = record else { + bail!("a ledger record is a JSON object"); + }; + let mut record = record.clone(); + let given_id = record.remove("id"); + if let Some(given) = record.get("index") { + if given.as_u64() != Some(index as u64) { + bail!("record says index {given}, but it would be line {index}"); + } + } + record.insert("index".to_string(), Value::from(index as u64)); + record + .entry("ts".to_string()) + .or_insert_with(|| Value::from(now)); + check_fields(&record)?; + let id = record_id(&record)?; + if let Some(given) = given_id { + if given.as_str() != Some(id.as_str()) { + bail!("record says id {given}, but it hashes to {id}"); + } + } + record.insert("id".to_string(), Value::from(id.clone())); + let line = canonical(&Value::Object(record.clone()))?; + Ok(Entry { + index, + id, + line, + record, + }) +} + +/// A signed tree head, as written to a head file. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct SignedHead { + /// Always [`DOMAIN`]. + pub domain: String, + /// Number of records the root covers. + pub size: usize, + /// Merkle root over the first `size` leaves, lower-case hex. + pub root: String, + /// UTC time the head was signed, RFC 3339. + pub ts: String, + /// Ed25519 public key of the signer, lower-case hex. + pub public_key: String, + /// Ed25519 signature over [`SignedHead::material`], lower-case hex. + pub signature: String, +} + +impl SignedHead { + /// The bytes the signature covers: [`rfc9162::Head::material`] under + /// [`DOMAIN`], a tab, and `ts`. + #[must_use] + pub fn material(size: usize, root: &str, ts: &str) -> Vec { + let head = rfc9162::Head { + size, + root: root.to_string(), + }; + let mut bytes = head.material(DOMAIN); + bytes.push(b'\t'); + bytes.extend_from_slice(ts.as_bytes()); + bytes + } + + /// Sign the head over `entries`. + #[must_use] + pub fn sign(entries: &[Entry], key: &SigningKey, ts: &str) -> Self { + let head = rfc9162::Head::over(&leaves(entries)); + let signature = key.sign(&Self::material(head.size, &head.root, ts)); + Self { + domain: DOMAIN.to_string(), + size: head.size, + root: head.root, + ts: ts.to_string(), + public_key: rfc9162::hex(key.verifying_key().as_bytes()), + signature: rfc9162::hex(&signature.to_bytes()), + } + } + + /// Check the domain, that the signer is `trusted`, and the signature. + pub fn check(&self, trusted: &VerifyingKey) -> Result<()> { + if self.domain != DOMAIN { + bail!("head domain is {:?}, not {DOMAIN}", self.domain); + } + let signer = rfc9162::hex(trusted.as_bytes()); + if self.public_key != signer { + bail!( + "head is signed by {}, not by the trusted key {signer}", + self.public_key + ); + } + let raw = decode_signature(&self.signature) + .ok_or_else(|| anyhow!("head signature is not 128 hex characters"))?; + trusted + .verify_strict( + &Self::material(self.size, &self.root, &self.ts), + &Signature::from_bytes(&raw), + ) + .map_err(|_| anyhow!("head signature does not verify")) + } + + /// Read a head file. + pub fn read(path: &Path) -> Result { + let text = std::fs::read_to_string(path) + .with_context(|| format!("read head {}", path.display()))?; + serde_json::from_str(&text).with_context(|| format!("parse head {}", path.display())) + } + + /// Write a head file: the head as JSON, one line. The file is replaced + /// atomically, so a reader sees the previous head or this one and a + /// failed write leaves the previous head in place. A symbolic link at + /// `path` is refused rather than replaced or followed. + pub fn write(&self, path: &Path) -> Result<()> { + let mut text = serde_json::to_string(self)?; + text.push('\n'); + replace_file(path, text.as_bytes()) + .with_context(|| format!("write head {}", path.display())) + } +} + +/// Write `bytes` to a new file beside `path`, flush it, rename it over +/// `path`, and flush the directory so the rename survives a crash. +fn replace_file(path: &Path, bytes: &[u8]) -> Result<()> { + static NEXT: AtomicU64 = AtomicU64::new(0); + let name = path + .file_name() + .ok_or_else(|| anyhow!("the path names no file"))?; + let directory = match path.parent() { + Some(parent) if !parent.as_os_str().is_empty() => parent, + _ => Path::new("."), + }; + if let Ok(existing) = std::fs::symlink_metadata(path) { + if existing.file_type().is_symlink() { + bail!("the path is a symbolic link; refusing to replace it"); + } + if existing.is_dir() { + bail!("the path is a directory"); + } + } + loop { + let mut temporary = OsString::from("."); + temporary.push(name); + temporary.push(format!( + ".{}.{}.tmp", + std::process::id(), + NEXT.fetch_add(1, Ordering::Relaxed) + )); + let temporary = directory.join(temporary); + let mut file = match OpenOptions::new() + .write(true) + .create_new(true) + .open(&temporary) + { + Ok(file) => file, + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => continue, + Err(error) => return Err(error).context("create a temporary file"), + }; + let written = file + .write_all(bytes) + .and_then(|()| file.sync_all()) + .context("write the temporary file") + .and_then(|()| std::fs::rename(&temporary, path).context("rename into place")); + drop(file); + if written.is_err() { + let _ = std::fs::remove_file(&temporary); + } + written?; + return sync_directory(directory); + } +} + +/// Flush a directory so entries created, renamed or removed in it survive a +/// crash. +fn sync_directory(directory: &Path) -> Result<()> { + #[cfg(unix)] + { + File::open(directory) + .and_then(|handle| handle.sync_all()) + .with_context(|| format!("flush directory {}", directory.display()))?; + } + #[cfg(not(unix))] + let _ = directory; + Ok(()) +} + +/// A 64 byte signature from 128 lower-case hex characters. +fn decode_signature(text: &str) -> Option<[u8; 64]> { + if text.len() != 128 || !text.is_ascii() { + return None; + } + let mut raw = [0u8; 64]; + raw[..32].copy_from_slice(&rfc9162::from_hex(&text[..64])?); + raw[32..].copy_from_slice(&rfc9162::from_hex(&text[64..])?); + Some(raw) +} + +/// Read an Ed25519 private key: PKCS#8 PEM, as `openssl genpkey -algorithm +/// ed25519` writes it. +pub fn read_signing_key(path: &Path) -> Result { + let text = std::fs::read_to_string(path) + .with_context(|| format!("read signing key {}", path.display()))?; + SigningKey::from_pkcs8_pem(&text).map_err(|error| { + anyhow!( + "signing key {} is not Ed25519 PKCS#8 PEM: {error}", + path.display() + ) + }) +} + +/// Read an Ed25519 public key: SubjectPublicKeyInfo PEM, as `openssl pkey +/// -pubout` writes it, or the 64 hex characters a head file carries. +pub fn read_public_key(path: &Path) -> Result { + let text = std::fs::read_to_string(path) + .with_context(|| format!("read public key {}", path.display()))?; + let trimmed = text.trim(); + if let Some(raw) = rfc9162::from_hex(trimmed) { + return VerifyingKey::from_bytes(&raw) + .map_err(|error| anyhow!("public key {} is not Ed25519: {error}", path.display())); + } + VerifyingKey::from_public_key_pem(trimmed).map_err(|error| { + anyhow!( + "public key {} is neither Ed25519 PEM nor 64 hex characters: {error}", + path.display() + ) + }) +} + +/// Refuse a signing key that sits inside the log's directory. +fn refuse_key_beside_log(key: &Path, log: &Path) -> Result<()> { + let key = key + .canonicalize() + .with_context(|| format!("signing key {}", key.display()))?; + let log = std::path::absolute(log).with_context(|| format!("ledger {}", log.display()))?; + let directory = log.parent().unwrap_or(Path::new("/")); + let directory = directory + .canonicalize() + .with_context(|| format!("ledger directory {}", directory.display()))?; + if key.starts_with(&directory) { + bail!( + "signing key {} is inside the ledger directory {}; keep the key off the log's tree", + key.display(), + directory.display() + ); + } + Ok(()) +} + +/// A signing key read for one log, with the file it was read from. +struct KeyFile { + key: SigningKey, + /// The resolved path of the key file. + path: PathBuf, + /// Device and inode of the key file. + inode: Option<(u64, u64)>, +} + +/// Read the signing key for `log` from the file whose resolved path was +/// checked to lie outside the log's directory. A key with a second hard +/// link is refused: that name could sit in the log's tree, where the +/// directory check cannot see it. +fn open_signing_key(path: &Path, log: &Path) -> Result { + let mut file = + File::open(path).with_context(|| format!("read signing key {}", path.display()))?; + let opened = file + .metadata() + .with_context(|| format!("signing key {}", path.display()))?; + if !opened.is_file() { + bail!("signing key {} is not a regular file", path.display()); + } + refuse_key_beside_log(path, log)?; + let resolved = path + .canonicalize() + .with_context(|| format!("signing key {}", path.display()))?; + let inode = file_identity(&opened); + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + if opened.nlink() != 1 { + bail!( + "signing key {} has {} hard links; a key with another name could travel with the log", + path.display(), + opened.nlink() + ); + } + let current = std::fs::metadata(&resolved) + .with_context(|| format!("signing key {}", path.display()))?; + if file_identity(¤t) != inode { + bail!("signing key {} changed while it was read", path.display()); + } + } + let mut text = String::new(); + file.read_to_string(&mut text) + .with_context(|| format!("read signing key {}", path.display()))?; + let key = SigningKey::from_pkcs8_pem(&text).map_err(|error| { + anyhow!( + "signing key {} is not Ed25519 PKCS#8 PEM: {error}", + path.display() + ) + })?; + Ok(KeyFile { + key, + path: resolved, + inode, + }) +} + +/// Device and inode, where the platform has them. +fn file_identity(metadata: &std::fs::Metadata) -> Option<(u64, u64)> { + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + Some((metadata.dev(), metadata.ino())) + } + #[cfg(not(unix))] + { + let _ = metadata; + None + } +} + +/// The files that belong to one log, by resolved path. +#[derive(Debug, Clone)] +struct LedgerPaths { + log: PathBuf, + directory: PathBuf, + record_lock: PathBuf, + /// File name prefix of tails moved out of the log. + quarantine: OsString, +} + +impl LedgerPaths { + fn of(log: &Path) -> Result { + let log = log + .canonicalize() + .with_context(|| format!("resolve ledger {}", log.display()))?; + let directory = log.parent().unwrap_or(Path::new("/")).to_path_buf(); + let name = log.file_name().unwrap_or_default(); + let mut record_lock = name.to_os_string(); + record_lock.push(".posix-lock"); + let mut quarantine = name.to_os_string(); + quarantine.push(".quarantine."); + Ok(Self { + record_lock: directory.join(record_lock), + log, + directory, + quarantine, + }) + } +} + +/// Refuse a head path that would replace or write through the log, its +/// record lock, a quarantined tail, or the signing key, by name, symbolic +/// link or hard link. A path whose directory does not exist names none of +/// them, and writing the head reports it. +fn check_head_target(out: &Path, paths: &LedgerPaths, key: Option<&KeyFile>) -> Result<()> { + let absolute = std::path::absolute(out).with_context(|| format!("head {}", out.display()))?; + let (Some(parent), Some(name)) = (absolute.parent(), absolute.file_name()) else { + bail!("head path {} names no file", out.display()); + }; + let Ok(parent) = parent.canonicalize() else { + return Ok(()); + }; + let resolved = parent.join(name); + let refuse = |what: &str| -> Result<()> { + bail!( + "head path {} is {what}; a head never replaces it", + out.display() + ) + }; + if resolved == paths.log { + return refuse("the ledger"); + } + if resolved == paths.record_lock { + return refuse("the ledger's record lock"); + } + if key.is_some_and(|key| key.path == resolved) { + return refuse("the signing key"); + } + if parent == paths.directory + && name + .as_encoded_bytes() + .starts_with(paths.quarantine.as_encoded_bytes()) + { + return refuse("a quarantined ledger tail"); + } + let Ok(existing) = std::fs::symlink_metadata(&resolved) else { + return Ok(()); + }; + if existing.file_type().is_symlink() { + bail!( + "head path {} is a symbolic link; refusing to replace it", + out.display() + ); + } + let Some(inode) = file_identity(&existing) else { + return Ok(()); + }; + let identity = |path: &Path| { + std::fs::symlink_metadata(path) + .ok() + .and_then(|metadata| file_identity(&metadata)) + }; + if identity(&paths.log) == Some(inode) { + return refuse("a link to the ledger"); + } + if identity(&paths.record_lock) == Some(inode) { + return refuse("a link to the ledger's record lock"); + } + if key.is_some_and(|key| key.inode == Some(inode)) { + return refuse("a link to the signing key"); + } + Ok(()) +} + +fn now_utc() -> String { + Utc::now().to_rfc3339_opts(SecondsFormat::Secs, true) +} + +/// What `ledger append` needs. +#[derive(Debug, Clone)] +pub struct AppendRequest { + /// The JSON Lines log. Created when absent. + pub log: PathBuf, + /// The record, a JSON object file. + pub record: PathBuf, + /// A head the log must still be at: a writer passes the head it last + /// saw, so two writers cannot both append on the same head unnoticed. + pub expect_head: Option, + /// Ed25519 PKCS#8 PEM key to sign the new head with. + pub key: Option, + /// Where to write the new signed head. Needs `key`. + pub head_out: Option, +} + +/// What `ledger append` wrote. +#[derive(Debug, Clone)] +pub struct AppendReport { + /// The appended record's leaf number. + pub index: usize, + /// The appended record's id. + pub id: String, + /// The canonical line written. + pub line: String, + /// The head after the append. + pub head: rfc9162::Head, + /// The signed head, when a key was given. + pub signed: Option, + /// Whether this acknowledges an identical event that is in the log. + pub replayed: bool, + /// The byte count and quarantine file of an interrupted tail that was + /// cut from the log before this append. + pub quarantined: Option<(usize, PathBuf)>, +} + +impl AppendReport { + /// One line for stdout. + #[must_use] + pub fn summary(&self) -> String { + let status = if self.replayed { + "already recorded" + } else { + "appended" + }; + let mut text = format!( + "{status} record {} id {}; head size {} root {}", + self.index, self.id, self.head.size, self.head.root + ); + if let Some((bytes, path)) = &self.quarantined { + let _ = write!( + text, + "; moved {bytes} bytes of an interrupted append to {}", + path.display() + ); + } + text + } +} + +/// POSIX record locks belong to a process, not to a descriptor: a second +/// lock request from the same process converts the first instead of +/// waiting, and closing any descriptor for the inode drops every lock the +/// process holds on it. One slot per ledger inode per process keeps a second +/// writer or signer in this process from reaching the record-lock inode +/// while another holds it. +#[cfg(unix)] +mod process_slot { + use std::sync::{Condvar, Mutex, PoisonError}; + + static HELD: Mutex> = Mutex::new(Vec::new()); + static RELEASED: Condvar = Condvar::new(); + + pub(super) struct Slot((u64, u64)); + + impl Slot { + pub(super) fn claim(inode: (u64, u64)) -> Self { + let mut held = HELD.lock().unwrap_or_else(PoisonError::into_inner); + while held.contains(&inode) { + held = RELEASED.wait(held).unwrap_or_else(PoisonError::into_inner); + } + held.push(inode); + Self(inode) + } + } + + impl Drop for Slot { + fn drop(&mut self) { + let mut held = HELD.lock().unwrap_or_else(PoisonError::into_inner); + held.retain(|inode| *inode != self.0); + RELEASED.notify_all(); + } + } +} + +/// Whether a guard excludes everyone or only writers. +#[derive(Clone, Copy, PartialEq, Eq)] +enum LockMode { + /// A writer: excludes writers and signers. + Write, + /// A signer: excludes writers. Needs read access only. + Read, +} + +/// A writer's local gate keeps contenders from opening the process-scoped +/// record-lock inode. Closing any descriptor for that inode releases the +/// process's lock. +struct LedgerLock { + gate: Option, + record: Option, + #[cfg(unix)] + slot: Option, + owner: u32, + paths: LedgerPaths, +} + +impl LedgerLock { + fn acquire(gate: &File, log: &Path) -> Result { + Self::acquire_as(gate, log, LockMode::Write) + } + + /// Take the signer's lock through a descriptor opened for reading. + fn acquire_shared(log_file: &File, log: &Path) -> Result { + Self::acquire_as(log_file, log, LockMode::Read) + } + + fn acquire_as(opened_log: &File, log: &Path, mode: LockMode) -> Result { + if !opened_log + .metadata() + .context("inspect ledger file")? + .is_file() + { + bail!("ledger must be a regular file: {}", log.display()); + } + let gate = opened_log + .try_clone() + .context("duplicate ledger lock descriptor")?; + match mode { + LockMode::Write => FileExt::lock_exclusive(&gate), + #[cfg(unix)] + LockMode::Read => Ok(()), + #[cfg(not(unix))] + LockMode::Read => FileExt::lock_shared(&gate), + } + .with_context(|| format!("lock ledger {}", log.display()))?; + let mut guard = Self { + gate: Some(gate), + record: None, + #[cfg(unix)] + slot: None, + owner: std::process::id(), + paths: LedgerPaths::of(log)?, + }; + #[cfg(unix)] + { + use std::os::fd::AsRawFd; + use std::os::unix::fs::{MetadataExt, OpenOptionsExt}; + + let opened = opened_log.metadata().context("inspect ledger lock inode")?; + if opened.nlink() != 1 { + bail!( + "ledger requires one link for writer exclusion: {}", + log.display() + ); + } + let resolved = guard + .paths + .log + .metadata() + .context("inspect resolved ledger")?; + if (opened.dev(), opened.ino()) != (resolved.dev(), resolved.ino()) { + bail!( + "ledger path changed while acquiring its lock: {}", + log.display() + ); + } + guard.slot = Some(process_slot::Slot::claim((opened.dev(), opened.ino()))); + let record = open_record_lock(&guard.paths.record_lock, mode) + .with_context(|| format!("open record lock for ledger {}", log.display()))?; + let info = record.metadata().context("inspect ledger record lock")?; + if !info.is_file() || info.nlink() != 1 { + bail!( + "ledger record lock requires a regular file with one link: {}", + log.display() + ); + } + // All-zero offsets cover the whole file; flock's scalar fields + // have valid zero representations on the supported Unix targets. + let mut lock: libc::flock = unsafe { std::mem::zeroed() }; + lock.l_type = match mode { + LockMode::Write => libc::F_WRLCK, + LockMode::Read => libc::F_RDLCK, + } as _; + lock.l_whence = libc::SEEK_SET as _; + loop { + // The descriptor stays open in this guard and lock points to + // a fully initialized flock for the duration of fcntl. + let result = unsafe { libc::fcntl(record.as_raw_fd(), libc::F_SETLKW, &lock) }; + if result != -1 { + break; + } + let error = std::io::Error::last_os_error(); + if error.kind() != std::io::ErrorKind::Interrupted { + return Err(error) + .with_context(|| format!("record-lock ledger {}", log.display())); + } + } + guard.record = Some(record); + + fn open_record_lock(path: &Path, mode: LockMode) -> std::io::Result { + let mut options = OpenOptions::new(); + options + .read(true) + .custom_flags(libc::O_NOFOLLOW | libc::O_NONBLOCK); + if mode == LockMode::Write { + return options.write(true).create(true).truncate(false).open(path); + } + match options.open(path) { + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + opened => return opened, + } + // No writer has created the lock yet. A signer that can create + // it does; one that cannot must not sign unlocked, because a + // writer may create it and append at any moment. + match options.clone().write(true).create_new(true).open(path) { + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => { + options.open(path) + } + Err(error) => Err(std::io::Error::new( + error.kind(), + format!( + "{} does not exist and cannot be created ({error}); \ + a writer creates it on its first append", + path.display() + ), + )), + created => created, + } + } + } + Ok(guard) + } +} + +impl Drop for LedgerLock { + fn drop(&mut self) { + // Closing the sidecar releases the record lock before another local + // contender can open it. A forked child must not unlock its parent. + drop(self.record.take()); + #[cfg(unix)] + drop(self.slot.take()); + if let Some(gate) = self.gate.take() { + if self.owner == std::process::id() { + let _ = FileExt::unlock(&gate); + } + } + } +} + +/// Move the bytes after the last newline, an append that never finished, +/// into a new quarantine file beside the log, flush it and the directory, +/// and only then cut them from the log. Runs under the writer lock. +fn quarantine_tail( + file: &File, + paths: &LedgerPaths, + committed: u64, + tail: &[u8], +) -> Result { + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::{OpenOptionsExt, PermissionsExt}; + let mode = file + .metadata() + .context("inspect ledger")? + .permissions() + .mode() + & 0o666; + options.mode(mode).custom_flags(libc::O_NOFOLLOW); + } + let stamp = Utc::now().format("%Y%m%dT%H%M%S%9fZ"); + let mut attempt = 0u32; + let (path, mut quarantine) = loop { + let mut name = paths.quarantine.clone(); + name.push(format!("{stamp}.{}.{attempt}", std::process::id())); + let path = paths.directory.join(name); + match options.open(&path) { + Ok(opened) => break (path, opened), + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists && attempt < 1000 => { + attempt += 1; + } + Err(error) => { + return Err(error) + .with_context(|| format!("create ledger quarantine {}", path.display())) + } + } + }; + let kept = quarantine + .write_all(tail) + .and_then(|()| quarantine.sync_all()) + .with_context(|| format!("write ledger quarantine {}", path.display())) + .and_then(|()| sync_directory(&paths.directory)); + if let Err(error) = kept { + drop(quarantine); + let _ = std::fs::remove_file(&path); + return Err(error); + } + file.set_len(committed) + .and_then(|()| file.sync_all()) + .with_context(|| { + format!( + "cut the interrupted tail from ledger {}; it is kept in {}", + paths.log.display(), + path.display() + ) + })?; + Ok(path) +} + +/// Write one line and flush it. A failed or short write is cut back to +/// `start`, so the log never keeps a partial line this writer left. +fn write_line(file: &mut File, start: u64, line: &str, log: &Path) -> Result<()> { + let written = file + .write_all(format!("{line}\n").as_bytes()) + .and_then(|()| file.sync_all()); + let Err(error) = written else { + return Ok(()); + }; + let error = anyhow!(error).context(format!("append to ledger {}", log.display())); + match file.set_len(start).and_then(|()| file.sync_all()) { + Ok(()) => Err(error), + Err(rollback) => Err(error.context(format!( + "the partial line could not be cut back ({rollback}); the next append quarantines it" + ))), + } +} + +/// Append one record to the log under an exclusive lock on the log file. +/// +/// A producer can persist a unique `build.event_id` for each event packet. +/// An identical retry acknowledges the existing entry and writes a current +/// signed head without appending another record. A reused identifier with +/// different content is refused. An expected head must match the history; +/// only an acknowledged retry can accept a matching prefix instead of the tip. +pub fn append(request: &AppendRequest) -> Result { + if request.head_out.is_some() != request.key.is_some() { + return Err(crate::cli_parse::usage( + "--key and --head-out go together: the key signs the head that --head-out names", + )); + } + let key = match &request.key { + Some(path) => Some(open_signing_key(path, &request.log)?), + None => None, + }; + let text = std::fs::read_to_string(&request.record) + .with_context(|| format!("read record {}", request.record.display()))?; + let record: Value = serde_json::from_str(&text) + .with_context(|| format!("parse record {}", request.record.display()))?; + let expected = match &request.expect_head { + Some(path) => Some(SignedHead::read(path)?), + None => None, + }; + + let mut options = std::fs::OpenOptions::new(); + options.read(true).append(true).create(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.custom_flags(libc::O_NOFOLLOW | libc::O_NONBLOCK); + } + let mut file = options + .open(&request.log) + .with_context(|| format!("open ledger {}", request.log.display()))?; + let lock = LedgerLock::acquire(&file, &request.log)?; + if let Some(out) = &request.head_out { + check_head_target(out, &lock.paths, key.as_ref())?; + } + let mut current = Vec::new(); + file.seek(SeekFrom::Start(0))?; + file.read_to_end(&mut current)?; + let committed = current + .iter() + .rposition(|byte| *byte == b'\n') + .map_or(0, |at| at + 1); + if committed < current.len() && serde_json::from_slice::(¤t[committed..]).is_ok() { + bail!( + "the last line of ledger {} is complete JSON without a newline; it may be a \ + committed record, so it is neither repaired nor appended after", + request.log.display() + ); + } + let mut entries = std::str::from_utf8(¤t[..committed]) + .context("the log is not UTF-8") + .and_then(parse_log) + .with_context(|| format!("ledger {}", request.log.display()))?; + let id = match record.as_object() { + Some(record) => event_id(record)?, + None => None, + }; + let previous = if let Some(id) = id { + let mut matching = entries.iter().filter(|entry| { + entry + .record + .get("build") + .and_then(|build| build.get("event_id")) + .and_then(Value::as_str) + == Some(id) + }); + let previous = matching.next().cloned(); + if matching.next().is_some() { + bail!("ledger contains multiple records for build.event_id {id:?}"); + } + previous + } else { + None + }; + let replayed = previous.is_some(); + let now = now_utc(); + let entry = if let Some(previous) = previous { + let original_ts = previous.record["ts"].as_str().unwrap_or(&now); + let retry = seal_record(&record, previous.index, original_ts) + .with_context(|| format!("record {}", request.record.display()))?; + if retry.line != previous.line { + bail!( + "build.event_id {:?} is already recorded with different content", + id + ); + } + previous + } else { + seal_record(&record, entries.len(), &now) + .with_context(|| format!("record {}", request.record.display()))? + }; + if let Some(expected) = &expected { + let current_leaves = leaves(&entries); + let prefix_matches = expected.size <= current_leaves.len() + && expected.root == rfc9162::Head::over(¤t_leaves[..expected.size]).root; + if expected.domain != DOMAIN + || !prefix_matches + || (!replayed && expected.size != entries.len()) + { + let head = rfc9162::Head::over(¤t_leaves); + bail!( + "ledger is at size {} root {}, not the expected head size {} root {}", + head.size, + head.root, + expected.size, + expected.root + ); + } + } + if let Some(first) = entries.first() { + if first.record["stack"] != entry.record["stack"] { + bail!( + "record is stack {} but ledger {} is stack {}", + entry.record["stack"], + request.log.display(), + first.record["stack"] + ); + } + } + let quarantined = if committed < current.len() { + Some(quarantine_tail( + &file, + &lock.paths, + committed as u64, + ¤t[committed..], + )?) + } else { + None + }; + if replayed { + // The earlier attempt may have stopped before its flush; acknowledge + // only what is on stable storage. + file.sync_all() + .with_context(|| format!("flush ledger {}", request.log.display()))?; + } else { + write_line(&mut file, committed as u64, &entry.line, &request.log)?; + entries.push(entry.clone()); + } + sync_directory(&lock.paths.directory)?; + let signed = key.map(|key| SignedHead::sign(&entries, &key.key, &now)); + if let (Some(signed), Some(out)) = (&signed, &request.head_out) { + signed.write(out)?; + } + drop(lock); + Ok(AppendReport { + index: entry.index, + id: entry.id, + line: entry.line, + head: rfc9162::Head::over(&leaves(&entries)), + signed, + replayed, + quarantined: quarantined.map(|path| (current.len() - committed, path)), + }) +} + +/// What `ledger head` needs. +#[derive(Debug, Clone)] +pub struct HeadRequest { + /// The JSON Lines log. + pub log: PathBuf, + /// Ed25519 PKCS#8 PEM key. + pub key: PathBuf, + /// The head file to write. + pub out: PathBuf, + /// Signing time, RFC 3339 UTC. Defaults to now. + pub ts: Option, +} + +/// Sign the log's current head. The log is read and the head written under +/// the signer's side of the writer lock, so the head is the log between two +/// appends and a head an append wrote is never replaced by an older one. +/// The signer opens the log and its record lock for reading only. +pub fn sign_head(request: &HeadRequest) -> Result { + let key = open_signing_key(&request.key, &request.log)?; + let ts = match &request.ts { + Some(ts) => { + let parsed = DateTime::parse_from_rfc3339(ts) + .map_err(|_| crate::cli_parse::usage(format!("--ts {ts:?} is not RFC 3339")))?; + if parsed.offset().local_minus_utc() != 0 { + return Err(crate::cli_parse::usage(format!("--ts {ts:?} is not UTC"))); + } + ts.clone() + } + None => now_utc(), + }; + let mut options = OpenOptions::new(); + options.read(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.custom_flags(libc::O_NOFOLLOW | libc::O_NONBLOCK); + } + let mut file = options + .open(&request.log) + .with_context(|| format!("read ledger {}", request.log.display()))?; + let lock = LedgerLock::acquire_shared(&file, &request.log)?; + check_head_target(&request.out, &lock.paths, Some(&key))?; + let mut text = String::new(); + file.read_to_string(&mut text) + .with_context(|| format!("read ledger {}", request.log.display()))?; + let entries = parse_log(&text).with_context(|| format!("ledger {}", request.log.display()))?; + let head = SignedHead::sign(&entries, &key.key, &ts); + head.write(&request.out)?; + drop(lock); + Ok(head) +} + +/// What `ledger show` needs. +#[derive(Debug, Clone)] +pub struct ShowRequest { + /// The JSON Lines log. + pub log: PathBuf, + /// A module (`PETSc/3.25.0-foss-2026.1`) or a software name (`PETSc`). + pub package: Option, + /// Print the records as JSON instead of a table. + pub json: bool, +} + +/// What `ledger show` found. +#[derive(Debug, Clone)] +pub struct ShowReport { + /// Matching records, newest first. + pub records: Vec, + /// What the command prints. + pub stdout: String, +} + +/// List records, newest first, optionally for one package. +pub fn show(request: &ShowRequest) -> Result { + let entries = read_log(&request.log)?; + let matches = |entry: &Entry| match &request.package { + None => true, + Some(name) => { + let module = entry.module(); + module == name || module.split('/').next() == Some(name.as_str()) + } + }; + let selected = entries + .iter() + .rev() + .filter(|entry| matches(entry)) + .collect::>(); + let records = selected + .iter() + .map(|entry| Value::Object(entry.record.clone())) + .collect::>(); + let stdout = if request.json { + let mut text = serde_json::to_string_pretty(&records)?; + text.push('\n'); + text + } else { + let mut text = String::from("index\tts\taction\ttarget\tmodule\tactor\n"); + for entry in &selected { + let field = |name: &str| entry.record[name].as_str().unwrap_or_default().to_string(); + let _ = writeln!( + text, + "{}\t{}\t{}\t{}\t{}\t{}", + entry.index, + field("ts"), + field("action"), + field("target"), + field("module"), + field("actor") + ); + } + let _ = writeln!(text, "{} of {} records", selected.len(), entries.len()); + text + }; + Ok(ShowReport { records, stdout }) +} + +/// Which proof `ledger verify` checks. +#[derive(Debug, Clone)] +pub enum VerifyMode { + /// Record `record` is in the signed head `head`. + Inclusion { + /// Leaf number. + record: usize, + /// Head file. + head: PathBuf, + }, + /// The signed head `to` extends the signed head `from`. + Consistency { + /// The earlier head file. + from: PathBuf, + /// The later head file. + to: PathBuf, + }, +} + +/// What `ledger verify` needs. +#[derive(Debug, Clone)] +pub struct VerifyRequest { + /// The JSON Lines log. + pub log: PathBuf, + /// The public key heads must be signed by. + pub public_key: PathBuf, + /// The proof to check. + pub mode: VerifyMode, +} + +/// A proof that checked. +#[derive(Debug, Clone, Serialize)] +pub struct VerifyReport { + /// `inclusion` or `consistency`. + pub kind: String, + /// The proof nodes, lower-case hex, innermost first. + pub proof: Vec, + /// What the command prints. + pub stdout: String, +} + +/// Check an inclusion or consistency proof computed from the log against +/// signed heads. Any failure is an error. +pub fn verify(request: &VerifyRequest) -> Result { + let trusted = read_public_key(&request.public_key)?; + let entries = read_log(&request.log)?; + let all = leaves(&entries); + match &request.mode { + VerifyMode::Inclusion { record, head } => { + let head = SignedHead::read(head)?; + head.check(&trusted)?; + if head.size > all.len() { + bail!( + "head covers {} records but the ledger has {}", + head.size, + all.len() + ); + } + if *record >= head.size { + bail!( + "record {record} is past the head, which covers {} records", + head.size + ); + } + let covered = &all[..head.size]; + let proof = rfc9162::inclusion_proof(covered, *record) + .ok_or_else(|| anyhow!("no inclusion proof for record {record}"))?; + if !rfc9162::verify_inclusion(&covered[*record], *record, head.size, &proof, &head.root) + { + bail!( + "record {record} is not in the head of size {} root {}", + head.size, + head.root + ); + } + Ok(VerifyReport { + kind: "inclusion".to_string(), + proof: proof.iter().map(|node| rfc9162::hex(node)).collect(), + stdout: format!( + "record {record} id {} is in the head of size {} root {}\n", + entries[*record].id, head.size, head.root + ), + }) + } + VerifyMode::Consistency { from, to } => { + let from = SignedHead::read(from)?; + let to = SignedHead::read(to)?; + from.check(&trusted).context("--from head")?; + to.check(&trusted).context("--to head")?; + if from.size > to.size { + bail!( + "the --from head covers {} records and the --to head {}; a log only grows", + from.size, + to.size + ); + } + if to.size > all.len() { + bail!( + "the --to head covers {} records but the ledger has {}", + to.size, + all.len() + ); + } + let covered = &all[..to.size]; + let proof = if from.size == 0 { + // The empty tree is a prefix of every tree; the only thing to + // check is that the earlier head is the empty head. + if from.root != rfc9162::hex(&rfc9162::root(&[])) { + bail!("the --from head has size 0 but not the empty root"); + } + if rfc9162::hex(&rfc9162::root(covered)) != to.root { + bail!("the ledger does not hash to the --to head"); + } + Vec::new() + } else { + let proof = rfc9162::consistency_proof(covered, from.size) + .ok_or_else(|| anyhow!("no consistency proof from size {}", from.size))?; + if !rfc9162::verify_consistency(from.size, &from.root, to.size, &to.root, &proof) { + bail!( + "the head of size {} root {} does not extend the head of size {} root {}", + to.size, + to.root, + from.size, + from.root + ); + } + proof + }; + Ok(VerifyReport { + kind: "consistency".to_string(), + proof: proof.iter().map(|node| rfc9162::hex(node)).collect(), + stdout: format!( + "the head of size {} root {} extends the head of size {} root {}\n", + to.size, to.root, from.size, from.root + ), + }) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + fn record(module: &str, ts: &str) -> Value { + json!({ + "ts": ts, + "actor": "jenkins/run/1", + "stack": "2026.06", + "target": "x86_64/amd/zen4", + "module": module, + "location": format!("/cvmfs/software.eessi.io/{module}"), + "action": "install", + }) + } + + #[test] + fn canonical_json_sorts_keys_and_drops_whitespace() { + let value: Value = + serde_json::from_str(r#"{ "b" : [1, {"y": 2, "x": "é\n"}], "a": null }"#).unwrap(); + assert_eq!( + canonical(&value).unwrap(), + r#"{"a":null,"b":[1,{"x":"é\n","y":2}]}"# + ); + } + + #[test] + fn canonical_json_refuses_floats() { + assert!(canonical(&json!({"a": 1.5})).is_err()); + } + + #[test] + fn a_sealed_record_parses_back() { + let entry = + seal_record(&record("zlib/1.3.1", "2026-11-03T10:00:00Z"), 0, "unused").unwrap(); + let parsed = parse_log(&format!("{}\n", entry.line)).unwrap(); + assert_eq!(parsed[0].id, entry.id); + assert_eq!(parsed[0].line, entry.line); + } + + #[test] + fn a_missing_ts_is_filled() { + let mut bare = record("zlib/1.3.1", "x"); + bare.as_object_mut().unwrap().remove("ts"); + let entry = seal_record(&bare, 0, "2026-11-03T10:00:00Z").unwrap(); + assert_eq!(entry.record["ts"], "2026-11-03T10:00:00Z"); + } + + #[test] + fn records_outside_the_format_are_refused() { + let mut unknown = record("zlib/1.3.1", "2026-11-03T10:00:00Z"); + unknown["licence_key"] = json!("secret"); + assert!(seal_record(&unknown, 0, "").is_err()); + let mut action = record("zlib/1.3.1", "2026-11-03T10:00:00Z"); + action["action"] = json!("reinstall"); + assert!(seal_record(&action, 0, "").is_err()); + let local = record("zlib/1.3.1", "2026-11-03T10:00:00+01:00"); + assert!(seal_record(&local, 0, "").is_err()); + let mut correction = record("zlib/1.3.1", "2026-11-03T10:00:00Z"); + correction["action"] = json!("correction"); + assert!(seal_record(&correction, 0, "").is_err()); + let wrong_index = { + let mut value = record("zlib/1.3.1", "2026-11-03T10:00:00Z"); + value["index"] = json!(4); + value + }; + assert!(seal_record(&wrong_index, 0, "").is_err()); + } + + #[test] + fn a_line_out_of_canonical_form_is_refused() { + let entry = seal_record(&record("zlib/1.3.1", "2026-11-03T10:00:00Z"), 0, "").unwrap(); + let spaced = entry.line.replacen(':', ": ", 1); + assert!(parse_log(&format!("{spaced}\n")).is_err()); + assert!(parse_log(&entry.line).is_err(), "no trailing newline"); + } + + #[test] + fn a_head_signature_binds_domain_size_root_and_time() { + let key = SigningKey::from_bytes(&[7u8; 32]); + let entry = seal_record(&record("zlib/1.3.1", "2026-11-03T10:00:00Z"), 0, "").unwrap(); + let head = SignedHead::sign(&[entry], &key, "2026-11-03T10:00:00Z"); + head.check(&key.verifying_key()).unwrap(); + for tamper in [ + SignedHead { + size: 2, + ..head.clone() + }, + SignedHead { + ts: "2026-11-04T10:00:00Z".to_string(), + ..head.clone() + }, + SignedHead { + domain: "deedar-log-v1".to_string(), + ..head.clone() + }, + ] { + assert!(tamper.check(&key.verifying_key()).is_err()); + } + let other = SigningKey::from_bytes(&[8u8; 32]); + assert!(head.check(&other.verifying_key()).is_err()); + assert!( + SignedHead::material(1, &head.root, &head.ts).starts_with(b"eb-stack-ledger-v1\t1\t") + ); + } + + /// Whether a second guard taken in this process waits for `first`. + #[cfg(unix)] + fn second_guard_waits(first: LockMode, second: LockMode) { + use std::sync::atomic::AtomicBool; + use std::sync::Arc; + use std::time::Duration; + + let root = tempfile::tempdir().unwrap(); + let log = root.path().join("stack.jsonl"); + std::fs::write(&log, "").unwrap(); + let open = || OpenOptions::new().read(true).open(&log).unwrap(); + let held = LedgerLock::acquire_as(&open(), &log, first).unwrap(); + let taken = Arc::new(AtomicBool::new(false)); + let waiter = { + let (log, taken, file) = (log.clone(), Arc::clone(&taken), open()); + std::thread::spawn(move || { + let guard = LedgerLock::acquire_as(&file, &log, second).unwrap(); + taken.store(true, Ordering::SeqCst); + drop(guard); + }) + }; + std::thread::sleep(Duration::from_millis(300)); + assert!( + !taken.load(Ordering::SeqCst), + "a second guard in the same process shared the record lock" + ); + drop(held); + waiter.join().unwrap(); + assert!(taken.load(Ordering::SeqCst)); + } + + #[cfg(unix)] + #[test] + fn a_signer_in_the_writer_process_waits_for_the_writer() { + second_guard_waits(LockMode::Write, LockMode::Read); + } + + #[cfg(unix)] + #[test] + fn signers_in_one_process_do_not_share_the_record_lock() { + second_guard_waits(LockMode::Read, LockMode::Read); + } + + #[cfg(unix)] + #[test] + fn a_writer_in_the_signer_process_waits_for_the_signer() { + second_guard_waits(LockMode::Read, LockMode::Write); + } +} diff --git a/src/lib.rs b/src/lib.rs index 76b2e237..60d4bf2e 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -47,6 +47,7 @@ mod index_bump; pub mod input_hash; mod install_prefix; pub mod jenkins; +pub mod ledger; pub mod luarocks; mod manifest; pub mod mcp; @@ -81,6 +82,7 @@ pub mod sbom; pub mod select; pub mod shadow; pub mod shard; +pub mod shard_trees; pub mod site_commands; pub mod source_fetch; pub mod source_probe; diff --git a/src/main.rs b/src/main.rs index 6e0b1d52..954447de 100644 --- a/src/main.rs +++ b/src/main.rs @@ -91,6 +91,14 @@ enum Command { #[command(subcommand)] command: ReproCommand, }, + /// Append to, list, and verify a per-stack software ledger. + #[command( + after_long_help = "Examples:\n eb-stack ledger append record.json --log ledger/2026.06.jsonl --key ~/keys/ledger.pem --head-out head.json\n eb-stack ledger show --log ledger/2026.06.jsonl --package PETSc\n eb-stack ledger verify --log ledger/2026.06.jsonl --public-key ledger.pub --record 3 --head head.json" + )] + Ledger { + #[command(subcommand)] + command: LedgerCommand, + }, /// Serve each CLI workflow over stdio, including pr-overlap, rebuild-set, /// component-reach, and easyblock-reach. #[command( @@ -452,6 +460,99 @@ enum ReproCommand { }, } +#[derive(Subcommand, Debug)] +enum LedgerCommand { + /// Append one record to the log and sign the new head. + /// + /// The record is a JSON object with the ledger fields. It is written as + /// one canonical line: keys sorted, no whitespace, integers only, with + /// `index` (its leaf number) and `id` (SHA-256 of the record without + /// `id` and `sig`) filled in. `ts` defaults to now, in UTC. The log file + /// is locked for the append. + #[command( + after_long_help = "Examples:\n eb-stack ledger append record.json --log ledger/2026.06.jsonl\n eb-stack ledger append record.json --log ledger/2026.06.jsonl --expect-head head.json --key ~/keys/ledger.pem --head-out head.json" + )] + Append { + /// The record, a JSON object file. + record: PathBuf, + /// The JSON Lines log. Created when absent. + #[arg(long)] + log: PathBuf, + /// Refuse the append unless the log is still at this head's size and root. + #[arg(long)] + expect_head: Option, + /// Ed25519 PKCS#8 PEM key that signs the new head. Refused inside the log's directory. + #[arg(long, requires = "head_out")] + key: Option, + /// File for the signed head after the append. + #[arg(long, requires = "key")] + head_out: Option, + }, + /// Sign the log's current head. + #[command( + after_long_help = "Examples:\n eb-stack ledger head --log ledger/2026.06.jsonl --key ~/keys/ledger.pem --out anchor.json" + )] + Head { + /// The JSON Lines log. + #[arg(long)] + log: PathBuf, + /// Ed25519 PKCS#8 PEM key. Refused inside the log's directory. + #[arg(long)] + key: PathBuf, + /// File for the signed head. + #[arg(long)] + out: PathBuf, + /// Signing time, RFC 3339 in UTC. Defaults to now. + #[arg(long)] + ts: Option, + }, + /// List records, newest first. + #[command( + after_long_help = "Examples:\n eb-stack ledger show --log ledger/2026.06.jsonl\n eb-stack ledger show --log ledger/2026.06.jsonl --package PETSc/3.25.0-foss-2026.1 --json" + )] + Show { + /// The JSON Lines log. + #[arg(long)] + log: PathBuf, + /// A module name (`PETSc/3.25.0-foss-2026.1`) or a software name (`PETSc`). + #[arg(long)] + package: Option, + /// Print the records as a JSON array instead of a table. + #[arg(long)] + json: bool, + }, + /// Check an inclusion or a consistency proof against signed heads. + /// + /// `--record N --head HEAD` checks that record N is in HEAD. + /// `--from HEAD1 --to HEAD2` checks that HEAD2 extends HEAD1. Every head + /// must be signed by `--public-key`. Exit status 1 when a proof or a + /// signature does not check. + #[command( + after_long_help = "Examples:\n eb-stack ledger verify --log ledger/2026.06.jsonl --public-key ledger.pub --record 3 --head head.json\n eb-stack ledger verify --log ledger/2026.06.jsonl --public-key ledger.pub --from anchor-1102.json --to anchor-1103.json", + group(clap::ArgGroup::new("proof").required(true).args(["record", "from"])) + )] + Verify { + /// The JSON Lines log. + #[arg(long)] + log: PathBuf, + /// Ed25519 public key heads are signed by: PEM, or 64 hex characters. + #[arg(long)] + public_key: PathBuf, + /// Leaf number of the record to prove. + #[arg(long, requires = "head", conflicts_with_all = ["from", "to"])] + record: Option, + /// Signed head the record must be in. + #[arg(long, requires = "record")] + head: Option, + /// The earlier signed head. + #[arg(long, requires = "to")] + from: Option, + /// The later signed head, which must extend --from. + #[arg(long, requires = "from")] + to: Option, + }, +} + #[derive(Subcommand, Debug)] enum PackageCommand { /// Parse a foreign recipe into a canonical build manifest and planned SBOM. @@ -1017,6 +1118,23 @@ struct ShardArgs { /// Robot-path easyconfig tree. Repeatable; later trees take precedence. #[arg(long, required = true)] easyconfigs: Vec, + /// Easyconfigs of one `--from-commit` pin, as `COMMIT=DIR`. A line with + /// that pin is resolved against DIR first, then the robot path. + /// Repeatable; an abbreviated commit matches. + #[arg(long = "commit-tree")] + commit_trees: Vec, + /// Easyconfigs one `--from-pr` pull request changes, at its head, as + /// `N=DIR`. Repeatable. + #[arg(long = "pr-tree")] + pr_trees: Vec, + /// Directory holding the trees of `--from-commit` and `--from-pr` lines + /// that no `--commit-tree` or `--pr-tree` names. A missing commit is + /// downloaded from codeload.github.com (easybuilders/easybuild-easyconfigs, + /// its `.eb` files only); a pull request's changed easyconfigs from + /// api.github.com at its head. `GITHUB_TOKEN` or `GH_TOKEN` is sent to + /// api.github.com when set. + #[arg(long)] + commit_cache: Option, /// Installed modules, one `Name/VersionFull` per line. Repeatable. #[arg(long)] installed: Vec, @@ -1316,15 +1434,21 @@ enum StackCommand { /// Split one Jenkins build list into shards that can build at once. /// /// Each line's closure is computed over the robot path minus the - /// installed modules, as `pr-overlap` computes a root's set. Lines whose - /// own easyconfig another line needs share a shard; any other module the + /// installed modules, as `pr-overlap` computes a root's set. A line with + /// `--from-commit` or `--from-pr` is resolved against that tree first, + /// then the robot path, as EasyBuild resolves it; the trees come from + /// `--commit-tree`, `--pr-tree`, or `--commit-cache`. Lines whose own + /// easyconfig another line needs share a shard; any other module the /// closures of two or more shards need goes to `prelude.list`, which /// builds first. Groups are packed into at most `--shards` shards by /// weight. Writes `prelude.list`, `shard-NN.list` (input lines verbatim, - /// dependencies first), and `plan.json`. A line whose easyconfig or - /// dependency cannot be resolved fails the run and nothing is written. + /// dependencies first), and `plan.json`. A fetched line that still cannot + /// be resolved, or a robot-path easyconfig that does not parse, is + /// planned on its own without a closure and reported on stderr. Any other + /// line whose easyconfig or dependency cannot be resolved fails the run + /// and nothing is written. #[command( - after_long_help = "Examples:\n eb-stack stack shard --build-list rome.list --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs --easyconfigs site-easyconfigs --installed-modules-dir /cvmfs/software.eessi.io/versions/2026.06/software/linux/x86_64/amd/zen2/modules/all --installed-modules-dir site-modules/all --shards 4 --weights build-seconds.tsv --out-dir shards" + after_long_help = "Examples:\n eb-stack stack shard --build-list rome.list --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs --easyconfigs site-easyconfigs --installed-modules-dir /cvmfs/software.eessi.io/versions/2026.06/software/linux/x86_64/amd/zen2/modules/all --installed-modules-dir site-modules/all --shards 4 --weights build-seconds.tsv --out-dir shards\n eb-stack stack shard --build-list rome.list --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs --commit-cache /scratch-shared/$USER/eb-stack-commit-trees --shards 4 --out-dir shards" )] Shard(Box), /// Refuse a plan an architecture cannot build, and flag shared entries @@ -1919,6 +2043,7 @@ fn dispatch() -> Result<()> { Command::Target { command } => run_target(command), Command::Campaign { command } => run_campaign(command), Command::Repro { command } => run_repro(command), + Command::Ledger { command } => run_ledger(command), Command::Mcp { workspace } => { if let Some(workspace) = workspace { std::env::set_var("EB_STACK_WORKSPACE", &workspace); @@ -1935,6 +2060,74 @@ fn dispatch() -> Result<()> { } } +fn run_ledger(command: LedgerCommand) -> Result<()> { + use eb_stack::ledger; + match command { + LedgerCommand::Append { + record, + log, + expect_head, + key, + head_out, + } => { + let report = ledger::append(&ledger::AppendRequest { + log, + record, + expect_head, + key, + head_out, + })?; + println!("{}", report.summary()); + } + LedgerCommand::Head { log, key, out, ts } => { + let head = ledger::sign_head(&ledger::HeadRequest { + log, + key, + out: out.clone(), + ts, + })?; + println!( + "signed head size {} root {} at {} to {}", + head.size, + head.root, + head.ts, + out.display() + ); + } + LedgerCommand::Show { log, package, json } => { + let report = ledger::show(&ledger::ShowRequest { log, package, json })?; + print!("{}", report.stdout); + } + LedgerCommand::Verify { + log, + public_key, + record, + head, + from, + to, + } => { + let mode = match (record, head, from, to) { + (Some(record), Some(head), None, None) => { + ledger::VerifyMode::Inclusion { record, head } + } + (None, None, Some(from), Some(to)) => ledger::VerifyMode::Consistency { from, to }, + _ => { + return Err(usage( + "pass --record N --head HEAD, or --from HEAD1 --to HEAD2", + )) + } + }; + let report = ledger::verify(&ledger::VerifyRequest { + log, + public_key, + mode, + })?; + print!("{}", report.stdout); + } + } + Ok(()) +} + #[derive(clap::Args, Debug)] struct EasyblockReachArgs { /// Easyblock trees. Earlier roots win a class name. @@ -3111,6 +3304,9 @@ fn run_stack(command: StackCommand) -> Result<()> { let ShardArgs { build_list, easyconfigs, + commit_trees, + pr_trees, + commit_cache, installed, installed_modules_dirs, shards, @@ -3124,6 +3320,9 @@ fn run_stack(command: StackCommand) -> Result<()> { &eb_stack::site_commands::StackShardRequest { build_list, easyconfigs, + commit_trees, + pr_trees, + commit_cache, installed, installed_modules_dirs, shards, @@ -3134,6 +3333,9 @@ fn run_stack(command: StackCommand) -> Result<()> { out_dir, }, )?; + for warning in &run.warnings { + eprintln!("warning: {warning}"); + } println!("{}", run.summary()); Ok(()) } diff --git a/src/mcp.rs b/src/mcp.rs index cc61f7c5..0ef278ac 100644 --- a/src/mcp.rs +++ b/src/mcp.rs @@ -458,7 +458,7 @@ fn tool_catalog() -> Vec { ), tool_with_optional( "eb_stack_shard", - "Split one Jenkins build list into shards that can build at the same time, as `stack shard` does. Each line's closure is computed over easyconfigs minus the modules in installed lists and installed_modules_dir Lmod trees. Lines whose own easyconfig another line needs share a shard; any other module two or more shards need goes to prelude.list. Groups are packed into at most shards shards by weight (weights TSV, else default_weight). Writes prelude.list, shard-NN.list (input lines verbatim) and plan.json to out_dir. A line that cannot be resolved is a tool error and nothing is written.", + "Split one Jenkins build list into shards that can build at the same time, as `stack shard` does. Each line's closure is computed over easyconfigs minus the modules in installed lists and installed_modules_dir Lmod trees. Lines whose own easyconfig another line needs share a shard; any other module two or more shards need goes to prelude.list. Groups are packed into at most shards shards by weight (weights TSV, else default_weight). Writes prelude.list, shard-NN.list (input lines verbatim) and plan.json to out_dir. A line with --from-commit or --from-pr is resolved against that tree first, then easyconfigs: commit_tree (COMMIT=DIR), pr_tree (N=DIR), or commit_cache, a directory a missing tree is downloaded into from GitHub; a prelude module only such a tree provides keeps the line's source options. A fetched line that still cannot be resolved, or an easyconfig that does not parse, is planned on its own and listed under warnings. Any other line that cannot be resolved is a tool error and nothing is written.", &[ ("build_list", "string"), ("easyconfigs", "array_min1"), @@ -466,6 +466,9 @@ fn tool_catalog() -> Vec { ("out_dir", "string"), ], &[ + ("commit_tree", "array"), + ("pr_tree", "array"), + ("commit_cache", "string"), ("installed", "array"), ("installed_modules_dir", "array"), ("weights", "string"), @@ -720,6 +723,39 @@ fn tool_catalog() -> Vec { &[("scores", "string")], &[("ratchet", "string"), ("json", "boolean")], ), + tool_with_optional( + "eb_ledger_append", + "Append one record to a per-stack ledger as a canonical JSON line and, with key and head_out, sign the new head. Same writer as `ledger append`.", + &[("log", "string"), ("record", "string")], + &[ + ("expect_head", "string"), + ("key", "string"), + ("head_out", "string"), + ], + ), + tool_with_optional( + "eb_ledger_head", + "Sign a ledger's current head with an Ed25519 key kept outside the log's directory. Same as `ledger head`.", + &[("log", "string"), ("key", "string"), ("out", "string")], + &[("ts", "string")], + ), + tool_with_optional( + "eb_ledger_show", + "List ledger records newest first, optionally for one module or software name. Same table as `ledger show`.", + &[("log", "string")], + &[("package", "string"), ("json", "boolean")], + ), + tool_with_optional( + "eb_ledger_verify", + "Check that a ledger record is in a signed head (record and head), or that one signed head extends another (from and to). Same checks as `ledger verify`.", + &[("log", "string"), ("public_key", "string")], + &[ + ("record", "integer"), + ("head", "string"), + ("from", "string"), + ("to", "string"), + ], + ), ]; if let Some(lint) = tools .iter_mut() @@ -860,6 +896,10 @@ fn call_tool(name: &str, arguments: &Value) -> Result { "eb_component_reach" => component_reach_tool(arguments), "eb_easyblock_reach" => easyblock_reach_tool(arguments), "eb_repro_summary" => repro_summary_tool(arguments), + "eb_ledger_append" => ledger_append_tool(arguments), + "eb_ledger_head" => ledger_head_tool(arguments), + "eb_ledger_show" => ledger_show_tool(arguments), + "eb_ledger_verify" => ledger_verify_tool(arguments), _ => Err(format!("unknown tool: {name}")), } } @@ -1284,7 +1324,7 @@ fn recipe_check(arguments: &Value) -> Result { && maintainer.ok_for_upstream() && easyblock_error.is_none() && !source_failed - && !(strict && !skipped.is_empty()) + && (!strict || skipped.is_empty()) && duplicate_error.is_none() && (metadata_only || (dep_ok && toolchain_verified)); let skipped_text: Vec = skipped.iter().map(ToString::to_string).collect(); @@ -3160,6 +3200,9 @@ fn stack_shard_tool(arguments: &Value) -> Result { let request = crate::site_commands::StackShardRequest { build_list: required_path(arguments, "build_list")?, easyconfigs: path_array(arguments, "easyconfigs")?, + commit_trees: string_array(arguments, "commit_tree")?, + pr_trees: string_array(arguments, "pr_tree")?, + commit_cache: optional_path(arguments, "commit_cache"), installed: path_array(arguments, "installed")?, installed_modules_dirs: path_array(arguments, "installed_modules_dir")?, shards: usize::try_from(shards).map_err(|error| format!("shards: {error}"))?, @@ -3175,6 +3218,8 @@ fn stack_shard_tool(arguments: &Value) -> Result { "shards": run.shards, "prelude": run.prelude, "lines": run.lines, + "opaque": run.opaque, + "warnings": run.warnings, })) } @@ -3550,6 +3595,89 @@ fn repro_summary_tool(arguments: &Value) -> Result { })) } +fn ledger_append_tool(arguments: &Value) -> Result { + let report = crate::ledger::append(&crate::ledger::AppendRequest { + log: required_path(arguments, "log")?, + record: required_path(arguments, "record")?, + expect_head: optional_path(arguments, "expect_head"), + key: optional_path(arguments, "key"), + head_out: optional_path(arguments, "head_out"), + }) + .map_err(workflow_error)?; + Ok(json!({ + "text": report.summary(), + "index": report.index, + "id": report.id, + "size": report.head.size, + "root": report.head.root, + "signed": report.signed, + "replayed": report.replayed, + })) +} + +fn ledger_head_tool(arguments: &Value) -> Result { + let out = required_path(arguments, "out")?; + let head = crate::ledger::sign_head(&crate::ledger::HeadRequest { + log: required_path(arguments, "log")?, + key: required_path(arguments, "key")?, + out: out.clone(), + ts: optional_string(arguments, "ts"), + }) + .map_err(workflow_error)?; + Ok(json!({ + "text": format!( + "signed head size {} root {} at {} to {}", + head.size, + head.root, + head.ts, + out.display() + ), + "head": head, + })) +} + +fn ledger_show_tool(arguments: &Value) -> Result { + let report = crate::ledger::show(&crate::ledger::ShowRequest { + log: required_path(arguments, "log")?, + package: optional_string(arguments, "package"), + json: optional_bool(arguments, "json"), + }) + .map_err(workflow_error)?; + Ok(json!({ + "text": report.stdout, + "records": report.records, + })) +} + +fn ledger_verify_tool(arguments: &Value) -> Result { + let record = optional_u64(arguments, "record")? + .map(|value| usize::try_from(value).map_err(|_| "argument record is too large".to_string())) + .transpose()?; + let mode = match ( + record, + optional_path(arguments, "head"), + optional_path(arguments, "from"), + optional_path(arguments, "to"), + ) { + (Some(record), Some(head), None, None) => { + crate::ledger::VerifyMode::Inclusion { record, head } + } + (None, None, Some(from), Some(to)) => crate::ledger::VerifyMode::Consistency { from, to }, + _ => return Err("pass record and head, or from and to".to_string()), + }; + let report = crate::ledger::verify(&crate::ledger::VerifyRequest { + log: required_path(arguments, "log")?, + public_key: required_path(arguments, "public_key")?, + mode, + }) + .map_err(workflow_error)?; + Ok(json!({ + "text": report.stdout, + "kind": report.kind, + "proof": report.proof, + })) +} + fn pr_overlap(arguments: &Value) -> Result { let kind = match optional_string(arguments, "kind") .as_deref() diff --git a/src/pr_overlap.rs b/src/pr_overlap.rs index ca070411..db4f0ab9 100644 --- a/src/pr_overlap.rs +++ b/src/pr_overlap.rs @@ -2546,6 +2546,36 @@ fn fetch_one_pull( })) } +/// The head commit of one pull request and the easyconfig files it adds or +/// changes there, with their head text: the files `eb --from-pr` downloads +/// and puts first on the robot path. A removed file is left out. +pub(crate) fn pull_head_easyconfigs( + http: &dyn HttpGet, + repo: &str, + number: u64, +) -> anyhow::Result<(String, Vec<(String, String)>)> { + check_repo(repo)?; + let url = format!("https://api.github.com/repos/{repo}/pulls/{number}"); + let pull: ListedPull = + serde_json::from_value(get_json(http, &url)?).with_context(|| format!("parse {url}"))?; + check_token("head sha", &pull.head.sha)?; + let listed = list_pull_files(http, repo, number, 100)?; + if listed.truncated { + bail!("pull request {number}: the changed-file list stops at GitHub's limit"); + } + let mut files = Vec::new(); + for file in listed.files { + if file.status == "removed" || !easyconfig_path(&file.filename) { + continue; + } + let url = raw_url(repo, &pull.head.sha, &file.filename); + let text = get_raw(http, &url)? + .ok_or_else(|| anyhow::anyhow!("pull request {number}: {url}: HTTP 404"))?; + files.push((file.filename, text)); + } + Ok((pull.head.sha, files)) +} + /// Reject a repository or base ref that cannot be placed in a GitHub URL. /// /// An empty or absent base ref means `develop` and is not checked. This does diff --git a/src/shard.rs b/src/shard.rs index 47b0323c..b650de93 100644 --- a/src/shard.rs +++ b/src/shard.rs @@ -39,6 +39,85 @@ pub struct ShardLine { /// Whether EasyBuild fetches the line's easyconfig itself /// (`--from-pr` or `--from-commit`), so it need not be on the robot path. pub fetched: bool, + /// Where EasyBuild fetches the easyconfigs from, when `fetched`. + pub source: Option, + /// The options that say where the line's easyconfigs and easyblocks come + /// from (`--from-commit`, `--from-pr`, `--include-easyblocks-from-commit`, + /// `--include-easyblocks-from-pr`), as written. A prelude line for a + /// module this line's tree provides carries them, so it builds the same + /// easyconfig with the same easyblock. + pub source_options: Vec, +} + +impl ShardLine { + /// A dependency-filter override that the list-wide closure cannot model. + /// EasyBuild accepts unambiguous prefixes of its long option names. + pub fn filter_deps_option(&self) -> Option<&str> { + const OPTION: &str = "--filter-deps"; + const SHORTEST_PREFIX: usize = "--filter-d".len(); + self.text + .split_whitespace() + .skip(1) + .map(|word| word.split('=').next().unwrap_or(word)) + .find(|key| key.len() >= SHORTEST_PREFIX && OPTION.starts_with(key)) + } +} + +/// The easyconfig tree a fetched line is resolved against. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] +pub enum LineSource { + /// `--from-commit=SHA`: the easyconfigs of that commit. + Commit(String), + /// `--from-pr=N[,M...]`: the easyconfig files those pull requests change, + /// at their heads. + Prs(Vec), + /// The option is there but its value cannot be used; the text says why. + Invalid(String), +} + +impl LineSource { + /// The option as EasyBuild takes it, `--from-commit=SHA` or + /// `--from-pr=N,M`. + pub fn option(&self) -> String { + match self { + Self::Commit(sha) => format!("--from-commit={sha}"), + Self::Prs(prs) => format!( + "--from-pr={}", + prs.iter().map(u64::to_string).collect::>().join(",") + ), + Self::Invalid(why) => why.clone(), + } + } +} + +/// A commit as `--from-commit` takes it: 7 to 40 hexadecimal digits. +pub fn is_commit_sha(value: &str) -> bool { + (7..=40).contains(&value.len()) && value.chars().all(|c| c.is_ascii_hexdigit()) +} + +fn line_source(commit: Option<&str>, prs: Option<&str>) -> Option { + match (commit, prs) { + (None, None) => None, + (Some(_), Some(_)) => Some(LineSource::Invalid( + "both --from-commit and --from-pr are given".into(), + )), + (Some(sha), None) if is_commit_sha(sha) => Some(LineSource::Commit(sha.to_string())), + (Some(sha), None) => Some(LineSource::Invalid(format!( + "--from-commit wants a commit sha, got '{sha}'" + ))), + (None, Some(list)) => { + let parsed: Option> = list + .split(',') + .map(|n| n.trim().parse::().ok().filter(|n| *n > 0)) + .collect(); + match parsed { + Some(prs) if !prs.is_empty() => Some(LineSource::Prs(prs)), + _ => Some(LineSource::Invalid(format!( + "--from-pr wants pull request numbers, got '{list}'" + ))), + } + } + } } /// Parse a build list. Blank lines and `#` comments are skipped; every other @@ -55,17 +134,57 @@ pub fn parse_lines(text: &str) -> Vec { } let mut words = trimmed.split_whitespace(); let file = words.next()?.to_string(); - let keys: Vec<&str> = words.map(|w| w.split('=').next().unwrap_or(w)).collect(); + let words: Vec<&str> = words.collect(); + let keys: Vec<&str> = words + .iter() + .map(|w| w.split('=').next().unwrap_or(w)) + .collect(); let forced = keys.iter().any(|k| *k == "--rebuild" || *k == "--force"); let fetched = keys .iter() .any(|k| *k == "--from-pr" || *k == "--from-commit"); + // An option's value is after `=`, or the next word when the + // option is written on its own, as EasyBuild's parser takes both. + let mut commit = None; + let mut prs = None; + let mut source_options = Vec::new(); + let mut j = 0; + while j < words.len() { + let key = keys[j]; + let sourced = matches!( + key, + "--from-commit" + | "--from-pr" + | "--include-easyblocks-from-commit" + | "--include-easyblocks-from-pr" + ); + if !sourced { + j += 1; + continue; + } + let (value, used) = match words[j].split_once('=') { + Some((_, value)) => (value, 1), + None => match words.get(j + 1).filter(|w| !w.starts_with("--")) { + Some(next) => (*next, 2), + None => ("", 1), + }, + }; + source_options.push(format!("{key}={value}")); + match key { + "--from-commit" => commit = Some(value), + "--from-pr" => prs = Some(value), + _ => {} + } + j += used; + } Some(ShardLine { number: i + 1, text: line.to_string(), file, forced, fetched, + source: line_source(commit, prs), + source_options, }) }) .collect() @@ -272,6 +391,34 @@ pub fn plan( } } +/// Merge dependency orders into one order every input agrees with: a module +/// comes after each module that precedes it in any input. A module goes next +/// when it heads the first input that still has modules and no input has it +/// further back. Orders that disagree, which closures over one robot path do +/// not, fall back to the first input's head. +pub fn merge_orders(orders: &[Vec]) -> Vec { + let mut rest: Vec> = orders + .iter() + .map(|o| o.iter().map(String::as_str).collect()) + .collect(); + let mut placed: BTreeSet<&str> = BTreeSet::new(); + let mut out = Vec::new(); + loop { + for order in &mut rest { + order.retain(|m| !placed.contains(m)); + } + let heads: Vec<&str> = rest.iter().filter_map(|o| o.first().copied()).collect(); + let Some(&first) = heads.first() else { + break; + }; + let free = |m: &str| rest.iter().all(|o| !o.iter().skip(1).any(|x| *x == m)); + let pick = heads.iter().copied().find(|m| free(m)).unwrap_or(first); + placed.insert(pick); + out.push(pick.to_string()); + } + out +} + /// Read a weights file: `module`, a tab, then `seconds` per line. `#` comments /// and blank lines are skipped. A fractional value rounds up to whole seconds. /// A later line for the same module replaces an earlier one. @@ -346,6 +493,49 @@ mod tests { ); } + #[test] + fn a_line_names_the_tree_easybuild_fetches() { + let lines = parse_lines( + "A-1.eb --from-commit=0123abcd --include-easyblocks-from-commit=4567ef01 --include-systems=x\n\ + B-2.eb --from-pr 27108,26954\n\ + C-3.eb --from-commit=main\n\ + D-4.eb --from-pr=1 --from-commit=0123abcd\n\ + E-5.eb --include-easyblocks-from-pr=4241\n", + ); + assert_eq!(lines[0].source, Some(LineSource::Commit("0123abcd".into()))); + assert_eq!( + lines[0].source_options, + [ + "--from-commit=0123abcd", + "--include-easyblocks-from-commit=4567ef01" + ] + ); + assert_eq!(lines[1].source, Some(LineSource::Prs(vec![27108, 26954]))); + assert_eq!(lines[1].source_options, ["--from-pr=27108,26954"]); + assert_eq!( + lines[1].source.as_ref().unwrap().option(), + "--from-pr=27108,26954" + ); + assert!(matches!(lines[2].source, Some(LineSource::Invalid(_)))); + assert!(matches!(lines[3].source, Some(LineSource::Invalid(_)))); + assert_eq!(lines[4].source, None); + assert_eq!( + lines[4].source_options, + ["--include-easyblocks-from-pr=4241"] + ); + } + + #[test] + fn merged_orders_keep_every_input_order() { + let a: Vec = ["M/1", "A/1"].map(String::from).to_vec(); + let b: Vec = ["N/1", "P/1", "M/1", "B/1"].map(String::from).to_vec(); + let merged = merge_orders(&[a, b]); + let at = |m: &str| merged.iter().position(|x| x == m).unwrap(); + assert_eq!(merged.len(), 5); + assert!(at("N/1") < at("P/1") && at("P/1") < at("M/1")); + assert!(at("M/1") < at("A/1") && at("M/1") < at("B/1")); + } + #[test] fn a_shared_dependency_goes_to_the_prelude_and_the_lines_stay_apart() { let closures = [ diff --git a/src/shard_trees.rs b/src/shard_trees.rs new file mode 100644 index 00000000..c5aa115c --- /dev/null +++ b/src/shard_trees.rs @@ -0,0 +1,518 @@ +//! Easyconfig trees for build-list lines whose easyconfigs EasyBuild fetches +//! from GitHub. +//! +//! `eb --from-commit=SHA` downloads the easybuild-easyconfigs archive of that +//! commit and puts its `easybuild/easyconfigs` first on the robot path. +//! `eb --from-pr=N` downloads the easyconfig files pull request N changes, at +//! its head, and puts them first. `stack shard` resolves such a line the same +//! way: the line's tree first, then the robot path. +//! +//! A tree is found, in this order, in an explicit `COMMIT=DIR` or `N=DIR` +//! entry, in the cache directory, or by downloading it into the cache. A +//! commit is served from +//! `https://codeload.github.com//tar.gz/`; a commit on a fork +//! that a pull request carries is served through the upstream repository too. +//! Only the `.eb` files under `easybuild/easyconfigs/` are extracted. A pull +//! request's head commit and changed files come from `api.github.com` and the +//! file text from `raw.githubusercontent.com`; its cache entry is keyed by +//! the head commit, so a pull request that moves is fetched again. +//! +//! A tree is written under a temporary name and renamed into place, so a +//! cache entry that exists is complete, and two planners filling one cache at +//! once both end with the same entry. + +use crate::pr_overlap::HttpGet; +use crate::shard::is_commit_sha; +use std::collections::BTreeMap; +use std::path::{Component, Path, PathBuf}; + +/// The repository `--from-commit` and `--from-pr` read. +pub const EASYCONFIGS_REPO: &str = "easybuilders/easybuild-easyconfigs"; + +/// File in a downloaded commit tree listing `sha256 relative/path.eb` for +/// every easyconfig in it, so a planner can skip the files the robot path +/// already has without reading them. +pub const MANIFEST: &str = ".eb-stack-sha256"; + +/// Prefix on a commit download that received no HTTP status. +/// +/// `run_stack_shard_with` treats a message that starts with this as a +/// failure and writes nothing. An HTTP status, including 404, stays a +/// per-line warning: the server answered, and the commit is simply absent. +pub const CACHE_DOWNLOAD_FAILED: &str = "cache download failed"; + +/// Where the trees of fetched lines come from. +pub struct TreeSources<'a> { + /// `--commit-tree COMMIT=DIR`: a commit, full or abbreviated, to an + /// easyconfigs directory. + pub commit_trees: BTreeMap, + /// `--pr-tree N=DIR`: a pull request to a directory of its easyconfigs. + pub pr_trees: BTreeMap, + /// `--commit-cache DIR`: trees found or downloaded there. + pub cache: Option, + /// The client downloads go through. + pub http: &'a dyn HttpGet, + /// `owner/name` of the easyconfigs repository. + pub repo: String, +} + +impl TreeSources<'_> { + /// Whether any source is configured. + pub fn any(&self) -> bool { + !self.commit_trees.is_empty() || !self.pr_trees.is_empty() || self.cache.is_some() + } + + /// The easyconfigs directory of `sha`, or why there is none. + pub fn commit_tree(&self, sha: &str) -> Result { + if !is_commit_sha(sha) { + return Err(format!("{sha} is not a commit sha")); + } + let given: Vec<&PathBuf> = self + .commit_trees + .iter() + .filter(|(key, _)| same_commit(key, sha)) + .map(|(_, dir)| dir) + .collect(); + match given.as_slice() { + [dir] => return Ok((*dir).clone()), + [] => {} + _ => return Err(format!("--commit-tree names {sha} more than once")), + } + let Some(cache) = &self.cache else { + return Err(format!( + "no easyconfigs for commit {sha}: pass --commit-tree {sha}=DIR or --commit-cache DIR" + )); + }; + let entry = cache.join(sha.to_ascii_lowercase()); + let tree = entry.join("easybuild/easyconfigs"); + if tree.is_dir() { + return Ok(tree); + } + let url = format!("https://codeload.github.com/{}/tar.gz/{sha}", self.repo); + let response = self.http.get(&url).map_err(|error| { + format!( + "{CACHE_DOWNLOAD_FAILED}: cache {} has no tree for commit {sha}, and downloading {url} failed: {error}", + cache.display() + ) + })?; + if response.status != 200 { + return Err(format!("commit {sha}: {url}: HTTP {}", response.status)); + } + fill(cache, &entry, |tmp| { + extract_easyconfigs(&response.body, tmp) + }) + .map_err(|error| format!("commit {sha}: {url}: {error}"))?; + Ok(tree) + } + + /// The directory of pull request `number`'s easyconfigs at its head, or + /// why there is none. + pub fn pr_tree(&self, number: u64) -> Result { + if let Some(dir) = self.pr_trees.get(&number) { + return Ok(dir.clone()); + } + let Some(cache) = &self.cache else { + return Err(format!( + "no easyconfigs for pull request {number}: pass --pr-tree {number}=DIR or --commit-cache DIR" + )); + }; + let (head, files) = crate::pr_overlap::pull_head_easyconfigs(self.http, &self.repo, number) + .map_err(|error| format!("pull request {number}: {error:#}"))?; + let entry = cache.join(format!("pr-{number}-{}", head.to_ascii_lowercase())); + if entry.is_dir() { + return Ok(entry); + } + fill(cache, &entry, |tmp| { + for (path, text) in &files { + let dst = tmp.join(safe_relative(path)?); + std::fs::create_dir_all(dst.parent().expect("a file has a parent")) + .map_err(|error| error.to_string())?; + std::fs::write(&dst, text).map_err(|error| error.to_string())?; + } + Ok(files.len()) + }) + .map_err(|error| format!("pull request {number}: {error}"))?; + Ok(entry) + } +} + +/// Whether a `--commit-tree` key and a build-list sha name one commit: equal, +/// or one an abbreviation of the other. +fn same_commit(key: &str, sha: &str) -> bool { + let (key, sha) = (key.to_ascii_lowercase(), sha.to_ascii_lowercase()); + key.len() >= 7 && sha.len() >= 7 && (key.starts_with(&sha) || sha.starts_with(&key)) +} + +/// Write a cache entry through `write` into a temporary directory and rename +/// it to `entry`. An entry another process renamed into place first wins. +fn fill( + cache: &Path, + entry: &Path, + write: impl FnOnce(&Path) -> Result, +) -> Result<(), String> { + std::fs::create_dir_all(cache).map_err(|error| format!("{}: {error}", cache.display()))?; + let name = entry + .file_name() + .expect("a cache entry has a name") + .to_string_lossy(); + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or_default(); + let tmp = cache.join(format!(".{name}.tmp-{}-{nanos}", std::process::id())); + let written = write(&tmp); + let result = match written { + Ok(0) => Err("no easyconfig files".to_string()), + Ok(_) => match std::fs::rename(&tmp, entry) { + Ok(()) => Ok(()), + Err(_) if entry.is_dir() => Ok(()), + Err(error) => Err(format!("rename to {}: {error}", entry.display())), + }, + Err(error) => Err(error), + }; + if tmp.exists() { + let _ = std::fs::remove_dir_all(&tmp); + } + result +} + +/// A repository path as a relative path with no `..`, root, or prefix. +fn safe_relative(path: &str) -> Result { + let relative = Path::new(path); + if path.is_empty() + || relative + .components() + .any(|c| !matches!(c, Component::Normal(_))) + { + return Err(format!("unsafe path in the archive: {path}")); + } + Ok(relative.to_path_buf()) +} + +/// Extract the `.eb` files under `easybuild/easyconfigs/` of a GitHub +/// archive (``-/`` first) into `dest`, at the same relative paths. +/// Returns how many were written. +pub fn extract_easyconfigs(archive: &[u8], dest: &Path) -> Result { + use sha2::{Digest, Sha256}; + use std::io::Read; + let decoder = flate2::read::GzDecoder::new(std::io::Cursor::new(archive)); + let mut tar = tar::Archive::new(decoder); + let mut count = 0; + let mut manifest = String::new(); + for entry in tar.entries().map_err(|error| error.to_string())? { + let mut entry = entry.map_err(|error| error.to_string())?; + if !entry.header().entry_type().is_file() { + continue; + } + let path = entry + .path() + .map_err(|error| error.to_string())? + .to_string_lossy() + .into_owned(); + let Some((_, inner)) = path.split_once('/') else { + continue; + }; + if !inner.starts_with("easybuild/easyconfigs/") || !inner.ends_with(".eb") { + continue; + } + let dst = dest.join(safe_relative(inner)?); + std::fs::create_dir_all(dst.parent().expect("a file has a parent")) + .map_err(|error| error.to_string())?; + let mut bytes = Vec::new(); + entry + .read_to_end(&mut bytes) + .map_err(|error| error.to_string())?; + std::fs::write(&dst, &bytes).map_err(|error| error.to_string())?; + let digest: String = Sha256::digest(&bytes) + .iter() + .map(|b| format!("{b:02x}")) + .collect(); + let relative = &inner["easybuild/easyconfigs/".len()..]; + manifest.push_str(&format!("{digest} {relative}\n")); + count += 1; + } + if count > 0 { + let tree = dest.join("easybuild/easyconfigs"); + std::fs::write(tree.join(MANIFEST), manifest).map_err(|error| error.to_string())?; + } + Ok(count) +} + +/// The easyconfigs of `tree` that differ from the robot path's: a file whose +/// name and content match the robot path's effective easyconfig of that name +/// resolves the same either way and is left out. `base` maps a file name to +/// the SHA-256 of the robot path's file. A tree's [`MANIFEST`], when present, +/// stands in for reading its files. Returns the parsed candidates and the +/// files that did not parse. +pub fn overlay_candidates( + tree: &Path, + base: &BTreeMap, +) -> Result<(Vec, Vec), String> { + use sha2::{Digest, Sha256}; + let hex = |bytes: &[u8]| -> String { + Sha256::digest(bytes) + .iter() + .map(|b| format!("{b:02x}")) + .collect() + }; + let same = |relative: &str, digest: &str| { + let name = relative.rsplit('/').next().unwrap_or(relative); + base.get(name).is_some_and(|known| known == digest) + }; + let mut files: Vec = Vec::new(); + match std::fs::read_to_string(tree.join(MANIFEST)) { + Ok(manifest) => { + for line in manifest.lines() { + let Some((digest, relative)) = line.split_once(" ") else { + continue; + }; + if !same(relative, digest) { + files.push(tree.join(safe_relative(relative)?)); + } + } + } + Err(_) => { + if !tree.is_dir() { + return Err(format!("{} is not a directory", tree.display())); + } + let mut stack = vec![tree.to_path_buf()]; + while let Some(dir) = stack.pop() { + let entries = std::fs::read_dir(&dir) + .map_err(|error| format!("{}: {error}", dir.display()))?; + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + if entry.file_name() != ".git" { + stack.push(path); + } + continue; + } + if path.extension().and_then(|e| e.to_str()) != Some("eb") { + continue; + } + let bytes = std::fs::read(&path) + .map_err(|error| format!("{}: {error}", path.display()))?; + let name = entry.file_name().to_string_lossy().into_owned(); + if !same(&name, &hex(&bytes)) { + files.push(path); + } + } + } + } + } + files.sort(); + let mut candidates = Vec::new(); + let mut skipped = Vec::new(); + for path in files { + match crate::eb_parse::parse_easyconfig_file(&path) { + Ok(candidate) => candidates.push(candidate), + Err(error) => skipped.push(format!("{}: {error}", path.display())), + } + } + Ok((candidates, skipped)) +} + +#[cfg(test)] +pub(crate) mod tests { + use super::*; + use crate::pr_overlap::HttpResponse; + use std::cell::RefCell; + + /// GET from a map; every URL asked for is recorded. + #[derive(Default)] + pub(crate) struct MapHttp { + pub pages: BTreeMap>, + pub asked: RefCell>, + } + + impl HttpGet for MapHttp { + fn get(&self, url: &str) -> Result { + self.asked.borrow_mut().push(url.to_string()); + Ok(match self.pages.get(url) { + Some(body) => HttpResponse { + status: 200, + body: body.clone(), + link: None, + }, + None => HttpResponse { + status: 404, + body: b"Not Found".to_vec(), + link: None, + }, + }) + } + } + + /// A GitHub-shaped `.tar.gz`: every file under `/`. + pub(crate) fn archive(top: &str, files: &[(&str, &str)]) -> Vec { + let mut builder = tar::Builder::new(Vec::new()); + for (path, text) in files { + let mut header = tar::Header::new_gnu(); + header.set_size(text.len() as u64); + header.set_mode(0o644); + header.set_cksum(); + builder + .append_data(&mut header, format!("{top}/{path}"), text.as_bytes()) + .unwrap(); + } + let tar = builder.into_inner().unwrap(); + let mut gz = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::fast()); + std::io::Write::write_all(&mut gz, &tar).unwrap(); + gz.finish().unwrap() + } + + fn sources<'a>(http: &'a MapHttp, cache: Option) -> TreeSources<'a> { + TreeSources { + commit_trees: BTreeMap::new(), + pr_trees: BTreeMap::new(), + cache, + http, + repo: EASYCONFIGS_REPO.into(), + } + } + + const SHA: &str = "0d6b80c34f3553774e93162cf89088e2423ecf89"; + + #[test] + fn a_commit_is_downloaded_once_and_only_its_easyconfigs_are_kept() { + let dir = tempfile::tempdir().unwrap(); + let mut http = MapHttp::default(); + http.pages.insert( + format!("https://codeload.github.com/{EASYCONFIGS_REPO}/tar.gz/{SHA}"), + archive( + &format!("easybuild-easyconfigs-{SHA}"), + &[ + ( + "easybuild/easyconfigs/p/petsc4py/petsc4py-3.25.0-foss-2026.1.eb", + "name = 'petsc4py'\n", + ), + ( + "easybuild/easyconfigs/p/petsc4py/petsc4py-fix.patch", + "--- a\n", + ), + ("test/framework/x.eb", "name = 'x'\n"), + ("setup.py", "\n"), + ], + ), + ); + let trees = sources(&http, Some(dir.path().join("cache"))); + let tree = trees.commit_tree(SHA).unwrap(); + assert_eq!( + tree, + dir.path() + .join("cache") + .join(SHA) + .join("easybuild/easyconfigs") + ); + assert!(tree + .join("p/petsc4py/petsc4py-3.25.0-foss-2026.1.eb") + .is_file()); + assert!(!tree.join("p/petsc4py/petsc4py-fix.patch").exists()); + assert!(!dir.path().join("cache").join(SHA).join("test").exists()); + let manifest = std::fs::read_to_string(tree.join(MANIFEST)).unwrap(); + assert!( + manifest.ends_with(" p/petsc4py/petsc4py-3.25.0-foss-2026.1.eb\n"), + "{manifest}" + ); + assert_eq!(manifest.lines().count(), 1); + // The second lookup is the cache: no request. + assert_eq!(trees.commit_tree(SHA).unwrap(), tree); + assert_eq!(http.asked.borrow().len(), 1); + // No temporary directory is left behind. + let names: Vec = std::fs::read_dir(dir.path().join("cache")) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + assert_eq!(names, [SHA]); + } + + /// GET that fails before an HTTP status, as a connect error does. + struct OfflineHttp; + + impl HttpGet for OfflineHttp { + fn get(&self, _url: &str) -> Result { + // The client returns this text; the planner must keep it. + Err("tcp connect error: Network is unreachable".into()) + } + } + + #[test] + fn a_cache_miss_with_no_network_names_the_cache_and_the_url() { + let dir = tempfile::tempdir().unwrap(); + let cache = dir.path().join("cache"); + let trees = TreeSources { + commit_trees: BTreeMap::new(), + pr_trees: BTreeMap::new(), + cache: Some(cache.clone()), + http: &OfflineHttp, + repo: EASYCONFIGS_REPO.into(), + }; + let why = trees.commit_tree(SHA).unwrap_err(); + let url = format!("https://codeload.github.com/{EASYCONFIGS_REPO}/tar.gz/{SHA}"); + assert!(why.starts_with(CACHE_DOWNLOAD_FAILED), "{why}"); + assert!(why.contains(&cache.display().to_string()), "{why}"); + assert!(why.contains(&format!("no tree for commit {SHA}")), "{why}"); + assert!(why.contains(&url), "{why}"); + assert!(why.contains("Network is unreachable"), "{why}"); + assert!(!cache.join(SHA).exists()); + } + + #[test] + fn a_missing_commit_says_why_and_leaves_no_cache_entry() { + let dir = tempfile::tempdir().unwrap(); + let http = MapHttp::default(); + let trees = sources(&http, Some(dir.path().join("cache"))); + let why = trees.commit_tree(SHA).unwrap_err(); + assert!(why.contains("HTTP 404"), "{why}"); + assert!(!dir.path().join("cache").join(SHA).exists()); + let none = sources(&http, None).commit_tree(SHA).unwrap_err(); + assert!(none.contains("--commit-cache"), "{none}"); + assert!(sources(&http, None).commit_tree("../etc").is_err()); + } + + #[test] + fn an_explicit_tree_matches_an_abbreviated_commit() { + let http = MapHttp::default(); + let mut trees = sources(&http, None); + trees + .commit_trees + .insert(SHA[..12].to_string(), PathBuf::from("/trees/petsc4py")); + assert_eq!( + trees.commit_tree(SHA).unwrap(), + PathBuf::from("/trees/petsc4py") + ); + assert!(http.asked.borrow().is_empty()); + } + + #[test] + fn a_pull_request_tree_is_its_changed_easyconfigs_at_its_head() { + let dir = tempfile::tempdir().unwrap(); + let mut http = MapHttp::default(); + let api = format!("https://api.github.com/repos/{EASYCONFIGS_REPO}/pulls/26954"); + http.pages.insert( + api.clone(), + format!(r#"{{"number":26954,"title":"VASP","head":{{"sha":"{SHA}"}}}}"#).into_bytes(), + ); + http.pages.insert( + format!("{api}/files?per_page=100&page=1"), + br#"[{"filename":"easybuild/easyconfigs/v/VASP/VASP-6.6.1-foss-2026.1.eb","status":"added"}, + {"filename":"easybuild/easyconfigs/o/Old/Old-1.eb","status":"removed"}, + {"filename":"README.md","status":"modified"}]"# + .to_vec(), + ); + http.pages.insert( + format!("https://raw.githubusercontent.com/{EASYCONFIGS_REPO}/{SHA}/easybuild/easyconfigs/v/VASP/VASP-6.6.1-foss-2026.1.eb"), + b"name = 'VASP'\n".to_vec(), + ); + let trees = sources(&http, Some(dir.path().join("cache"))); + let tree = trees.pr_tree(26954).unwrap(); + assert_eq!( + tree, + dir.path().join("cache").join(format!("pr-26954-{SHA}")) + ); + assert!(tree + .join("easybuild/easyconfigs/v/VASP/VASP-6.6.1-foss-2026.1.eb") + .is_file()); + assert!(!tree.join("easybuild/easyconfigs/o").exists()); + } +} diff --git a/src/site_commands.rs b/src/site_commands.rs index 044291e9..cfe90792 100644 --- a/src/site_commands.rs +++ b/src/site_commands.rs @@ -2204,6 +2204,12 @@ pub struct StackShardRequest { pub build_list: PathBuf, /// Robot-path easyconfig trees; later trees take precedence. pub easyconfigs: Vec, + /// `COMMIT=DIR`: the easyconfigs of one `--from-commit` pin. + pub commit_trees: Vec, + /// `N=DIR`: the easyconfigs one `--from-pr` pull request changes. + pub pr_trees: Vec, + /// Directory the trees of fetched lines are found in or downloaded to. + pub commit_cache: Option, /// Installed-module lists, one `Name/VersionFull` per line. pub installed: Vec, /// Lmod `modules/all` trees whose modules count as installed. @@ -2223,6 +2229,7 @@ pub struct StackShardRequest { } /// What `stack shard` wrote. +#[derive(Debug)] pub struct StackShardRun { /// Shards written. pub shards: usize, @@ -2230,15 +2237,23 @@ pub struct StackShardRun { pub prelude: usize, /// Build-list lines. pub lines: usize, + /// Lines planned without a closure. + pub opaque: usize, + /// One line per opaque line and per other planning caveat, for stderr. + pub warnings: Vec, } impl StackShardRun { /// One line for the CLI. pub fn summary(&self) -> String { - format!( + let mut text = format!( "shards={} prelude={} lines={}", self.shards, self.prelude, self.lines - ) + ); + if self.opaque > 0 { + text.push_str(&format!(" opaque={}", self.opaque)); + } + text } } @@ -2292,45 +2307,97 @@ fn sha256_hex(bytes: &[u8]) -> String { .collect() } -/// Whether a file named `name` exists anywhere under one of `roots`. -fn file_under(roots: &[PathBuf], name: &str) -> bool { - let mut stack: Vec = roots.to_vec(); - while let Some(dir) = stack.pop() { - let Ok(entries) = std::fs::read_dir(&dir) else { - continue; - }; - for entry in entries.flatten() { - let path = entry.path(); - let Ok(kind) = entry.file_type() else { +/// Find an explicit filename using the robot trees' later-path precedence. +fn file_on_robot(roots: &[PathBuf], name: &str) -> Option { + for root in roots.iter().rev() { + let mut stack = vec![root.clone()]; + while let Some(dir) = stack.pop() { + let Ok(entries) = std::fs::read_dir(&dir) else { continue; }; - if kind.is_dir() { - if entry.file_name() != ".git" { - stack.push(path); + for entry in entries.flatten() { + let path = entry.path(); + let Ok(kind) = entry.file_type() else { + continue; + }; + if kind.is_dir() { + if entry.file_name() != ".git" { + stack.push(path); + } + } else if entry.file_name() == name { + return Some(path); } - } else if entry.file_name() == name { - return true; } } } - false + None +} + +/// Whether a file named `name` exists anywhere under one of `roots`. +fn file_under(roots: &[PathBuf], name: &str) -> bool { + file_on_robot(roots, name).is_some() } /// Split a build list into shards whose builds do not overlap, plus a prelude -/// of the dependencies two or more shards need. A line eb-stack cannot -/// resolve is planned as an opaque line when EasyBuild fetches its easyconfig -/// (`--from-pr`, `--from-commit`) or the file is on the robot path but does -/// not parse; any other unresolved line writes nothing. +/// of the dependencies two or more shards need. +/// +/// A line with `--from-commit` or `--from-pr` is resolved the way EasyBuild +/// resolves it: against that commit's or pull request's easyconfigs first, +/// then the robot path. Its trees come from `--commit-tree`, `--pr-tree`, or +/// `--commit-cache`, which downloads a missing one. A prelude line for a +/// module only such a tree provides carries the line's source options. A line +/// eb-stack still cannot resolve is planned as an opaque line, and reported, +/// when EasyBuild fetches its easyconfig or the file is on the robot path but +/// does not parse; any other unresolved line writes nothing. pub fn run_stack_shard(args: &StackShardRequest) -> Result { + run_stack_shard_with(args, &pr_overlap::UreqGet) +} + +/// [`run_stack_shard`] with downloads through `http`. +pub fn run_stack_shard_with( + args: &StackShardRequest, + http: &dyn pr_overlap::HttpGet, +) -> Result { use crate::build_order::{closure_against_installed, module_name, Choice, ModuleKey}; use crate::domain::Candidate; - use crate::shard::{parse_lines, parse_weights, plan, LineClosure}; + use crate::eb_parse::merge_candidates_with_precedence; + use crate::shard::{ + is_commit_sha, merge_orders, parse_lines, parse_weights, plan, LineClosure, LineSource, + }; + use crate::shard_trees::{overlay_candidates, TreeSources, EASYCONFIGS_REPO}; use serde_json::json; use std::collections::{BTreeMap, BTreeSet}; + const FETCHED: &str = "fetched by EasyBuild (--from-pr or --from-commit)"; + const UNPARSED: &str = "on the robot path, not parsed by eb-stack"; + if args.shards == 0 { return Err(usage("--shards must be at least 1")); } + let mut commit_trees = BTreeMap::new(); + for spec in &args.commit_trees { + let (commit, dir) = spec + .split_once('=') + .filter(|(c, d)| is_commit_sha(c) && !d.is_empty()) + .ok_or_else(|| usage(format!("--commit-tree wants COMMIT=DIR, got {spec}")))?; + commit_trees.insert(commit.to_string(), PathBuf::from(dir)); + } + let mut pr_trees = BTreeMap::new(); + for spec in &args.pr_trees { + let (number, dir) = spec + .split_once('=') + .and_then(|(n, d)| Some((n.parse::().ok()?, d))) + .filter(|(n, d)| *n > 0 && !d.is_empty()) + .ok_or_else(|| usage(format!("--pr-tree wants N=DIR, got {spec}")))?; + pr_trees.insert(number, PathBuf::from(dir)); + } + let trees = TreeSources { + commit_trees, + pr_trees, + cache: args.commit_cache.clone(), + http, + repo: EASYCONFIGS_REPO.into(), + }; let list_bytes = std::fs::read(&args.build_list) .with_context(|| format!("--build-list {}: not readable", args.build_list.display()))?; let list_text = String::from_utf8_lossy(&list_bytes).into_owned(); @@ -2341,6 +2408,25 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { args.build_list.display() ))); } + let overrides: Vec = lines + .iter() + .filter_map(|line| { + line.filter_deps_option().map(|option| { + format!( + "line {}: {}: {option} overrides dependency filters for one line; \ + pass --filter-deps or EASYBUILD_FILTER_DEPS for the whole list", + line.number, line.file + ) + }) + }) + .collect(); + if !overrides.is_empty() { + return Err(anyhow!( + "{} dependency-filter override(s); nothing written\n{}", + overrides.len(), + overrides.join("\n") + )); + } let (weights, weights_digest) = match &args.weights { Some(path) => { let bytes = std::fs::read(path) @@ -2368,10 +2454,7 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { } let (robot, _) = load_robot_trees(&args.easyconfigs, &[])?; - let mut universe = robot.base; - if !filter.is_empty() { - filter.apply(&mut universe); - } + let mut base = robot.base; let file_name = |c: &Candidate| -> String { c.easyconfig_path .rsplit('/') @@ -2379,50 +2462,211 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { .unwrap_or_default() .to_string() }; - let by_file: BTreeMap = - universe.iter().map(|c| (file_name(c), c)).collect(); - let by_module: BTreeMap = universe + // Identity merging can discard a named alias with a distinct dependency + // set. Explicit build-list filenames select their own robot candidate. + let mut selected = Vec::new(); + let mut selected_names: BTreeMap = BTreeMap::new(); + for line in lines.iter().filter(|line| !line.fetched) { + let candidate = base.iter().find(|c| file_name(c) == line.file).cloned(); + let candidate = candidate.or_else(|| { + file_on_robot(&args.easyconfigs, &line.file) + .and_then(|path| crate::eb_parse::parse_easyconfig_file(&path).ok()) + }); + if let Some(candidate) = candidate { + if let Some(other) = selected_names.insert(ModuleKey::of(&candidate), line.file.clone()) + { + if other != line.file { + return Err(usage(format!( + "{} and {} select different easyconfigs for the same module; nothing written", + other, line.file + ))); + } + } + selected.push(candidate); + } + } + base = merge_candidates_with_precedence(&[base, selected]); + let filtered = |mut universe: Vec| -> Vec { + if !filter.is_empty() { + filter.apply(&mut universe); + } + universe + }; + + // Universe 0 is the robot path. Each fetched source a tree was found for + // is one more: its easyconfigs first, then the robot path. + let mut sources: Vec = Vec::new(); + let mut overlays: Vec> = Vec::new(); + let mut tree_status: BTreeMap> = BTreeMap::new(); + let mut source_failure: BTreeMap = BTreeMap::new(); + let mut warnings: Vec = Vec::new(); + let wanted: BTreeSet<&LineSource> = lines .iter() - .map(|c| (module_name(&ModuleKey::of(c)), c)) + .filter(|l| l.fetched) + .filter_map(|l| l.source.as_ref()) + .filter(|s| !matches!(s, LineSource::Invalid(_))) + .collect(); + if !wanted.is_empty() && trees.any() { + // SHA-256 of the robot path's effective easyconfig of each name, so a + // tree's unchanged files are not parsed again. + let mut base_digests: BTreeMap = BTreeMap::new(); + for c in &base { + if let Ok(bytes) = std::fs::read(&c.easyconfig_path) { + base_digests.insert(file_name(c), sha256_hex(&bytes)); + } + } + let mut parsed: BTreeMap, String>> = BTreeMap::new(); + for source in wanted { + let dirs: Result, String> = match source { + LineSource::Commit(sha) => trees.commit_tree(sha).map(|d| vec![d]), + LineSource::Prs(prs) => prs.iter().map(|n| trees.pr_tree(*n)).collect(), + LineSource::Invalid(_) => unreachable!("invalid sources are not wanted"), + }; + // A cache miss that never reaches an HTTP status cannot fall back: + // the tree was requested and the download did not happen. An HTTP + // status still becomes an opaque line below. + if let Err(why) = &dirs { + if why.starts_with(crate::shard_trees::CACHE_DOWNLOAD_FAILED) { + return Err(anyhow!("{why}; nothing written")); + } + } + let layers = dirs.and_then(|dirs| { + let mut layer = Vec::new(); + for dir in dirs { + let tree = parsed.entry(dir.clone()).or_insert_with(|| { + let (candidates, skipped) = overlay_candidates(&dir, &base_digests)?; + for file in skipped { + warnings.push(format!("{}: does not parse: {file}", source.option())); + } + Ok(candidates) + }); + layer.extend(tree.clone()?); + } + Ok(layer) + }); + match layers { + Ok(layer) => { + tree_status.insert(source.option(), None); + sources.push(source.clone()); + overlays.push(layer); + } + Err(why) => { + tree_status.insert(source.option(), Some(why.clone())); + source_failure.insert(source.clone(), why); + } + } + } + } + let universe_of = |source: &LineSource| -> Option { + sources.iter().position(|s| s == source).map(|i| i + 1) + }; + // The own easyconfig of every fetched line, from its tree, with the + // source options that fetch it. A line that needs another line's module + // finds it installed by that line when it builds, whichever tree that + // module's easyconfig is in, so these sit under every universe at the + // lowest precedence and the two lines share a shard. + let mut fetched_roots: Vec = Vec::new(); + let mut root_options: BTreeMap> = BTreeMap::new(); + for line in &lines { + let Some(u) = line.source.as_ref().and_then(universe_of) else { + continue; + }; + if let Some(c) = overlays[u - 1].iter().find(|c| file_name(c) == line.file) { + root_options + .entry(c.easyconfig_path.clone()) + .or_insert_with(|| line.source_options.clone()); + fetched_roots.push(c.clone()); + } + } + let build_universe = |u: usize| -> Vec { + let mut layers = vec![fetched_roots.clone(), base.clone()]; + if u > 0 { + layers.push(overlays[u - 1].clone()); + } + filtered(merge_candidates_with_precedence(&layers)) + }; + let overlay_paths: Vec> = std::iter::once(BTreeSet::new()) + .chain(overlays.iter().map(|layer| { + layer + .iter() + .map(|c| c.easyconfig_path.clone()) + .collect::>() + })) .collect(); // Resolve every line before computing anything, so a bad list fails with // every bad line named. let mut problems = Vec::new(); - let mut starts: Vec> = Vec::new(); + let mut universe_for: Vec> = vec![None; lines.len()]; // A line eb-stack cannot resolve but EasyBuild can still build is planned // as an opaque line: its own group, no closure, the default weight. - let mut opaque: Vec> = Vec::new(); - for line in &lines { + let mut opaque: Vec> = vec![None; lines.len()]; + let mut opaque_reason: Vec> = vec![None; lines.len()]; + // A fetched line whose tree is missing falls back to the robot path when + // its easyconfig is there; this is why. + let mut fallback: Vec> = vec![None; lines.len()]; + for (i, line) in lines.iter().enumerate() { if !line.file.ends_with(".eb") { problems.push(format!( "line {}: {} is not an easyconfig filename", line.number, line.file )); - starts.push(None); - opaque.push(None); continue; } - match by_file.get(&line.file) { - Some(c) => { - starts.push(Some(ModuleKey::of(c))); - opaque.push(None); - } - None if line.fetched => { - starts.push(None); - opaque.push(Some("fetched by EasyBuild (--from-pr or --from-commit)")); - } - None if file_under(&args.easyconfigs, &line.file) => { - starts.push(None); - opaque.push(Some("on the robot path, not parsed by eb-stack")); + if !line.fetched { + universe_for[i] = Some(0); + continue; + } + match &line.source { + Some(LineSource::Invalid(why)) => fallback[i] = Some(why.clone()), + Some(source) => match universe_of(source) { + Some(u) => universe_for[i] = Some(u), + None => { + fallback[i] = Some(source_failure.get(source).cloned().unwrap_or_else(|| { + format!( + "no tree for {}: pass --commit-tree, --pr-tree, or --commit-cache", + source.option() + ) + })) + } + }, + None => fallback[i] = Some("no --from-pr or --from-commit value".into()), + } + if fallback[i].is_some() { + universe_for[i] = Some(0); + } + } + + // First pass, one universe at a time: each line's own module. + let used: BTreeSet = universe_for.iter().flatten().copied().collect(); + let mut starts: Vec> = vec![None; lines.len()]; + for &u in &used { + let universe = build_universe(u); + let by_file: BTreeMap = + universe.iter().map(|c| (file_name(c), c)).collect(); + for (i, line) in lines.iter().enumerate() { + if universe_for[i] != Some(u) { + continue; } - None => { - problems.push(format!( + match by_file.get(&line.file) { + Some(c) => starts[i] = Some(ModuleKey::of(c)), + None if line.fetched => { + opaque[i] = Some(FETCHED); + opaque_reason[i] = Some(match (&fallback[i], &line.source) { + (Some(why), _) => why.clone(), + (None, Some(source)) => format!( + "{} is in neither the tree of {} nor the robot path", + line.file, + source.option() + ), + (None, None) => "no tree".into(), + }); + } + None if file_under(&args.easyconfigs, &line.file) => opaque[i] = Some(UNPARSED), + None => problems.push(format!( "line {}: {}: no such easyconfig on the robot path", line.number, line.file - )); - starts.push(None); - opaque.push(None); + )), } } } @@ -2434,38 +2678,117 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { .collect(); let is_installed = |key: &ModuleKey| !forced.contains(key) && installed.contains(&module_name(key)); - let mut closures = Vec::new(); - let mut reached: BTreeSet = BTreeSet::new(); - for ((line, start), why) in lines.iter().zip(&starts).zip(&opaque) { - if why.is_some() { - closures.push(LineClosure { - root: line.file.clone(), - to_build: vec![line.file.clone()], + + // Second pass: closures, a dependency order per universe, and where each + // module's easyconfig comes from (the first line that reaches it wins). + struct Origin { + file: String, + path: String, + options: Vec, + } + let mut origin: BTreeMap = BTreeMap::new(); + let mut closures: Vec> = vec![None; lines.len()]; + let mut orders: Vec> = Vec::new(); + for &u in &used { + let universe = build_universe(u); + let by_key: BTreeMap = + universe.iter().map(|c| (ModuleKey::of(c), c)).collect(); + let mut universe_starts = Vec::new(); + for (i, line) in lines.iter().enumerate() { + if universe_for[i] != Some(u) || opaque[i].is_some() { + continue; + } + let Some(start) = &starts[i] else { continue }; + let closure = closure_against_installed( + &universe, + std::slice::from_ref(start), + Choice::Newest, + &is_installed, + ); + if !closure.unresolved.is_empty() { + let needs: Vec = closure + .unresolved + .iter() + .map(|r| format!("{} needs {}", module_name(&r.from), r.requirement)) + .collect(); + if line.fetched { + // EasyBuild may still find it; plan the line apart. + opaque[i] = Some(FETCHED); + let tree = match &line.source { + Some(source) if u > 0 => format!("the tree of {} or ", source.option()), + _ => String::new(), + }; + opaque_reason[i] = Some(format!( + "{}, which {tree}the robot path does not provide", + needs.join("; ") + )); + continue; + } + for need in needs { + problems.push(format!("line {}: {}: {need}", line.number, line.file)); + } + continue; + } + if let Some(why) = &fallback[i] { + warnings.push(format!( + "line {}: {}: closure over the robot path only: {why}", + line.number, line.file + )); + } + for key in closure.to_build.iter().chain(&closure.installed) { + let Some(c) = by_key.get(key) else { continue }; + let options = if overlay_paths[u].contains(&c.easyconfig_path) { + line.source_options.clone() + } else { + root_options + .get(&c.easyconfig_path) + .cloned() + .unwrap_or_default() + }; + let module = module_name(key); + match origin.get(&module) { + Some(known) if known.path != c.easyconfig_path => { + let differs = std::fs::read(&known.path).ok() + != std::fs::read(&c.easyconfig_path).ok(); + if differs && closure.to_build.contains(key) { + warnings.push(format!( + "line {}: {}: {module} resolves to {} here and to {} {} for an earlier line; the earlier one is planned", + line.number, + line.file, + c.easyconfig_path, + known.path, + known.options.join(" ") + )); + } + } + Some(_) => {} + None => { + origin.insert( + module, + Origin { + file: file_name(c), + path: c.easyconfig_path.clone(), + options, + }, + ); + } + } + } + universe_starts.push(start.clone()); + closures[i] = Some(LineClosure { + root: module_name(start), + to_build: closure.to_build.iter().map(module_name).collect(), }); - continue; } - let Some(start) = start else { continue }; - let closure = closure_against_installed( - &universe, - std::slice::from_ref(start), - Choice::Newest, - &is_installed, + // One closure over every line of the universe gives a dependency + // order for the prelude. + orders.push( + closure_against_installed(&universe, &universe_starts, Choice::Newest, &is_installed) + .to_build + .iter() + .map(module_name) + .collect(), ); - for requirement in &closure.unresolved { - problems.push(format!( - "line {}: {}: {} needs {}", - line.number, - line.file, - module_name(&requirement.from), - requirement.requirement - )); - } - reached.extend(closure.to_build.iter().cloned()); - reached.extend(closure.installed.iter().cloned()); - closures.push(LineClosure { - root: module_name(start), - to_build: closure.to_build.iter().map(module_name).collect(), - }); } if !problems.is_empty() { return Err(anyhow!( @@ -2474,21 +2797,47 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { problems.join("\n") )); } + let closures: Vec = closures + .into_iter() + .zip(&lines) + .map(|(closure, line)| { + closure.unwrap_or_else(|| LineClosure { + root: line.file.clone(), + to_build: vec![line.file.clone()], + }) + }) + .collect(); + let mut opaque_json = Vec::new(); + for (i, line) in lines.iter().enumerate() { + let Some(why) = opaque[i] else { continue }; + let reason = opaque_reason[i].as_deref(); + warnings.push(format!( + "line {}: {}: opaque, planned without a closure: {}", + line.number, + line.file, + reason.unwrap_or(why) + )); + opaque_json.push(json!({ + "line": line.number, + "easyconfig": line.file, + "opaque": why, + "reason": reason, + })); + } - // One closure over every line gives a dependency order for the prelude. - let all_starts: Vec = starts.iter().flatten().cloned().collect(); - let order: Vec = - closure_against_installed(&universe, &all_starts, Choice::Newest, &is_installed) - .to_build - .iter() - .map(module_name) - .collect(); + let order = merge_orders(&orders); let easyconfig_of = |module: &str| -> String { - by_module + origin .get(module) - .map(|c| file_name(c)) + .map(|o| o.file.clone()) .unwrap_or_else(|| module.to_string()) }; + let prelude_line = |module: &str| -> String { + match origin.get(module) { + Some(o) if !o.options.is_empty() => format!("{} {}", o.file, o.options.join(" ")), + _ => easyconfig_of(module), + } + }; let weight = |module: &str| -> u64 { weights .get(module) @@ -2500,14 +2849,16 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { // The digest covers what decides the plan: the list, the weights, the // installed modules, and every easyconfig a closure reached, by file name - // and content. + // (and source, for one a fetched tree provides) and content. let mut easyconfig_digests: BTreeMap = BTreeMap::new(); - for key in &reached { - if let Some(c) = by_module.get(&module_name(key)) { - let content = std::fs::read(&c.easyconfig_path) - .with_context(|| format!("read {}", c.easyconfig_path))?; - easyconfig_digests.insert(file_name(c), sha256_hex(&content)); - } + for o in origin.values() { + let content = std::fs::read(&o.path).with_context(|| format!("read {}", o.path))?; + let key = if o.options.is_empty() { + o.file.clone() + } else { + format!("{} {}", o.file, o.options.join(" ")) + }; + easyconfig_digests.insert(key, sha256_hex(&content)); } let mut digest_input = format!( "shards={}\ndefault_weight={}\nbuild_list={}\n", @@ -2527,6 +2878,12 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { for (file, digest) in &easyconfig_digests { digest_input.push_str(&format!("easyconfig={file} {digest}\n")); } + for (source, status) in &tree_status { + digest_input.push_str(&format!( + "tree={source} {}\n", + status.as_deref().unwrap_or("found") + )); + } let inputs_digest = sha256_hex(digest_input.as_bytes()); let width = planned.shards.len().to_string().len().max(2); @@ -2536,7 +2893,7 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { planned .prelude .iter() - .map(|m| format!("{}\n", easyconfig_of(m))) + .map(|m| format!("{}\n", prelude_line(m))) .collect(), )]; let mut shard_json = Vec::new(); @@ -2566,10 +2923,12 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { "text": line.text, "easyconfig": line.file, "module": closures[i].root, + "source": line.source.as_ref().map(LineSource::option), "group": planned.group_of[i] + 1, "shard": shard_name(planned.shard_of[i]), "closure": closures[i].to_build, "opaque": opaque[i], + "opaque_reason": opaque_reason[i], }) }) .collect(); @@ -2583,6 +2942,7 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { "installed_modules": installed.len(), "filter_deps": filter_specs, "easyconfigs": easyconfig_digests, + "trees": tree_status, }, "shards_requested": args.shards, "default_weight": args.default_weight, @@ -2592,11 +2952,18 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { "modules": planned .prelude .iter() - .map(|m| json!({"module": m, "easyconfig": easyconfig_of(m), "weight": weight(m)})) + .map(|m| json!({ + "module": m, + "easyconfig": easyconfig_of(m), + "options": origin.get(m.as_str()).map(|o| o.options.clone()).unwrap_or_default(), + "weight": weight(m), + })) .collect::>(), }, "shards": shard_json, "lines": line_json, + "opaque": opaque_json, + "warnings": warnings, }); std::fs::create_dir_all(&args.out_dir) @@ -2623,6 +2990,8 @@ pub fn run_stack_shard(args: &StackShardRequest) -> Result { shards: planned.shards.len(), prelude: planned.prelude.len(), lines: lines.len(), + opaque: opaque_json.len(), + warnings, }) } @@ -2811,4 +3180,469 @@ mod tests { && descendant == "720f484b605f18887a48eef448d0084e2b76902d") } } + + mod shard_fetched_lines { + use super::super::*; + use crate::shard_trees::tests::{archive, MapHttp}; + use serde_json::Value; + use std::collections::BTreeMap; + use std::path::Path; + + const SHA: &str = "0d6b80c34f3553774e93162cf89088e2423ecf89"; + + /// `///--GCCcore-15.2.0.eb`, its + /// dependencies at the same toolchain. Returns the path below `root`. + fn ec(root: &Path, name: &str, version: &str, deps: &[(&str, &str)]) -> String { + let letter = name[..1].to_ascii_lowercase(); + let relative = format!("{letter}/{name}/{name}-{version}-GCCcore-15.2.0.eb"); + let deps: Vec = deps + .iter() + .map(|(n, v)| format!("('{n}', '{v}')")) + .collect(); + let text = format!( + "easyblock = 'ConfigureMake'\nname = '{name}'\nversion = '{version}'\nhomepage = 'https://example.org/'\ndescription = 'stack shard test'\ntoolchain = {{'name': 'GCCcore', 'version': '15.2.0'}}\nsource_urls = ['https://example.org/']\nsources = [SOURCE_TAR_GZ]\ndependencies = [{}]\nmoduleclass = 'lib'\n", + deps.join(", ") + ); + let path = root.join(&relative); + std::fs::create_dir_all(path.parent().unwrap()).unwrap(); + std::fs::write(&path, &text).unwrap(); + relative + } + + fn request(root: &Path, list: &str) -> StackShardRequest { + std::fs::write(root.join("build.list"), list).unwrap(); + StackShardRequest { + build_list: root.join("build.list"), + easyconfigs: vec![root.join("robot")], + commit_trees: Vec::new(), + pr_trees: Vec::new(), + commit_cache: None, + installed: Vec::new(), + installed_modules_dirs: Vec::new(), + shards: 4, + weights: None, + default_weight: 1, + filter_deps: vec!["NoSuchFilter".into()], + generated_at: Some("2026-11-04T00:00:00Z".into()), + out_dir: root.join("out"), + } + } + + #[test] + fn explicit_recipe_alias_selects_its_own_dependency_closure() { + for alias in [false, true] { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + let robot = root.join("robot"); + ec(&robot, "OldLib", "1.0", &[]); + ec(&robot, "NewLib", "1.0", &[]); + let canonical = ec(&robot, "Alpha", "1.0", &[("OldLib", "1.0")]); + let alternate = "a/Alpha/Alpha-1.0-GCCcore-15.2.0-site.eb"; + let text = read(robot.join(&canonical)).replace("OldLib", "NewLib"); + std::fs::write(robot.join(alternate), text).unwrap(); + let (selected, required, excluded) = if alias { + (alternate, "NewLib", "OldLib") + } else { + (canonical.as_str(), "OldLib", "NewLib") + }; + ec(&robot, "Beta", "1.0", &[(required, "1.0")]); + let file = Path::new(selected).file_name().unwrap().to_str().unwrap(); + let args = request(root, &format!("{file}\nBeta-1.0-GCCcore-15.2.0.eb\n")); + let run = run_stack_shard_with(&args, &MapHttp::default()).unwrap(); + assert_eq!(run.opaque, 0, "{:?}", run.warnings); + let plan = plan_of(root); + let closure = line(&plan, file)["closure"].as_array().unwrap().clone(); + assert!( + closure.contains(&Value::String(format!("{required}/1.0-GCCcore-15.2.0"))), + "{plan}" + ); + assert!( + !closure.contains(&Value::String(format!("{excluded}/1.0-GCCcore-15.2.0"))), + "{plan}" + ); + assert_eq!( + read(root.join("out/prelude.list")).trim(), + format!("{required}-1.0-GCCcore-15.2.0.eb") + ); + assert_no_module_twice(&plan); + } + } + + fn read(path: PathBuf) -> String { + std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("{}: {e}", path.display())) + } + + fn plan_of(root: &Path) -> Value { + serde_json::from_str(&read(root.join("out/plan.json"))).unwrap() + } + + fn line<'a>(plan: &'a Value, file: &str) -> &'a Value { + plan["lines"] + .as_array() + .unwrap() + .iter() + .find(|l| l["easyconfig"] == file) + .unwrap_or_else(|| panic!("{file} not in {plan}")) + } + + /// Every shard's modules, so a test can check no module builds twice. + fn shard_modules(plan: &Value) -> Vec> { + plan["shards"] + .as_array() + .unwrap() + .iter() + .map(|s| { + s["modules"] + .as_array() + .unwrap() + .iter() + .map(|m| m.as_str().unwrap().to_string()) + .collect() + }) + .collect() + } + + fn assert_no_module_twice(plan: &Value) { + let prelude: Vec = plan["prelude"]["modules"] + .as_array() + .unwrap() + .iter() + .map(|m| m["module"].as_str().unwrap().to_string()) + .collect(); + let mut seen: BTreeMap = BTreeMap::new(); + for module in shard_modules(plan).into_iter().flatten().chain(prelude) { + *seen.entry(module).or_default() += 1; + } + let twice: Vec<&String> = seen + .iter() + .filter(|(_, n)| **n > 1) + .map(|(m, _)| m) + .collect(); + assert!(twice.is_empty(), "built by two jobs: {twice:?} in {plan}"); + } + + /// A fetched line's dependency that a robot-path line also needs goes + /// to the prelude. The commit is downloaded into the cache; a module + /// only the commit has, needed by two fetched lines, goes to the + /// prelude with the line's source options. + #[test] + fn a_fetched_line_shares_its_dependencies_through_the_prelude() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + let robot = root.join("robot"); + ec(&robot, "MathLib", "1.0", &[]); + ec(&robot, "Alpha", "1.0", &[("MathLib", "1.0")]); + let top = format!("easybuild-easyconfigs-{SHA}"); + let in_commit = |rel: &str| format!("easybuild/easyconfigs/{rel}"); + let staging = root.join("staging"); + let files: Vec<(String, String)> = [ + ec( + &staging, + "Fetched", + "1.0", + &[("MathLib", "1.0"), ("NewLib", "2.0")], + ), + ec(&staging, "Other", "1.0", &[("NewLib", "2.0")]), + ec(&staging, "NewLib", "2.0", &[]), + ec(&staging, "MathLib", "1.0", &[]), + ] + .iter() + .map(|rel| (in_commit(rel), read(staging.join(rel)))) + .collect(); + let borrowed: Vec<(&str, &str)> = files + .iter() + .map(|(p, t)| (p.as_str(), t.as_str())) + .collect(); + let mut http = MapHttp::default(); + http.pages.insert( + format!( + "https://codeload.github.com/easybuilders/easybuild-easyconfigs/tar.gz/{SHA}" + ), + archive(&top, &borrowed), + ); + let mut args = request( + root, + &format!( + "Alpha-1.0-GCCcore-15.2.0.eb\n\ + Fetched-1.0-GCCcore-15.2.0.eb --from-commit={SHA} --include-systems=snel-zen2\n\ + Other-1.0-GCCcore-15.2.0.eb --from-commit={SHA}\n" + ), + ); + args.commit_cache = Some(root.join("cache")); + let run = run_stack_shard_with(&args, &http).unwrap(); + assert_eq!(run.summary(), "shards=3 prelude=2 lines=3"); + assert!(run.warnings.is_empty(), "{:?}", run.warnings); + + let plan = plan_of(root); + let fetched = line(&plan, "Fetched-1.0-GCCcore-15.2.0.eb"); + assert!(fetched["opaque"].is_null(), "{fetched}"); + assert_eq!(fetched["source"], format!("--from-commit={SHA}")); + assert_eq!(fetched["module"], "Fetched/1.0-GCCcore-15.2.0"); + assert_eq!( + fetched["closure"].as_array().unwrap().len(), + 3, + "MathLib, NewLib, Fetched: {fetched}" + ); + let alpha = line(&plan, "Alpha-1.0-GCCcore-15.2.0.eb"); + assert_ne!(alpha["shard"], fetched["shard"]); + assert_no_module_twice(&plan); + // MathLib is the robot path's (the commit's copy is the same + // file), so it builds without a source option; NewLib only the + // commit has. + let prelude = read(root.join("out/prelude.list")); + let mut prelude: Vec<&str> = prelude.lines().collect(); + prelude.sort_unstable(); + assert_eq!( + prelude, + [ + "MathLib-1.0-GCCcore-15.2.0.eb".to_string(), + format!("NewLib-2.0-GCCcore-15.2.0.eb --from-commit={SHA}") + ] + ); + assert_eq!( + plan["inputs"]["trees"], + serde_json::json!({ format!("--from-commit={SHA}"): null }) + ); + assert!(plan["inputs"]["easyconfigs"] + .get(format!("NewLib-2.0-GCCcore-15.2.0.eb --from-commit={SHA}")) + .is_some()); + // One download for both lines. + assert_eq!(http.asked.borrow().len(), 1); + } + + /// petsc4py from a commit needs PETSc, which is a robot-path line of + /// its own: the two build in one shard, PETSc first, and no other + /// shard builds PETSc. Hypre, which PETSc and another line need, is + /// in the prelude. + #[test] + fn petsc4py_from_a_commit_builds_after_the_petsc_line_in_its_shard() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + let robot = root.join("robot"); + ec(&robot, "Hypre", "2.33.0", &[]); + ec(&robot, "PETSc", "3.25.0", &[("Hypre", "2.33.0")]); + ec(&robot, "MUMPS", "5.8.1", &[("Hypre", "2.33.0")]); + let tree = root.join("commit-tree"); + ec(&tree, "petsc4py", "3.25.0", &[("PETSc", "3.25.0")]); + let mut args = request( + root, + &format!( + "PETSc-3.25.0-GCCcore-15.2.0.eb --include-systems=snel-zen2\n\ + MUMPS-5.8.1-GCCcore-15.2.0.eb\n\ + petsc4py-3.25.0-GCCcore-15.2.0.eb --from-commit={SHA} --include-systems=snel-zen2\n" + ), + ); + args.commit_trees = vec![format!("{}={}", &SHA[..10], tree.display())]; + let run = run_stack_shard_with(&args, &MapHttp::default()).unwrap(); + assert_eq!(run.opaque, 0, "{:?}", run.warnings); + + let plan = plan_of(root); + let petsc = line(&plan, "PETSc-3.25.0-GCCcore-15.2.0.eb"); + let petsc4py = line(&plan, "petsc4py-3.25.0-GCCcore-15.2.0.eb"); + assert!(petsc4py["opaque"].is_null(), "{petsc4py}"); + assert!(petsc4py["closure"] + .as_array() + .unwrap() + .contains(&Value::from("PETSc/3.25.0-GCCcore-15.2.0"))); + assert_eq!(petsc["group"], petsc4py["group"]); + assert_eq!(petsc["shard"], petsc4py["shard"]); + let shard_file = format!("out/{}.list", petsc["shard"].as_str().unwrap()); + assert_eq!( + read(root.join(shard_file)), + format!( + "PETSc-3.25.0-GCCcore-15.2.0.eb --include-systems=snel-zen2\n\ + petsc4py-3.25.0-GCCcore-15.2.0.eb --from-commit={SHA} --include-systems=snel-zen2\n" + ) + ); + let with_petsc = shard_modules(&plan) + .iter() + .filter(|m| m.contains(&"PETSc/3.25.0-GCCcore-15.2.0".to_string())) + .count(); + assert_eq!(with_petsc, 1); + assert_eq!( + read(root.join("out/prelude.list")), + "Hypre-2.33.0-GCCcore-15.2.0.eb\n" + ); + assert_no_module_twice(&plan); + } + + /// A fetched line that cannot be resolved stays opaque, with the + /// reason in the plan and in the warnings: its easyconfig is in + /// neither tree, its tree needs something nothing provides, or its + /// commit cannot be downloaded. + #[test] + fn an_unresolvable_fetched_line_stays_opaque_and_is_reported() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + let robot = root.join("robot"); + ec(&robot, "MathLib", "1.0", &[]); + ec(&robot, "Alpha", "1.0", &[("MathLib", "1.0")]); + let tree = root.join("commit-tree"); + ec(&tree, "Needy", "1.0", &[("Nowhere", "9.9")]); + let missing = "1111111111111111111111111111111111111111"; + let mut args = request( + root, + &format!( + "Alpha-1.0-GCCcore-15.2.0.eb\n\ + Absent-1.0-GCCcore-15.2.0.eb --from-commit={SHA}\n\ + Needy-1.0-GCCcore-15.2.0.eb --from-commit={SHA}\n\ + Gone-1.0-GCCcore-15.2.0.eb --from-commit={missing}\n" + ), + ); + args.commit_trees = vec![format!("{SHA}={}", tree.display())]; + args.commit_cache = Some(root.join("cache")); + let run = run_stack_shard_with(&args, &MapHttp::default()).unwrap(); + assert_eq!(run.opaque, 3); + assert!(run.summary().ends_with(" opaque=3"), "{}", run.summary()); + + let plan = plan_of(root); + let absent = line(&plan, "Absent-1.0-GCCcore-15.2.0.eb"); + assert_eq!( + absent["opaque"], + "fetched by EasyBuild (--from-pr or --from-commit)" + ); + assert!( + absent["opaque_reason"] + .as_str() + .unwrap() + .contains("is in neither the tree"), + "{absent}" + ); + assert_eq!( + absent["closure"], + serde_json::json!(["Absent-1.0-GCCcore-15.2.0.eb"]) + ); + let needy = line(&plan, "Needy-1.0-GCCcore-15.2.0.eb"); + assert!( + needy["opaque_reason"] + .as_str() + .unwrap() + .contains("needs Nowhere"), + "{needy}" + ); + let gone = line(&plan, "Gone-1.0-GCCcore-15.2.0.eb"); + assert!( + gone["opaque_reason"].as_str().unwrap().contains("HTTP 404"), + "{gone}" + ); + assert!(plan["inputs"]["trees"][format!("--from-commit={missing}")] + .as_str() + .unwrap() + .contains("HTTP 404")); + assert_eq!(plan["opaque"].as_array().unwrap().len(), 3); + for file in ["Absent", "Needy", "Gone"] { + assert!( + run.warnings + .iter() + .any(|w| w.contains(&format!("{file}-1.0-GCCcore-15.2.0.eb: opaque"))), + "{file}: {:?}", + run.warnings + ); + } + // Each opaque line is still written to a shard, unchanged. + let all: String = (1..=run.shards) + .map(|s| read(root.join(format!("out/shard-{s:02}.list")))) + .collect(); + assert!(all.contains(&format!( + "Gone-1.0-GCCcore-15.2.0.eb --from-commit={missing}\n" + ))); + assert!(!root.join("cache").join(missing).exists()); + } + + /// texlive on the robot path needs libGLU, which only a fetched + /// line's commit provides. The libGLU line builds it first in the + /// same shard, so texlive resolves and the two lines are one group. + #[test] + fn a_robot_line_needing_a_fetched_line_shares_its_shard() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + let robot = root.join("robot"); + ec(&robot, "Mesa", "26.1.1", &[]); + ec(&robot, "texlive", "20250308", &[("libGLU", "9.0.3")]); + ec(&robot, "Other", "1.0", &[("Mesa", "26.1.1")]); + let tree = root.join("commit-tree"); + ec(&tree, "libGLU", "9.0.3", &[("Mesa", "26.1.1")]); + let mut args = request( + root, + &format!( + "libGLU-9.0.3-GCCcore-15.2.0.eb --from-commit={SHA}\n\ + texlive-20250308-GCCcore-15.2.0.eb\n\ + Other-1.0-GCCcore-15.2.0.eb\n" + ), + ); + args.commit_trees = vec![format!("{SHA}={}", tree.display())]; + let run = run_stack_shard_with(&args, &MapHttp::default()).unwrap(); + assert_eq!(run.opaque, 0, "{:?}", run.warnings); + let plan = plan_of(root); + let glu = line(&plan, "libGLU-9.0.3-GCCcore-15.2.0.eb"); + let tex = line(&plan, "texlive-20250308-GCCcore-15.2.0.eb"); + assert_eq!(glu["group"], tex["group"]); + let shard_file = format!("out/{}.list", tex["shard"].as_str().unwrap()); + assert_eq!( + read(root.join(shard_file)), + format!( + "libGLU-9.0.3-GCCcore-15.2.0.eb --from-commit={SHA}\n\ + texlive-20250308-GCCcore-15.2.0.eb\n" + ) + ); + assert_eq!( + read(root.join("out/prelude.list")), + "Mesa-26.1.1-GCCcore-15.2.0.eb\n" + ); + assert_no_module_twice(&plan); + } + + /// A connect error is not an HTTP status. The plan is not written. + struct OfflineHttp; + + impl crate::pr_overlap::HttpGet for OfflineHttp { + fn get(&self, _url: &str) -> Result { + Err("tcp connect error: Network is unreachable".into()) + } + } + + #[test] + fn a_cache_miss_with_no_network_fails_and_writes_nothing() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + ec(&root.join("robot"), "MathLib", "1.0", &[]); + let mut args = request( + root, + &format!("Absent-1.0-GCCcore-15.2.0.eb --from-commit={SHA}\n"), + ); + args.commit_cache = Some(root.join("cache")); + let error = run_stack_shard_with(&args, &OfflineHttp).unwrap_err(); + let message = format!("{error:#}"); + assert!( + message.starts_with(crate::shard_trees::CACHE_DOWNLOAD_FAILED), + "{message}" + ); + assert!(message.contains("Network is unreachable"), "{message}"); + assert!(message.contains("nothing written"), "{message}"); + assert!(!root.join("out").exists(), "nothing is written"); + assert!(!root.join("cache").join(SHA).exists()); + } + + #[test] + fn a_bad_tree_spec_is_a_usage_error() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + ec(&root.join("robot"), "MathLib", "1.0", &[]); + let mut args = request(root, "MathLib-1.0-GCCcore-15.2.0.eb\n"); + args.commit_trees = vec!["main=/tmp/x".into()]; + let error = run_stack_shard_with(&args, &MapHttp::default()).unwrap_err(); + assert!( + error.to_string().contains("--commit-tree wants COMMIT=DIR"), + "{error}" + ); + args.commit_trees.clear(); + args.pr_trees = vec!["x=/tmp/x".into()]; + let error = run_stack_shard_with(&args, &MapHttp::default()).unwrap_err(); + assert!( + error.to_string().contains("--pr-tree wants N=DIR"), + "{error}" + ); + assert!(!root.join("out").exists()); + } + } } diff --git a/src/workspace_fs.rs b/src/workspace_fs.rs index d231e5d1..b9b3f61d 100644 --- a/src/workspace_fs.rs +++ b/src/workspace_fs.rs @@ -893,6 +893,13 @@ pub fn resolve_inside(root: &Path, requested: &Path) -> Result<(PathBuf, PathBuf if requested.as_os_str().is_empty() { return Err("path is empty".into()); } + let supplied_root = if root.is_absolute() { + root.to_path_buf() + } else { + std::env::current_dir() + .map_err(|error| format!("current directory: {error}"))? + .join(root) + }; let root = root .canonicalize() .map_err(|error| format!("workspace root {}: {error}", root.display()))?; @@ -903,7 +910,10 @@ pub fn resolve_inside(root: &Path, requested: &Path) -> Result<(PathBuf, PathBuf )); } let joined = if requested.is_absolute() { - requested.to_path_buf() + match requested.strip_prefix(&supplied_root) { + Ok(relative) => root.join(relative), + Err(_) => requested.to_path_buf(), + } } else { root.join(requested) }; @@ -1124,6 +1134,34 @@ mod tests { ); } + #[cfg(unix)] + #[test] + fn absolute_paths_resolve_through_a_symlinked_workspace_root() { + let temp = tempfile::tempdir().expect("tempdir"); + let real = temp.path().join("real"); + let alias = temp.path().join("alias"); + fs::create_dir(&real).expect("workspace"); + std::os::unix::fs::symlink(&real, &alias).expect("root symlink"); + let requested = alias.join("inside.eb"); + write_workspace(&alias, &requested, "ok\n").expect("write through root alias"); + assert_eq!( + read_workspace(&alias, &requested) + .expect("read alias") + .content, + "ok\n" + ); + let canonical = real.canonicalize().expect("canonical root"); + assert_eq!( + resolve_inside(&alias, &requested).expect("resolve alias"), + (canonical.clone(), canonical.join("inside.eb")) + ); + let outside = alias.join("..").join("outside.eb"); + assert!(resolve_inside(&alias, &outside).is_err()); + let leak = alias.join("leak"); + std::os::unix::fs::symlink(temp.path(), &leak).expect("escaping symlink"); + assert!(resolve_inside(&alias, &leak.join("outside.eb")).is_err()); + } + #[cfg(unix)] #[test] fn rejects_symlinks_that_leave_the_workspace() { diff --git a/tests/bundle_component_checksums.rs b/tests/bundle_component_checksums.rs new file mode 100644 index 00000000..b5291a69 --- /dev/null +++ b/tests/bundle_component_checksums.rs @@ -0,0 +1,144 @@ +use eb_stack::{packaging_gate, resolve_easyconfig_str}; + +fn recipe(body: &str) -> String { + format!( + "easyblock = 'Bundle'\nname = 'RenderBundle'\nversion = '1.0'\n\ + toolchain = {{'name': 'GCC', 'version': '15.2.0'}}\nmoduleclass = 'lib'\n{body}" + ) +} + +#[test] +fn complete_component_archives_pass_without_inventing_root_checksums() { + let source = recipe(&format!( + "components = [('Renderer', '1.0', {{'sources': ['render.tar.xz'], \ + 'checksums': ['{}']}}), ('Data', '2.0', {{'sources': ['data.tar.gz'], \ + 'checksums': ['{}']}})]\n", + "a".repeat(64), + "b".repeat(64) + )); + let resolved = resolve_easyconfig_str(&source).unwrap(); + assert!(resolved.checksums.is_empty()); + assert!(resolved.component_checksums_complete); + packaging_gate(&resolved, &[]).expect("all component sources carry checksums"); +} + +#[test] +fn inherited_component_sources_and_checksums_pass() { + let source = recipe(&format!( + "default_component_specs = {{'sources': [SOURCE_TAR_GZ], 'checksums': ['{}']}}\n\ + components = [('Renderer', '1.0'), ('Data', '2.0')]\n", + "c".repeat(64) + )); + let resolved = resolve_easyconfig_str(&source).unwrap(); + assert!(resolved.component_checksums_complete); + packaging_gate(&resolved, &[]).expect("each component inherits its source checksum"); +} + +fn rejects(body: &str) { + let resolved = resolve_easyconfig_str(&recipe(body)).unwrap(); + assert!( + !resolved.component_checksums_complete, + "component checksums should be incomplete: {body}" + ); + let errs = packaging_gate(&resolved, &[]).expect_err(body); + assert!( + errs.iter().any(|err| err == "missing checksums"), + "{body}: {errs:?}" + ); +} + +#[test] +fn one_unchecked_component_refuses_the_bundle() { + rejects(&format!( + "components = [('Renderer', '1.0', {{'sources': ['render.tar.gz'], \ + 'checksums': ['{}']}}), ('Data', '2.0', {{'sources': ['data.tar.gz']}})]\n", + "a".repeat(64) + )); +} + +#[test] +fn empty_component_checksum_is_not_coverage() { + rejects( + "components = [('Renderer', '1.0', {'sources': ['render.tar.gz'], 'checksums': ['']})]\n", + ); +} + +#[test] +fn a_patch_needs_its_own_checksum_entry() { + rejects(&format!( + "components = [('Renderer', '1.0', {{'sources': ['render.tar.gz'], \ + 'patches': ['render.patch'], 'checksums': ['{}']}})]\n", + "a".repeat(64) + )); +} + +#[test] +fn a_postinstall_patch_needs_its_own_checksum_entry() { + rejects(&format!( + "components = [('Renderer', '1.0', {{'sources': ['render.tar.gz'], \ + 'postinstallpatches': ['render-post.patch'], 'checksums': ['{}']}})]\n", + "a".repeat(64) + )); +} + +#[test] +fn an_alternate_postinstall_patch_name_needs_its_own_checksum_entry() { + rejects(&format!( + "components = [('Renderer', '1.0', {{'sources': ['render.tar.gz'], \ + 'post_install_patches': ['render-post.patch'], 'checksums': ['{}']}})]\n", + "a".repeat(64) + )); +} + +#[test] +fn postinstall_patches_pass_when_their_checksum_is_present() { + let source = recipe(&format!( + "components = [('Renderer', '1.0', {{'sources': ['render.tar.gz'], \ + 'patches': ['render.patch'], 'postinstallpatches': ['render-post.patch'], \ + 'checksums': ['{}', '{}', '{}']}})]\n", + "a".repeat(64), + "d".repeat(64), + "e".repeat(64) + )); + let resolved = resolve_easyconfig_str(&source).unwrap(); + assert!(resolved.checksums.is_empty()); + assert!(resolved.component_checksums_complete); + packaging_gate(&resolved, &[]) + .expect("source, patch, and postinstall patch each have a checksum"); +} + +#[test] +fn malformed_component_cannot_disappear_from_coverage() { + rejects(&format!( + "components = [('Renderer', '1.0', {{'sources': ['render.tar.gz'], \ + 'checksums': ['{}']}}), 'unparsed-component']\n", + "a".repeat(64) + )); +} + +#[test] +fn component_hashes_cannot_cover_an_unchecked_root_archive() { + let source = recipe(&format!( + "sources = ['bundle.tar.gz']\ncomponents = [('Renderer', '1.0', \ + {{'sources': ['render.tar.gz'], 'checksums': ['{}']}})]\n", + "a".repeat(64) + )); + let resolved = resolve_easyconfig_str(&source).unwrap(); + assert!(resolved.component_checksums_complete); + assert_eq!(resolved.sources_count, 1); + assert!(resolved.checksums.is_empty()); + let errs = packaging_gate(&resolved, &[]).expect_err("unchecked root archive"); + assert!( + errs.iter().any(|err| err == "missing checksums"), + "{errs:?}" + ); +} + +#[test] +fn a_skipped_component_mutation_refuses_partial_coverage() { + rejects(&format!( + "components = [('Renderer', '1.0', {{'sources': ['render.tar.gz'], \ + 'checksums': ['{}']}})]\ncomponents += unsupported_components()\n", + "a".repeat(64) + )); +} diff --git a/tests/cli_reference.rs b/tests/cli_reference.rs index 99cdf743..a2fc38d6 100644 --- a/tests/cli_reference.rs +++ b/tests/cli_reference.rs @@ -150,6 +150,7 @@ fn section_for(args: &[String]) -> &'static str { Some("target") => "Target commands", Some("campaign") => "Campaign commands", Some("repro") => "repro summary", + Some("ledger") => "Ledger commands", Some("pr-overlap") => "pr-overlap", Some("rebuild-set") => "rebuild-set", Some("component-reach") => "component-reach", @@ -204,6 +205,7 @@ fn the_cli_reference_names_every_help_flag() { "target", "campaign", "repro", + "ledger", "mcp", "easyblock-reach", "pr-overlap", diff --git a/tests/ledger.rs b/tests/ledger.rs new file mode 100644 index 00000000..e3baa5f0 --- /dev/null +++ b/tests/ledger.rs @@ -0,0 +1,584 @@ +//! `ledger append`, `ledger head`, `ledger show` and `ledger verify` on the +//! binary, and the same writer over MCP. + +use eb_stack::ledger; +use ed25519_dalek::pkcs8::spki::der::pem::LineEnding; +use ed25519_dalek::pkcs8::{EncodePrivateKey, EncodePublicKey}; +use ed25519_dalek::SigningKey; +use serde_json::{json, Value}; +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +struct Fixture { + _temp: tempfile::TempDir, + log: PathBuf, + key: PathBuf, + public: PathBuf, + work: PathBuf, +} + +/// A log directory, and a key directory that is not inside it. +fn fixture() -> Fixture { + let temp = tempfile::tempdir().expect("tempdir"); + let ledger_dir = temp.path().join("ledger"); + let keys = temp.path().join("keys"); + let work = temp.path().join("work"); + for dir in [&ledger_dir, &keys, &work] { + std::fs::create_dir_all(dir).expect("mkdir"); + } + let signing = SigningKey::from_bytes(&[11u8; 32]); + let key = keys.join("ledger.pem"); + std::fs::write( + &key, + signing + .to_pkcs8_pem(LineEnding::LF) + .expect("pkcs8 pem") + .as_bytes(), + ) + .expect("write key"); + let public = work.join("ledger.pub"); + std::fs::write( + &public, + signing + .verifying_key() + .to_public_key_pem(LineEnding::LF) + .expect("spki pem"), + ) + .expect("write public key"); + Fixture { + log: ledger_dir.join("2026.06.jsonl"), + key, + public, + work, + _temp: temp, + } +} + +fn record(module: &str, actor: &str) -> Value { + json!({ + "ts": "2026-11-03T10:00:00Z", + "actor": actor, + "stack": "2026.06", + "target": "x86_64/amd/zen4", + "module": module, + "location": format!( + "/cvmfs/software.eessi.io/host_injections/2026.06/software/linux/x86_64/amd/zen4/software/{module}" + ), + "action": "install", + "reason": {"text": "first install", "refs": ["CVE-2023-45853"]}, + "easyconfig": {"file": "zlib-1.3.1-GCCcore-15.2.0.eb", "sha256": "00".repeat(32)}, + }) +} + +fn eb(args: &[&str]) -> Output { + Command::new(env!("CARGO_BIN_EXE_eb-stack")) + .args(args) + .output() + .expect("run eb-stack") +} + +fn ok(output: &Output) -> String { + assert!( + output.status.success(), + "status {:?}\nstdout:\n{}\nstderr:\n{}", + output.status.code(), + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + String::from_utf8(output.stdout.clone()).expect("utf8 stdout") +} + +fn failed(output: &Output) -> String { + assert_eq!( + output.status.code(), + Some(1), + "expected a failed check\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + String::from_utf8_lossy(&output.stderr).into_owned() +} + +fn s(path: &Path) -> &str { + path.to_str().expect("utf8 path") +} + +/// Append `value` and write the signed head to `head`. +fn append(fixture: &Fixture, value: &Value, head: &Path) -> String { + let file = fixture.work.join("record.json"); + std::fs::write(&file, serde_json::to_string_pretty(value).unwrap()).expect("record"); + ok(&eb(&[ + "ledger", + "append", + s(&file), + "--log", + s(&fixture.log), + "--key", + s(&fixture.key), + "--head-out", + s(head), + ])) +} + +fn verify_record(fixture: &Fixture, record: usize, head: &Path) -> Output { + eb(&[ + "ledger", + "verify", + "--log", + s(&fixture.log), + "--public-key", + s(&fixture.public), + "--record", + &record.to_string(), + "--head", + s(head), + ]) +} + +fn verify_range(fixture: &Fixture, from: &Path, to: &Path) -> Output { + eb(&[ + "ledger", + "verify", + "--log", + s(&fixture.log), + "--public-key", + s(&fixture.public), + "--from", + s(from), + "--to", + s(to), + ]) +} + +/// Three records, with the signed head after each. +fn three(fixture: &Fixture) -> [PathBuf; 3] { + let heads = [1, 2, 3].map(|n| fixture.work.join(format!("head-{n}.json"))); + for (n, module) in [ + "zlib/1.3.1-GCCcore-15.2.0", + "PETSc/3.25.0-foss-2026.1", + "PETSc/3.25.1-foss-2026.1", + ] + .iter() + .enumerate() + { + let out = append(fixture, &record(module, "jenkins/run/7"), &heads[n]); + assert!( + out.starts_with(&format!("appended record {n} id ")), + "{out}" + ); + } + heads +} + +/// Rewrite the log file from records, renumbering them. +fn rewrite(fixture: &Fixture, records: &[Value]) { + let mut text = String::new(); + for (index, value) in records.iter().enumerate() { + let mut value = value.clone(); + let object = value.as_object_mut().unwrap(); + object.remove("id"); + object.remove("index"); + let entry = ledger::seal_record(&value, index, "").expect("seal"); + text.push_str(&entry.line); + text.push('\n'); + } + std::fs::write(&fixture.log, text).expect("rewrite log"); +} + +#[test] +fn append_and_verify_round_trip() { + let fixture = fixture(); + let heads = three(&fixture); + + let text = std::fs::read_to_string(&fixture.log).expect("log"); + let lines = text.lines().collect::>(); + assert_eq!(lines.len(), 3); + for (index, line) in lines.iter().enumerate() { + let value: Value = serde_json::from_str(line).expect("json line"); + assert_eq!(value["index"], index); + assert_eq!(ledger::canonical(&value).unwrap(), *line); + } + + for (n, head) in heads.iter().enumerate() { + for record in 0..=n { + let out = ok(&verify_record(&fixture, record, head)); + assert!(out.starts_with(&format!("record {record} id ")), "{out}"); + } + failed(&verify_record(&fixture, n + 1, head)); + } + for (from, to) in [(0, 1), (0, 2), (1, 2), (2, 2)] { + let out = ok(&verify_range(&fixture, &heads[from], &heads[to])); + assert!( + out.contains(&format!("extends the head of size {}", from + 1)), + "{out}" + ); + } + failed(&verify_range(&fixture, &heads[2], &heads[0])); + + let head: Value = + serde_json::from_str(&std::fs::read_to_string(&heads[2]).unwrap()).expect("head json"); + assert_eq!(head["domain"], "eb-stack-ledger-v1"); + assert_eq!(head["size"], 3); + let hex = fixture.work.join("ledger.hex"); + std::fs::write(&hex, format!("{}\n", head["public_key"].as_str().unwrap())).unwrap(); + ok(&eb(&[ + "ledger", + "verify", + "--log", + s(&fixture.log), + "--public-key", + s(&hex), + "--record", + "2", + "--head", + s(&heads[2]), + ])); + + let show = ok(&eb(&[ + "ledger", + "show", + "--log", + s(&fixture.log), + "--package", + "PETSc", + ])); + let rows = show.lines().collect::>(); + assert_eq!(rows[0], "index\tts\taction\ttarget\tmodule\tactor"); + assert!(rows[1].starts_with("2\t"), "newest first: {show}"); + assert!(rows[2].starts_with("1\t"), "{show}"); + assert_eq!(rows[3], "2 of 3 records"); + let exact = ok(&eb(&[ + "ledger", + "show", + "--log", + s(&fixture.log), + "--package", + "PETSc/3.25.0-foss-2026.1", + "--json", + ])); + let records: Value = serde_json::from_str(&exact).expect("show json"); + assert_eq!(records.as_array().unwrap().len(), 1); + assert_eq!(records[0]["index"], 1); +} + +#[test] +fn a_head_signed_by_another_key_fails() { + let fixture = fixture(); + let heads = three(&fixture); + let other = SigningKey::from_bytes(&[12u8; 32]); + std::fs::write( + &fixture.public, + other + .verifying_key() + .to_public_key_pem(LineEnding::LF) + .unwrap(), + ) + .unwrap(); + let error = failed(&verify_record(&fixture, 0, &heads[2])); + assert!(error.contains("not by the trusted key"), "{error}"); +} + +#[test] +fn a_removed_record_fails_inclusion() { + let fixture = fixture(); + let heads = three(&fixture); + let text = std::fs::read_to_string(&fixture.log).unwrap(); + let lines = text.lines().map(str::to_string).collect::>(); + + // The line deleted as it stands: the log no longer reads. + std::fs::write(&fixture.log, format!("{}\n{}\n", lines[0], lines[2])).unwrap(); + failed(&verify_record(&fixture, 1, &heads[2])); + + // The line deleted and the rest renumbered, then a record appended so the + // log has as many records as the head: every line reads and the proof + // still fails. + let mut kept = vec![ + serde_json::from_str::(&lines[0]).unwrap(), + serde_json::from_str::(&lines[2]).unwrap(), + ]; + kept.push(record("HDF5/1.14.6-gompi-2026.1", "jenkins/run/8")); + rewrite(&fixture, &kept); + ok(&eb(&["ledger", "show", "--log", s(&fixture.log)])); + for record in 0..3 { + let error = failed(&verify_record(&fixture, record, &heads[2])); + assert!(error.contains("is not in the head of size 3"), "{error}"); + } + // Record 0 is unchanged and still proves against the head that covers + // only it. + ok(&verify_record(&fixture, 0, &heads[0])); +} + +#[test] +fn a_rewritten_log_fails_consistency() { + let fixture = fixture(); + let heads = three(&fixture); + let text = std::fs::read_to_string(&fixture.log).unwrap(); + let mut records = text + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .collect::>(); + // History rewritten: record 0 gets another actor, then one more record + // goes on the end and the same key signs the new head. + records[0]["actor"] = json!("someone-else"); + records.push(record("HDF5/1.14.6-gompi-2026.1", "jenkins/run/8")); + rewrite(&fixture, &records); + let later = fixture.work.join("head-4.json"); + let signed = ok(&eb(&[ + "ledger", + "head", + "--log", + s(&fixture.log), + "--key", + s(&fixture.key), + "--out", + s(&later), + "--ts", + "2026-11-04T00:00:00Z", + ])); + assert!(signed.starts_with("signed head size 4 root "), "{signed}"); + for earlier in &heads { + let error = failed(&verify_range(&fixture, earlier, &later)); + assert!(error.contains("does not extend"), "{error}"); + } + // The rewritten log is consistent with itself. + let again = fixture.work.join("head-4b.json"); + ok(&eb(&[ + "ledger", + "head", + "--log", + s(&fixture.log), + "--key", + s(&fixture.key), + "--out", + s(&again), + ])); + ok(&verify_range(&fixture, &later, &again)); +} + +#[test] +fn canonical_form_is_stable_across_key_order() { + let forward = record("zlib/1.3.1-GCCcore-15.2.0", "jenkins/run/7"); + let mut reversed = serde_json::Map::new(); + let object = forward.as_object().unwrap(); + let mut keys = object.keys().cloned().collect::>(); + keys.reverse(); + for key in keys { + let value = match &object[&key] { + Value::Object(inner) => { + let mut flipped = serde_json::Map::new(); + let mut names = inner.keys().cloned().collect::>(); + names.reverse(); + for name in names { + flipped.insert(name.clone(), inner[&name].clone()); + } + Value::Object(flipped) + } + other => other.clone(), + }; + reversed.insert(key, value); + } + let reversed = Value::Object(reversed); + + let a = fixture(); + let b = fixture(); + let one = a.work.join("forward.json"); + let two = b.work.join("reversed.json"); + std::fs::write(&one, serde_json::to_string_pretty(&forward).unwrap()).unwrap(); + // Hand-written text with the keys in reverse order and extra spaces. + let mut text = String::from("{ "); + for (at, (key, value)) in reversed.as_object().unwrap().iter().enumerate() { + if at > 0 { + text.push_str(" ,\n "); + } + text.push_str(&format!("{} : {}", json!(key), value)); + } + text.push_str(" }\n"); + std::fs::write(&two, &text).unwrap(); + for (fixture, file) in [(&a, &one), (&b, &two)] { + ok(&eb(&[ + "ledger", + "append", + s(file), + "--log", + s(&fixture.log), + ])); + } + let left = std::fs::read(&a.log).unwrap(); + let right = std::fs::read(&b.log).unwrap(); + assert_eq!( + String::from_utf8_lossy(&left), + String::from_utf8_lossy(&right) + ); + let line = String::from_utf8(left).unwrap(); + assert!( + line.starts_with("{\"action\":\"install\",\"actor\":"), + "{line}" + ); + assert!(line.contains(",\"easyconfig\":{\"file\":"), "{line}"); + assert!( + line.ends_with(",\"ts\":\"2026-11-03T10:00:00Z\"}\n"), + "{line}" + ); + assert_eq!( + ledger::record_id(forward.as_object().unwrap()).unwrap(), + ledger::record_id(reversed.as_object().unwrap()).unwrap() + ); +} + +#[test] +fn a_key_inside_the_ledger_directory_is_refused() { + let fixture = fixture(); + let beside = fixture.log.parent().unwrap().join("ledger.pem"); + std::fs::copy(&fixture.key, &beside).unwrap(); + let file = fixture.work.join("record.json"); + std::fs::write(&file, record("zlib/1.3.1", "jenkins/run/7").to_string()).unwrap(); + let error = failed(&eb(&[ + "ledger", + "append", + s(&file), + "--log", + s(&fixture.log), + "--key", + s(&beside), + "--head-out", + s(&fixture.work.join("head.json")), + ])); + assert!(error.contains("inside the ledger directory"), "{error}"); + assert!(!fixture.log.exists(), "nothing is appended"); +} + +#[test] +fn append_refuses_a_stale_head_and_another_stack() { + let fixture = fixture(); + let heads = three(&fixture); + let file = fixture.work.join("record.json"); + std::fs::write(&file, record("HDF5/1.14.6", "jenkins/run/8").to_string()).unwrap(); + let error = failed(&eb(&[ + "ledger", + "append", + s(&file), + "--log", + s(&fixture.log), + "--expect-head", + s(&heads[1]), + ])); + assert!(error.contains("not the expected head size 2"), "{error}"); + ok(&eb(&[ + "ledger", + "append", + s(&file), + "--log", + s(&fixture.log), + "--expect-head", + s(&heads[2]), + ])); + + let mut other = record("HDF5/1.14.6", "jenkins/run/9"); + other["stack"] = json!("2025"); + std::fs::write(&file, other.to_string()).unwrap(); + let error = failed(&eb(&[ + "ledger", + "append", + s(&file), + "--log", + s(&fixture.log), + ])); + assert!(error.contains("stack"), "{error}"); +} + +#[test] +fn verify_needs_one_proof_on_the_command_line() { + let fixture = fixture(); + let bare = eb(&[ + "ledger", + "verify", + "--log", + s(&fixture.log), + "--public-key", + s(&fixture.public), + ]); + assert_eq!(bare.status.code(), Some(2)); + let half = eb(&[ + "ledger", + "verify", + "--log", + s(&fixture.log), + "--public-key", + s(&fixture.public), + "--record", + "0", + ]); + assert_eq!(half.status.code(), Some(2)); +} + +fn mcp(name: &str, arguments: Value) -> Value { + let response = eb_stack::mcp::handle_message(&json!({ + "jsonrpc": "2.0", + "id": 1, + "method": "tools/call", + "params": {"name": name, "arguments": arguments} + })) + .expect("tools/call response"); + response["result"].clone() +} + +#[test] +fn mcp_ledger_tools_append_show_and_verify() { + let fixture = fixture(); + let file = fixture.work.join("record.json"); + let mut heads = Vec::new(); + for (n, module) in ["zlib/1.3.1", "PETSc/3.25.0-foss-2026.1"] + .iter() + .enumerate() + { + std::fs::write(&file, record(module, "jenkins/run/7").to_string()).unwrap(); + let head = fixture.work.join(format!("mcp-head-{n}.json")); + let result = mcp( + "eb_ledger_append", + json!({ + "log": s(&fixture.log), + "record": s(&file), + "key": s(&fixture.key), + "head_out": s(&head), + }), + ); + assert_ne!(result["isError"], true, "{result}"); + heads.push(head); + } + let show = mcp( + "eb_ledger_show", + json!({"log": s(&fixture.log), "package": "PETSc"}), + ); + assert_ne!(show["isError"], true, "{show}"); + assert!(show.to_string().contains("1 of 2 records"), "{show}"); + let inclusion = mcp( + "eb_ledger_verify", + json!({ + "log": s(&fixture.log), + "public_key": s(&fixture.public), + "record": 0, + "head": s(&heads[1]), + }), + ); + assert_ne!(inclusion["isError"], true, "{inclusion}"); + let consistency = mcp( + "eb_ledger_verify", + json!({ + "log": s(&fixture.log), + "public_key": s(&fixture.public), + "from": s(&heads[0]), + "to": s(&heads[1]), + }), + ); + assert_ne!(consistency["isError"], true, "{consistency}"); + let past = mcp( + "eb_ledger_verify", + json!({ + "log": s(&fixture.log), + "public_key": s(&fixture.public), + "record": 1, + "head": s(&heads[0]), + }), + ); + assert_eq!(past["isError"], true, "{past}"); +} diff --git a/tests/ledger_durability.rs b/tests/ledger_durability.rs new file mode 100644 index 00000000..303aa1c0 --- /dev/null +++ b/tests/ledger_durability.rs @@ -0,0 +1,566 @@ +//! Crash, alias and concurrency behavior of the ledger writer and signer. + +#![cfg(unix)] + +use eb_stack::ledger::{self, SignedHead}; +use ed25519_dalek::pkcs8::spki::der::pem::LineEnding; +use ed25519_dalek::pkcs8::EncodePrivateKey; +use ed25519_dalek::SigningKey; +use serde_json::{json, Value}; +use std::fs; +use std::os::fd::AsRawFd; +use std::os::unix::fs::PermissionsExt; +use std::os::unix::process::CommandExt; +use std::path::{Path, PathBuf}; +use std::process::{Command, Output, Stdio}; +use std::time::Duration; + +const SEED: [u8; 32] = [23; 32]; + +struct Fixture { + root: tempfile::TempDir, + dir: PathBuf, + log: PathBuf, + lock: PathBuf, + key: PathBuf, + work: PathBuf, +} + +impl Fixture { + fn new() -> Self { + let root = tempfile::tempdir().unwrap(); + for name in ["ledger", "keys", "work"] { + fs::create_dir(root.path().join(name)).unwrap(); + } + let key = root.path().join("keys/key.pem"); + let pem = SigningKey::from_bytes(&SEED) + .to_pkcs8_pem(LineEnding::LF) + .unwrap(); + fs::write(&key, pem.as_bytes()).unwrap(); + Self { + dir: root.path().join("ledger"), + log: root.path().join("ledger/stack.jsonl"), + lock: root.path().join("ledger/stack.jsonl.posix-lock"), + work: root.path().join("work"), + key, + root, + } + } + + fn record_file(&self, event: &str) -> PathBuf { + let path = self.work.join(format!("{}.json", event.replace('/', "-"))); + fs::write(&path, serde_json::to_vec(&record(event)).unwrap()).unwrap(); + path + } + + fn append(&self, event: &str) -> Command { + let mut command = Command::new(env!("CARGO_BIN_EXE_eb-stack")); + command + .args(["ledger", "append"]) + .arg(self.record_file(event)) + .arg("--log") + .arg(&self.log); + command + } + + fn append_signed(&self, event: &str, head: &Path) -> Command { + let mut command = self.append(event); + command + .arg("--key") + .arg(&self.key) + .arg("--head-out") + .arg(head); + command + } + + fn head(&self, out: &Path) -> Command { + let mut command = Command::new(env!("CARGO_BIN_EXE_eb-stack")); + command + .args(["ledger", "head", "--log"]) + .arg(&self.log) + .arg("--key") + .arg(&self.key) + .arg("--out") + .arg(out); + command + } + + fn entries(&self) -> Vec { + ledger::read_log(&self.log).unwrap() + } + + fn ledger_files(&self) -> Vec { + let mut names = fs::read_dir(&self.dir) + .unwrap() + .map(|entry| entry.unwrap().file_name().into_string().unwrap()) + .collect::>(); + names.sort(); + names + } +} + +fn record(event: &str) -> Value { + json!({ + "ts": "2026-11-05T00:00:00Z", "actor": "durability-test", "stack": "2026.06", + "target": "x86_64/amd/zen4", "module": "example/1", "location": "/example/1", + "action": "install", "build": {"event_id": event} + }) +} + +fn line(event: &str, index: usize) -> String { + ledger::seal_record(&record(event), index, "").unwrap().line +} + +fn success(output: &Output) -> String { + assert!( + output.status.success(), + "status {:?}\nstdout:\n{}\nstderr:\n{}", + output.status.code(), + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + String::from_utf8_lossy(&output.stdout).into_owned() +} + +fn failure(output: &Output) -> String { + assert!( + !output.status.success(), + "expected a refusal\nstdout:\n{}", + String::from_utf8_lossy(&output.stdout) + ); + String::from_utf8_lossy(&output.stderr).into_owned() +} + +/// Run with writes past `bytes` failing with `EFBIG`, as a full disk or +/// quota fails them part way. +fn with_file_size_limit(command: &mut Command, bytes: u64) -> Output { + // The closure only calls async-signal-safe functions between fork and exec. + unsafe { + command.pre_exec(move || { + libc::signal(libc::SIGXFSZ, libc::SIG_IGN); + let limit = libc::rlimit { + rlim_cur: bytes as libc::rlim_t, + rlim_max: bytes as libc::rlim_t, + }; + if libc::setrlimit(libc::RLIMIT_FSIZE, &limit) != 0 { + return Err(std::io::Error::last_os_error()); + } + Ok(()) + }); + } + command.output().unwrap() +} + +fn record_lock(file: &fs::File, kind: libc::c_int) { + // The descriptor is live and the initialized flock covers the whole file. + let mut lock: libc::flock = unsafe { std::mem::zeroed() }; + lock.l_type = kind as _; + lock.l_whence = libc::SEEK_SET as _; + assert_eq!( + unsafe { libc::fcntl(file.as_raw_fd(), libc::F_SETLK, &lock) }, + 0, + "{}", + std::io::Error::last_os_error() + ); +} + +fn check_head(fixture: &Fixture, head: &Path) -> SignedHead { + let signed = SignedHead::read(head).unwrap(); + signed + .check(&SigningKey::from_bytes(&SEED).verifying_key()) + .unwrap(); + let entries = fixture.entries(); + assert_eq!( + signed.root, + rfc9162::Head::over(&ledger::leaves(&entries[..signed.size])).root + ); + signed +} + +fn chmod(path: &Path, mode: u32) { + fs::set_permissions(path, fs::Permissions::from_mode(mode)).unwrap(); +} + +#[test] +fn a_short_append_is_cut_back_and_the_next_append_succeeds() { + let fixture = Fixture::new(); + success(&fixture.append("first").output().unwrap()); + let original = fs::read(&fixture.log).unwrap(); + let error = failure(&with_file_size_limit( + &mut fixture.append("second"), + original.len() as u64 + 20, + )); + assert!(error.contains("append to ledger"), "{error}"); + assert_eq!(fs::read(&fixture.log).unwrap(), original); + success(&fixture.append("second").output().unwrap()); + assert_eq!(fixture.entries().len(), 2); + assert!( + !fixture + .ledger_files() + .iter() + .any(|name| name.contains("quarantine")), + "a rolled back write leaves no tail to quarantine" + ); +} + +#[test] +fn an_interrupted_tail_is_quarantined_before_the_log_is_repaired() { + let fixture = Fixture::new(); + let first = line("first", 0); + let torn = line("torn", 1); + let mut tail = torn.as_bytes()[..40].to_vec(); + tail.extend_from_slice("é".as_bytes().split_at(1).0); + let mut bytes = format!("{first}\n").into_bytes(); + bytes.extend_from_slice(&tail); + fs::write(&fixture.log, &bytes).unwrap(); + + let out = success(&fixture.append("second").output().unwrap()); + assert!(out.starts_with("appended record 1 id "), "{out}"); + assert!(out.contains("interrupted append"), "{out}"); + let text = fs::read_to_string(&fixture.log).unwrap(); + assert_eq!(text, format!("{first}\n{}\n", line("second", 1))); + let quarantined = fixture + .ledger_files() + .into_iter() + .filter(|name| name.starts_with("stack.jsonl.quarantine.")) + .collect::>(); + assert_eq!(quarantined.len(), 1, "{:?}", fixture.ledger_files()); + assert_eq!(fs::read(fixture.dir.join(&quarantined[0])).unwrap(), tail); +} + +#[test] +fn an_acknowledged_retry_also_repairs_an_interrupted_tail() { + let fixture = Fixture::new(); + let head = fixture.work.join("head.json"); + success(&fixture.append_signed("first", &head).output().unwrap()); + let committed = fs::read(&fixture.log).unwrap(); + let mut bytes = committed.clone(); + bytes.extend_from_slice(&line("first", 1).as_bytes()[..17]); + fs::write(&fixture.log, &bytes).unwrap(); + let out = success(&fixture.append_signed("first", &head).output().unwrap()); + assert!(out.starts_with("already recorded record 0"), "{out}"); + assert_eq!(fs::read(&fixture.log).unwrap(), committed); + assert_eq!(check_head(&fixture, &head).size, 1); +} + +#[test] +fn a_complete_record_without_its_newline_is_not_repaired() { + let fixture = Fixture::new(); + let original = format!("{}\n{}", line("first", 0), line("second", 1)); + fs::write(&fixture.log, &original).unwrap(); + failure(&fixture.append("third").output().unwrap()); + assert_eq!(fs::read_to_string(&fixture.log).unwrap(), original); + assert!(!fixture + .ledger_files() + .iter() + .any(|name| name.contains("quarantine"))); +} + +#[test] +fn a_tail_after_a_corrupt_history_is_left_alone() { + let fixture = Fixture::new(); + let mut original = format!("{}\n", line("first", 0)).replacen(':', ": ", 1); + original.push_str(&line("second", 1)[..30]); + fs::write(&fixture.log, &original).unwrap(); + failure(&fixture.append("third").output().unwrap()); + assert_eq!(fs::read_to_string(&fixture.log).unwrap(), original); + assert!(!fixture + .ledger_files() + .iter() + .any(|name| name.contains("quarantine"))); +} + +#[test] +fn an_unsaved_tail_stays_in_the_log() { + let fixture = Fixture::new(); + success(&fixture.append("first").output().unwrap()); + let mut original = fs::read(&fixture.log).unwrap(); + original.extend_from_slice(&line("torn", 1).as_bytes()[..25]); + fs::write(&fixture.log, &original).unwrap(); + chmod(&fixture.dir, 0o555); + let output = fixture.append("second").output().unwrap(); + chmod(&fixture.dir, 0o755); + failure(&output); + assert_eq!(fs::read(&fixture.log).unwrap(), original); +} + +#[test] +fn a_failed_head_write_keeps_the_previous_head() { + let fixture = Fixture::new(); + let head = fixture.work.join("head.json"); + success(&fixture.append_signed("first", &head).output().unwrap()); + success(&fixture.append("second").output().unwrap()); + let previous = fs::read(&head).unwrap(); + failure(&with_file_size_limit(&mut fixture.head(&head), 16)); + assert_eq!(fs::read(&head).unwrap(), previous); + assert_eq!(check_head(&fixture, &head).size, 1); + let names = fs::read_dir(&fixture.work) + .unwrap() + .map(|entry| entry.unwrap().file_name().into_string().unwrap()) + .filter(|name| name.ends_with(".tmp")) + .collect::>(); + assert!(names.is_empty(), "temporary files left: {names:?}"); +} + +#[test] +fn a_head_path_naming_the_log_is_refused() { + let fixture = Fixture::new(); + success(&fixture.append("first").output().unwrap()); + let original = fs::read(&fixture.log).unwrap(); + let lexical = fixture.dir.join(".").join("stack.jsonl"); + failure(&fixture.append_signed("second", &lexical).output().unwrap()); + assert_eq!(fs::read(&fixture.log).unwrap(), original); + let dotted = fixture.work.join("..").join("ledger/stack.jsonl"); + failure(&fixture.head(&dotted).output().unwrap()); + assert_eq!(fs::read(&fixture.log).unwrap(), original); + let link = fixture.work.join("head.json"); + std::os::unix::fs::symlink(&fixture.log, &link).unwrap(); + failure(&fixture.head(&link).output().unwrap()); + failure(&fixture.append_signed("second", &link).output().unwrap()); + assert_eq!(fs::read(&fixture.log).unwrap(), original); + assert!(fs::symlink_metadata(&link) + .unwrap() + .file_type() + .is_symlink()); +} + +#[test] +fn a_head_path_naming_the_record_lock_or_a_quarantine_is_refused() { + let fixture = Fixture::new(); + success(&fixture.append("first").output().unwrap()); + let lock = fs::metadata(&fixture.lock).unwrap(); + failure(&fixture.head(&fixture.lock).output().unwrap()); + let after = fs::metadata(&fixture.lock).unwrap(); + use std::os::unix::fs::MetadataExt; + assert_eq!((lock.ino(), lock.len()), (after.ino(), after.len())); + let quarantine = fixture.dir.join("stack.jsonl.quarantine.saved"); + fs::write(&quarantine, b"{\"torn").unwrap(); + failure(&fixture.head(&quarantine).output().unwrap()); + assert_eq!(fs::read(&quarantine).unwrap(), b"{\"torn"); +} + +#[test] +fn a_head_path_naming_the_signing_key_is_refused() { + let fixture = Fixture::new(); + success(&fixture.append("first").output().unwrap()); + let key = fs::read(&fixture.key).unwrap(); + let lexical = fixture.work.join("../keys/./key.pem"); + failure(&fixture.append_signed("second", &lexical).output().unwrap()); + failure(&fixture.head(&lexical).output().unwrap()); + let link = fixture.work.join("key-link.pem"); + std::os::unix::fs::symlink(&fixture.key, &link).unwrap(); + failure(&fixture.head(&link).output().unwrap()); + let hard = fixture.work.join("key-hard.pem"); + fs::hard_link(&fixture.key, &hard).unwrap(); + failure(&fixture.head(&hard).output().unwrap()); + failure(&fixture.append_signed("second", &hard).output().unwrap()); + assert_eq!(fs::read(&fixture.key).unwrap(), key); + assert_eq!(fs::read(&hard).unwrap(), key); + assert_eq!(fixture.entries().len(), 1); +} + +#[test] +fn a_signing_key_hard_linked_into_the_ledger_directory_is_refused() { + let fixture = Fixture::new(); + fs::hard_link(&fixture.key, fixture.dir.join("travels.pem")).unwrap(); + let head = fixture.work.join("head.json"); + failure(&fixture.append_signed("first", &head).output().unwrap()); + assert!(!fixture.log.exists(), "nothing is appended"); + assert!(!head.exists()); + fs::write(&fixture.log, "").unwrap(); + failure(&fixture.head(&head).output().unwrap()); + assert!(!head.exists()); +} + +#[test] +fn a_signer_waits_for_the_writer_and_signs_its_append() { + let fixture = Fixture::new(); + let head = fixture.work.join("head.json"); + success(&fixture.append("first").output().unwrap()); + let holder = fs::OpenOptions::new() + .read(true) + .write(true) + .open(&fixture.lock) + .unwrap(); + record_lock(&holder, libc::F_WRLCK); + let mut signer = fixture + .head(&head) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(); + std::thread::sleep(Duration::from_millis(400)); + assert!( + signer.try_wait().unwrap().is_none(), + "the signer read the log while a writer held it" + ); + let mut log = fs::OpenOptions::new() + .append(true) + .open(&fixture.log) + .unwrap(); + std::io::Write::write_all(&mut log, format!("{}\n", line("second", 1)).as_bytes()).unwrap(); + drop(holder); + success(&signer.wait_with_output().unwrap()); + assert_eq!(check_head(&fixture, &head).size, 2); +} + +#[test] +fn a_read_only_signer_needs_no_write_access_to_the_ledger() { + let fixture = Fixture::new(); + success(&fixture.append("first").output().unwrap()); + success(&fixture.append("second").output().unwrap()); + let before = fixture.ledger_files(); + let reader = fs::File::open(&fixture.lock).unwrap(); + record_lock(&reader, libc::F_RDLCK); + chmod(&fixture.log, 0o444); + chmod(&fixture.lock, 0o444); + chmod(&fixture.dir, 0o555); + let head = fixture.work.join("head.json"); + let output = fixture.head(&head).output().unwrap(); + chmod(&fixture.dir, 0o755); + success(&output); + assert_eq!(check_head(&fixture, &head).size, 2); + assert_eq!(fixture.ledger_files(), before); +} + +#[test] +fn a_read_only_signer_still_waits_for_the_writer() { + let fixture = Fixture::new(); + success(&fixture.append("first").output().unwrap()); + let holder = fs::OpenOptions::new() + .read(true) + .write(true) + .open(&fixture.lock) + .unwrap(); + record_lock(&holder, libc::F_WRLCK); + let mut log = fs::OpenOptions::new() + .append(true) + .open(&fixture.log) + .unwrap(); + chmod(&fixture.log, 0o444); + chmod(&fixture.lock, 0o444); + chmod(&fixture.dir, 0o555); + let head = fixture.work.join("head.json"); + let mut signer = fixture + .head(&head) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(); + std::thread::sleep(Duration::from_millis(400)); + let waited = signer.try_wait().unwrap().is_none(); + std::io::Write::write_all(&mut log, format!("{}\n", line("second", 1)).as_bytes()).unwrap(); + drop(holder); + let output = signer.wait_with_output().unwrap(); + chmod(&fixture.dir, 0o755); + assert!(waited, "the read-only signer did not wait for the writer"); + success(&output); + assert_eq!(check_head(&fixture, &head).size, 2); +} + +#[test] +fn a_read_only_signer_refuses_a_log_without_its_record_lock() { + let fixture = Fixture::new(); + fs::write(&fixture.log, format!("{}\n", line("first", 0))).unwrap(); + chmod(&fixture.dir, 0o555); + let head = fixture.work.join("head.json"); + let output = fixture.head(&head).output().unwrap(); + chmod(&fixture.dir, 0o755); + let error = failure(&output); + assert!(error.contains("record lock"), "{error}"); + assert!(!head.exists()); + assert_eq!(fixture.ledger_files(), vec!["stack.jsonl".to_string()]); +} + +#[test] +fn concurrent_writers_and_signers_leave_the_newest_head() { + let fixture = Fixture::new(); + let head = fixture.work.join("head.json"); + success(&fixture.append("seed").output().unwrap()); + let mut total = 1; + for round in 0..3 { + let mut children = Vec::new(); + for n in 0..8 { + let event = format!("round-{round}/event-{n}"); + children.push( + fixture + .append_signed(&event, &head) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(), + ); + children.push( + fixture + .head(&head) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(), + ); + } + for child in children { + success(&child.wait_with_output().unwrap()); + } + total += 8; + assert_eq!(fixture.entries().len(), total); + assert_eq!(check_head(&fixture, &head).size, total, "round {round}"); + } +} + +#[test] +fn in_process_signers_keep_cross_process_writers_out() { + let fixture = Fixture::new(); + let head = fixture.work.join("head.json"); + success(&fixture.append("seed").output().unwrap()); + let mut children = (0..12) + .map(|n| { + fixture + .append_signed(&format!("child-{n}"), &head) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap() + }) + .collect::>(); + let records = (0..12) + .map(|n| fixture.record_file(&format!("thread-{n}"))) + .collect::>(); + std::thread::scope(|scope| { + for n in 0..4 { + let (fixture, head) = (&fixture, &head); + scope.spawn(move || { + for _ in 0..15 { + ledger::sign_head(&ledger::HeadRequest { + log: fixture.log.clone(), + key: fixture.key.clone(), + out: head.clone(), + ts: None, + }) + .unwrap_or_else(|error| panic!("signer {n}: {error:#}")); + } + }); + } + for chunk in records.chunks(3) { + let (fixture, head) = (&fixture, &head); + scope.spawn(move || { + for record in chunk { + ledger::append(&ledger::AppendRequest { + log: fixture.log.clone(), + record: record.clone(), + expect_head: None, + key: Some(fixture.key.clone()), + head_out: Some(head.clone()), + }) + .unwrap_or_else(|error| panic!("writer: {error:#}")); + } + }); + } + }); + for child in children.drain(..) { + success(&child.wait_with_output().unwrap()); + } + assert_eq!(fixture.entries().len(), 25); + assert_eq!(check_head(&fixture, &head).size, 25); + drop(fixture.root); +} diff --git a/tests/ledger_lock_aliases.rs b/tests/ledger_lock_aliases.rs new file mode 100644 index 00000000..5a8c444e --- /dev/null +++ b/tests/ledger_lock_aliases.rs @@ -0,0 +1,65 @@ +#![cfg(unix)] + +use std::fs; +use std::path::Path; +use std::process::Command; + +fn append(root: &Path, log: &Path) -> std::process::Output { + let record = root.join("record.json"); + fs::write( + &record, + serde_json::to_vec(&serde_json::json!({ + "ts": "2026-11-05T00:00:00Z", "actor": "lock-test", "stack": "2026.06", + "target": "x86_64/amd/zen4", "module": "example/1", "location": "/example/1", + "action": "install" + })) + .unwrap(), + ) + .unwrap(); + Command::new(env!("CARGO_BIN_EXE_eb-stack")) + .args(["ledger", "append"]) + .arg(record) + .arg("--log") + .arg(log) + .output() + .unwrap() +} + +#[test] +fn hardlinked_log_cannot_select_a_different_record_lock() { + let root = tempfile::tempdir().unwrap(); + let log = root.path().join("ledger.jsonl"); + let alias = root.path().join("alias.jsonl"); + fs::write(&log, "").unwrap(); + fs::hard_link(&log, &alias).unwrap(); + let output = append(root.path(), &alias); + assert!(!output.status.success(), "hardlinked ledger accepted"); + assert_eq!(fs::read(&log).unwrap(), b""); + assert_eq!(fs::read(&alias).unwrap(), b""); +} + +#[test] +fn symbolic_record_lock_is_refused() { + let root = tempfile::tempdir().unwrap(); + let log = root.path().join("ledger.jsonl"); + let outside = root.path().join("outside"); + fs::write(&outside, "protected").unwrap(); + std::os::unix::fs::symlink(&outside, root.path().join("ledger.jsonl.posix-lock")).unwrap(); + let output = append(root.path(), &log); + assert!(!output.status.success(), "symbolic record lock accepted"); + assert_eq!(fs::read(&log).unwrap(), b""); + assert_eq!(fs::read(&outside).unwrap(), b"protected"); +} + +#[test] +fn hardlinked_record_lock_is_refused() { + let root = tempfile::tempdir().unwrap(); + let log = root.path().join("ledger.jsonl"); + let outside = root.path().join("outside"); + fs::write(&outside, "protected").unwrap(); + fs::hard_link(&outside, root.path().join("ledger.jsonl.posix-lock")).unwrap(); + let output = append(root.path(), &log); + assert!(!output.status.success(), "hardlinked record lock accepted"); + assert_eq!(fs::read(&log).unwrap(), b""); + assert_eq!(fs::read(&outside).unwrap(), b"protected"); +} diff --git a/tests/ledger_posix_lock.rs b/tests/ledger_posix_lock.rs new file mode 100644 index 00000000..705394eb --- /dev/null +++ b/tests/ledger_posix_lock.rs @@ -0,0 +1,61 @@ +#![cfg(unix)] + +use std::fs::OpenOptions; +use std::os::fd::AsRawFd; +use std::process::{Command, Stdio}; +use std::time::Duration; + +#[test] +fn append_waits_for_the_posix_writer() { + let root = tempfile::tempdir().unwrap(); + let log = root.path().join("ledger.jsonl"); + let sidecar = root.path().join("ledger.jsonl.posix-lock"); + let record = root.path().join("record.json"); + std::fs::write(&record, serde_json::to_vec(&serde_json::json!({ + "ts": "2026-11-05T00:00:00Z", "actor": "lock-test", "stack": "2026.06", "target": "x86_64/amd/zen4", + "module": "example/1", "location": "/example/1", "action": "install" + })).unwrap()).unwrap(); + let holder = OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(&sidecar) + .unwrap(); + // The file descriptor is live and the initialized flock covers the file. + let mut lock: libc::flock = unsafe { std::mem::zeroed() }; + lock.l_type = libc::F_WRLCK as _; + lock.l_whence = libc::SEEK_SET as _; + assert_eq!( + unsafe { libc::fcntl(holder.as_raw_fd(), libc::F_SETLK, &lock) }, + 0 + ); + let mut child = Command::new(env!("CARGO_BIN_EXE_eb-stack")) + .args(["ledger", "append"]) + .arg(&record) + .arg("--log") + .arg(&log) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(); + std::thread::sleep(Duration::from_millis(300)); + assert!( + child.try_wait().unwrap().is_none(), + "append bypassed the POSIX writer lock" + ); + assert_eq!(std::fs::metadata(&log).unwrap().len(), 0); + drop(holder); + let result = child.wait_with_output().unwrap(); + assert!( + result.status.success(), + "{}", + String::from_utf8_lossy(&result.stderr) + ); + let lines = std::fs::read_to_string(log).unwrap(); + let entries: Vec<_> = lines.lines().collect(); + assert_eq!(entries.len(), 1); + let entry: serde_json::Value = serde_json::from_str(entries[0]).unwrap(); + assert_eq!(entry["index"], 0); + assert_eq!(entry["module"], "example/1"); +} diff --git a/tests/ledger_retry.rs b/tests/ledger_retry.rs new file mode 100644 index 00000000..f1943654 --- /dev/null +++ b/tests/ledger_retry.rs @@ -0,0 +1,221 @@ +use eb_stack::ledger::{self, SignedHead}; +use ed25519_dalek::pkcs8::spki::der::pem::LineEnding; +use ed25519_dalek::pkcs8::EncodePrivateKey; +use ed25519_dalek::SigningKey; +use serde_json::{json, Value}; +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +struct Fixture { + _root: tempfile::TempDir, + log: PathBuf, + record: PathBuf, + key: PathBuf, + head: PathBuf, +} + +fn fixture() -> Fixture { + let root = tempfile::tempdir().unwrap(); + for directory in ["ledger", "keys", "work"] { + std::fs::create_dir(root.path().join(directory)).unwrap(); + } + let key = root.path().join("keys/key.pem"); + let signing = SigningKey::from_bytes(&[19; 32]); + std::fs::write( + &key, + signing.to_pkcs8_pem(LineEnding::LF).unwrap().as_bytes(), + ) + .unwrap(); + Fixture { + log: root.path().join("ledger/stack.jsonl"), + record: root.path().join("work/record.json"), + head: root.path().join("work/head.json"), + key, + _root: root, + } +} + +fn record(event: &str) -> Value { + json!({ + "ts": "2026-11-05T00:00:00Z", "actor": "retry-test", "stack": "2026.06", + "target": "x86_64/amd/zen4", "module": "example/1", "location": "/example/1", + "action": "install", "stage": "finished", "build": {"event_id": event} + }) +} + +fn save(fixture: &Fixture, value: &Value) { + std::fs::write(&fixture.record, serde_json::to_vec(value).unwrap()).unwrap(); +} + +fn command(fixture: &Fixture, head: &Path) -> Command { + let mut command = Command::new(env!("CARGO_BIN_EXE_eb-stack")); + command + .args(["ledger", "append"]) + .arg(&fixture.record) + .arg("--log") + .arg(&fixture.log) + .arg("--key") + .arg(&fixture.key) + .arg("--head-out") + .arg(head); + command +} + +fn success(output: Output) -> String { + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + String::from_utf8(output.stdout).unwrap() +} + +fn rows(fixture: &Fixture) -> Vec { + ledger::read_log(&fixture.log).unwrap() +} + +fn check_head(fixture: &Fixture, size: usize) { + let head = SignedHead::read(&fixture.head).unwrap(); + head.check(&SigningKey::from_bytes(&[19; 32]).verifying_key()) + .unwrap(); + assert_eq!(head.size, size); + let entries = rows(fixture); + assert_eq!( + head.root, + rfc9162::Head::over(&ledger::leaves(&entries)).root + ); +} + +#[test] +fn repeated_event_acknowledges_the_existing_entry() { + let fixture = fixture(); + save(&fixture, &record("run/module/finished")); + success(command(&fixture, &fixture.head).output().unwrap()); + let original = std::fs::read(&fixture.log).unwrap(); + let output = success(command(&fixture, &fixture.head).output().unwrap()); + assert_eq!(std::fs::read(&fixture.log).unwrap(), original); + assert!(output.contains("already recorded"), "{output}"); + check_head(&fixture, 1); +} + +#[test] +fn reused_event_identifier_with_different_content_is_refused() { + let fixture = fixture(); + let mut value = record("run/module/finished"); + save(&fixture, &value); + success(command(&fixture, &fixture.head).output().unwrap()); + let original = std::fs::read(&fixture.log).unwrap(); + value["reason"] = json!("different operation"); + save(&fixture, &value); + assert!(!command(&fixture, &fixture.head) + .output() + .unwrap() + .status + .success()); + assert_eq!(std::fs::read(&fixture.log).unwrap(), original); + check_head(&fixture, 1); +} + +#[test] +fn head_output_failure_can_be_retried_without_appending_again() { + let fixture = fixture(); + save(&fixture, &record("run/module/finished")); + let missing = fixture.head.parent().unwrap().join("missing/head.json"); + assert!(!command(&fixture, &missing) + .output() + .unwrap() + .status + .success()); + assert_eq!(rows(&fixture).len(), 1); + success(command(&fixture, &fixture.head).output().unwrap()); + assert_eq!(rows(&fixture).len(), 1); + check_head(&fixture, 1); +} + +#[test] +fn concurrent_retries_append_one_entry() { + let fixture = fixture(); + save(&fixture, &record("run/module/finished")); + let first = command(&fixture, &fixture.head).spawn().unwrap(); + let second = command(&fixture, &fixture.head).spawn().unwrap(); + assert!(first.wait_with_output().unwrap().status.success()); + assert!(second.wait_with_output().unwrap().status.success()); + assert_eq!(rows(&fixture).len(), 1); + check_head(&fixture, 1); +} + +#[test] +fn event_identifier_requires_a_nonempty_string() { + let fixture = fixture(); + let mut value = record("run/module/finished"); + value["build"]["event_id"] = json!(42); + save(&fixture, &value); + assert!(!command(&fixture, &fixture.head) + .output() + .unwrap() + .status + .success()); + assert!(rows(&fixture).is_empty()); +} + +#[test] +fn acknowledged_retry_checks_the_expected_history_prefix() { + let fixture = fixture(); + let empty = fixture.head.parent().unwrap().join("empty.json"); + SignedHead::sign( + &[], + &SigningKey::from_bytes(&[19; 32]), + "2026-11-05T00:00:00Z", + ) + .write(&empty) + .unwrap(); + save(&fixture, &record("first")); + success( + command(&fixture, &fixture.head) + .arg("--expect-head") + .arg(&empty) + .output() + .unwrap(), + ); + save(&fixture, &record("second")); + success(command(&fixture, &fixture.head).output().unwrap()); + save(&fixture, &record("first")); + success( + command(&fixture, &fixture.head) + .arg("--expect-head") + .arg(&empty) + .output() + .unwrap(), + ); + assert_eq!(rows(&fixture).len(), 2); + check_head(&fixture, 2); + let original = std::fs::read(&fixture.log).unwrap(); + let mut changed = SignedHead::read(&empty).unwrap(); + changed.root = "00".repeat(32); + changed.write(&empty).unwrap(); + assert!(!command(&fixture, &fixture.head) + .arg("--expect-head") + .arg(&empty) + .output() + .unwrap() + .status + .success()); + assert_eq!(std::fs::read(&fixture.log).unwrap(), original); +} + +#[test] +fn duplicated_event_identifiers_in_existing_history_are_refused() { + let fixture = fixture(); + let value = record("run/module/finished"); + let first = ledger::seal_record(&value, 0, "2026-11-05T00:00:00Z").unwrap(); + let second = ledger::seal_record(&value, 1, "2026-11-05T00:00:00Z").unwrap(); + let original = format!("{}\n{}\n", first.line, second.line); + std::fs::write(&fixture.log, &original).unwrap(); + save(&fixture, &value); + assert!(!command(&fixture, &fixture.head) + .output() + .unwrap() + .status + .success()); + assert_eq!(std::fs::read_to_string(&fixture.log).unwrap(), original); +} diff --git a/tests/llms_txt.rs b/tests/llms_txt.rs index 3ec1ddf8..b1b1b851 100644 --- a/tests/llms_txt.rs +++ b/tests/llms_txt.rs @@ -42,7 +42,7 @@ fn llms_txt_matches_the_org_manual() { assert!(index.contains("https://eb-stack.rgoswami.me/reference/mcp.html.md")); assert!(index.contains("server.json")); assert!(index.contains("skills/new-package/SKILL.md")); - assert!(index.contains("AGENTS.md")); + assert!(index.contains("CONTRIBUTING.md")); // A page summary is already markdown, so summarize() must not re-read it // as Org. A regression ran inline_plain() over the markdown and paired diff --git a/tests/mcp_stdio.rs b/tests/mcp_stdio.rs index 763f8812..607e14fa 100644 --- a/tests/mcp_stdio.rs +++ b/tests/mcp_stdio.rs @@ -345,10 +345,14 @@ fn mcp_stdio_catalog_lists_the_overlap_and_reach_tools() { "eb_recipe_templates", "eb_recipe_hierarchy", "eb_repro_summary", + "eb_ledger_append", + "eb_ledger_head", + "eb_ledger_show", + "eb_ledger_verify", ] { assert!(names.contains(&expected), "missing {expected}: {names:?}"); } - assert_eq!(names.len(), 51, "{names:?}"); + assert_eq!(names.len(), 55, "{names:?}"); } fn software_layer_prs() -> Vec { diff --git a/tests/mcp_v1.rs b/tests/mcp_v1.rs index d6e8cf77..5d452c8f 100644 --- a/tests/mcp_v1.rs +++ b/tests/mcp_v1.rs @@ -169,10 +169,14 @@ fn mcp_catalog_matches_the_version_one_workflows() { "eb_component_reach", "eb_easyblock_reach", "eb_repro_summary", + "eb_ledger_append", + "eb_ledger_head", + "eb_ledger_show", + "eb_ledger_verify", ] { assert!(names.contains(&expected), "missing {expected}: {names:?}"); } - assert_eq!(names.len(), 51, "unexpected MCP tools: {names:?}"); + assert_eq!(names.len(), 55, "unexpected MCP tools: {names:?}"); let inspect = response["result"]["tools"] .as_array() @@ -2565,6 +2569,10 @@ fn mcp_schemas_cover_cli_flags() { ("eb_stack_bot_classify", "stack bot-classify"), ("eb_package_latest", "package latest"), ("eb_repro_summary", "repro summary"), + ("eb_ledger_append", "ledger append"), + ("eb_ledger_head", "ledger head"), + ("eb_ledger_show", "ledger show"), + ("eb_ledger_verify", "ledger verify"), ("eb_component_reach", "component-reach"), ("eb_target_list", "target list"), ("eb_target_doctor", "target doctor"), @@ -3836,6 +3844,10 @@ fn every_cli_subcommand_has_an_mcp_tool() { &["campaign", "finding", "resolve"], ), ("eb_repro_summary", &["repro", "summary"]), + ("eb_ledger_append", &["ledger", "append"]), + ("eb_ledger_head", &["ledger", "head"]), + ("eb_ledger_show", &["ledger", "show"]), + ("eb_ledger_verify", &["ledger", "verify"]), ("eb_pr_overlap", &["pr-overlap"]), ("eb_rebuild_set", &["rebuild-set"]), ("eb_component_reach", &["component-reach"]), diff --git a/tests/public_metadata.rs b/tests/public_metadata.rs index 9ec1fdaf..1b6b150c 100644 --- a/tests/public_metadata.rs +++ b/tests/public_metadata.rs @@ -112,10 +112,48 @@ fn generated_documentation_builds_stay_out_of_source_packages() { #[test] fn local_agent_state_stays_out_of_public_commits() { let gitignore = std::fs::read_to_string(repo().join(".gitignore")).expect("read .gitignore"); - for path in [".claude/", ".codex/"] { - assert!( - gitignore.lines().any(|line| line == path), - ".gitignore must exclude {path}" + assert!(gitignore.lines().any(|line| line == ".*/")); + assert!(gitignore.lines().any(|line| line == "!/.github/")); + let fixture = tempfile::tempdir().expect("temporary repository"); + std::fs::write(fixture.path().join(".gitignore"), gitignore).expect("write ignore policy"); + let init = Command::new("git") + .args(["init", "--quiet"]) + .arg(fixture.path()) + .output() + .expect("initialize temporary repository"); + assert!( + init.status.success(), + "{}", + String::from_utf8_lossy(&init.stderr) + ); + for (path, ignored) in [ + (".working-state/marker", true), + (".cache/marker", true), + ("nested/.working-state/marker", true), + (".github/.working-state/marker", true), + (".github/workflows/check.yml", false), + ] { + let file = fixture.path().join(path); + std::fs::create_dir_all(file.parent().unwrap()).expect("create fixture directory"); + std::fs::write(file, "fixture").expect("write fixture file"); + let result = Command::new("git") + .arg("-C") + .arg(fixture.path()) + .args([ + "-c", + "core.excludesFile=/dev/null", + "check-ignore", + "--no-index", + "--", + path, + ]) + .output() + .expect("check ignore behavior"); + assert_eq!( + result.status.code(), + Some(if ignored { 0 } else { 1 }), + "unexpected ignore behavior for {path}: {}", + String::from_utf8_lossy(&result.stderr) ); } } @@ -586,7 +624,7 @@ fn discovery_metadata_uses_real_index_terms() { for needle in [ "llms.txt", "llms-full.txt", - "AGENTS.md", + "CONTRIBUTING.md", "server.json", "skills/", ] { @@ -597,7 +635,7 @@ fn discovery_metadata_uses_real_index_terms() { for needle in [ "llms.txt", "llms-full.txt", - "AGENTS.md", + "CONTRIBUTING.md", "server.json", "skills/", "reference/mcp.org", diff --git a/tests/stack_shard.rs b/tests/stack_shard.rs index 799a7a52..3ce22c39 100644 --- a/tests/stack_shard.rs +++ b/tests/stack_shard.rs @@ -333,3 +333,248 @@ fn the_mcp_tool_writes_the_same_files_as_the_command() { assert_eq!(read(cli.join(&name)), read(mcp.join(&name)), "{name}"); } } + +/// easybuilders/easybuild-easyconfigs commit that adds +/// `petsc4py-3.25.0-foss-2026.1.eb`. +/// +const PETSC4PY_COMMIT: &str = "0d6b80c34f3553774e93162cf89088e2423ecf89"; + +/// `.eb` files under `easybuild/easyconfigs/` in [`PETSC4PY_COMMIT`]. +/// `find easybuild/easyconfigs -name '*.eb' | wc -l` on the codeload +/// archive, 2026-11-04. +const EASYCONFIGS_IN_PETSC4PY_COMMIT: usize = 11897; + +/// The fetched fixture is at least this many lines. The lists that built +/// one dependency in two shards had about 30 lines EasyBuild fetches. +const MIN_FETCHED_LIST_LINES: usize = 30; + +/// Non-comment lines in `tests/stack_shard_fixtures/fetched.list`. +const FETCHED_LIST_LINES: usize = 34; + +/// `--shards` from the how-to. This list has at least that many groups. +const SHARDS_REQUESTED: &str = "4"; + +/// Prelude size of `fetched.list` over [`PETSC4PY_COMMIT`] with petsc4py +/// removed from the robot path. Counted 2026-11-04. +const FETCHED_LIST_PRELUDE: &str = "173"; + +/// Stamp written into `plan.json` so two runs match byte for byte. +const FETCHED_GENERATED_AT: &str = "2026-11-04T00:00:00Z"; + +/// `EB_STACK_COMMIT_MIRROR`, when set, is a local copy of the codeload +/// `.tar.gz` for [`PETSC4PY_COMMIT`]. The shard command still downloads +/// that URL into `--commit-cache`. +const COMMIT_MIRROR_ENV: &str = "EB_STACK_COMMIT_MIRROR"; + +/// Path of petsc4py inside the commit. The commit message adds this file. +const PETSC4PY_RELATIVE: &str = "p/petsc4py/petsc4py-3.25.0-foss-2026.1.eb"; + +/// Module name `stack shard` records for `PETSc-3.25.0-foss-2026.1.eb`. +const PETSC_MODULE: &str = "PETSc/3.25.0-foss-2026.1"; + +fn commit_archive() -> Vec { + if let Ok(path) = std::env::var(COMMIT_MIRROR_ENV) { + return std::fs::read(&path).unwrap_or_else(|error| panic!("{path}: {error}")); + } + let url = format!( + "https://codeload.github.com/easybuilders/easybuild-easyconfigs/tar.gz/{PETSC4PY_COMMIT}" + ); + use eb_stack::pr_overlap::HttpGet; + let response = eb_stack::pr_overlap::UreqGet + .get(&url) + .unwrap_or_else(|error| panic!("{url}: {error}")); + assert_eq!(response.status, 200, "{url}"); + response.body +} + +fn fetched_lines() -> Vec { + let text = std::fs::read_to_string(fixture("fetched.list")).expect("read fetched.list"); + text.lines() + .map(str::trim) + .filter(|line| !line.is_empty() && !line.starts_with('#')) + .map(str::to_string) + .collect() +} + +/// Plan `fetched.list` twice against the public commit archive. +/// +/// The robot path is that commit with petsc4py removed, so the file is +/// only in the tree `--commit-cache` downloads. petsc4py depends on PETSc, +/// and the PETSc line is fetched from the same commit. +#[test] +fn a_fetched_petsc_line_stays_with_petsc4py_and_the_plan_is_stable() { + let lines = fetched_lines(); + assert!( + lines.len() >= MIN_FETCHED_LIST_LINES, + "{} lines, want at least {MIN_FETCHED_LIST_LINES}", + lines.len() + ); + assert_eq!(lines.len(), FETCHED_LIST_LINES); + assert!(lines.iter().any(|line| !line.contains("--"))); + assert!(lines.iter().any(|line| line.contains("--from-commit="))); + assert!(lines.iter().any(|line| { + line.contains("--include-easyblocks-from-commit=") && !line.contains("--from-commit=") + })); + assert!(lines.iter().any(|line| { + line.starts_with("PETSc-3.25.0-foss-2026.1.eb ") && line.contains(PETSC4PY_COMMIT) + })); + assert!(lines + .iter() + .any(|line| line.starts_with("petsc4py-3.25.0-foss-2026.1.eb "))); + + let tmp = tempfile::tempdir().unwrap(); + let archive = commit_archive(); + let tree = tmp.path().join("tree"); + let extracted = eb_stack::shard_trees::extract_easyconfigs(&archive, &tree) + .unwrap_or_else(|error| panic!("extract: {error}")); + assert_eq!(extracted, EASYCONFIGS_IN_PETSC4PY_COMMIT); + let robot = tree.join("easybuild/easyconfigs"); + std::fs::remove_file(robot.join(PETSC4PY_RELATIVE)).expect("petsc4py is in the commit"); + let cache = tmp.path().join("cache"); + let summary = format!( + "shards={SHARDS_REQUESTED} prelude={FETCHED_LIST_PRELUDE} lines={FETCHED_LIST_LINES}\n" + ); + + let mut plans = Vec::new(); + for name in ["first", "second"] { + let out = tmp.path().join(name); + let run = Command::new(env!("CARGO_BIN_EXE_eb-stack")) + .env_remove("EASYBUILD_FILTER_DEPS") + .args([ + "stack", + "shard", + "--build-list", + &fixture("fetched.list"), + "--easyconfigs", + robot.to_str().unwrap(), + "--commit-cache", + cache.to_str().unwrap(), + "--shards", + SHARDS_REQUESTED, + "--generated-at", + FETCHED_GENERATED_AT, + "--out-dir", + out.to_str().unwrap(), + ]) + .output() + .expect("run eb-stack"); + assert!( + run.status.success(), + "stdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&run.stdout), + String::from_utf8_lossy(&run.stderr) + ); + assert_eq!(String::from_utf8_lossy(&run.stdout), summary); + assert!( + run.stderr.is_empty(), + "{}", + String::from_utf8_lossy(&run.stderr) + ); + plans.push(out); + } + + let petsc4py = cache + .join(PETSC4PY_COMMIT) + .join("easybuild/easyconfigs") + .join(PETSC4PY_RELATIVE); + assert!( + petsc4py.is_file(), + "commit cache did not download {PETSC4PY_COMMIT}" + ); + + for name in listing(&plans[0]) { + assert_eq!( + std::fs::read(plans[0].join(&name)).unwrap(), + std::fs::read(plans[1].join(&name)).unwrap(), + "{name} differs across runs" + ); + } + + let plan: Value = serde_json::from_str(&read(plans[0].join("plan.json"))).unwrap(); + let mut seen: std::collections::BTreeMap = std::collections::BTreeMap::new(); + for shard in plan["shards"].as_array().unwrap() { + for module in shard["modules"].as_array().unwrap() { + *seen + .entry(module.as_str().unwrap().to_string()) + .or_default() += 1; + } + } + let mut prelude_modules = Vec::new(); + for module in plan["prelude"]["modules"].as_array().unwrap() { + let name = module["module"].as_str().unwrap().to_string(); + prelude_modules.push(name.clone()); + *seen.entry(name).or_default() += 1; + } + let twice: Vec<_> = seen.iter().filter(|(_, n)| **n > 1).collect(); + assert!(twice.is_empty(), "built by two jobs: {twice:?}"); + + let line = |file: &str| { + plan["lines"] + .as_array() + .unwrap() + .iter() + .find(|line| line["easyconfig"] == file) + .unwrap_or_else(|| panic!("{file} not in the plan")) + .clone() + }; + let petsc = line("PETSc-3.25.0-foss-2026.1.eb"); + let py = line("petsc4py-3.25.0-foss-2026.1.eb"); + assert!(py["opaque"].is_null(), "{py}"); + assert!(py["closure"] + .as_array() + .unwrap() + .iter() + .any(|module| module == PETSC_MODULE)); + let together = + petsc["shard"] == py["shard"] || prelude_modules.iter().any(|m| m == PETSC_MODULE); + assert!(together, "PETSc {petsc} petsc4py {py}"); + let shard_file = plans[0].join(format!("{}.list", py["shard"].as_str().unwrap())); + let shard_text = read(shard_file); + let petsc_at = shard_text + .find("PETSc-3.25.0-foss-2026.1.eb ") + .expect("PETSc line in the petsc4py shard"); + let py_at = shard_text + .find("petsc4py-3.25.0-foss-2026.1.eb ") + .expect("petsc4py line"); + assert!(petsc_at < py_at, "{shard_text}"); +} + +#[test] +fn line_dependency_filters_are_refused_before_writing_a_plan() { + for option in [ + "--filter-deps=zlib", + "--filter-deps zlib", + "--filter-dep=zlib", + "--filter-d zlib", + ] { + for source in ["", " --from-commit=0123abcd"] { + let tmp = tempfile::tempdir().unwrap(); + let list = tmp.path().join("build.list"); + let out = tmp.path().join("out"); + std::fs::write(&list, format!("# list\nAlpha.eb {option}{source}\n")).unwrap(); + let run = Command::new(env!("CARGO_BIN_EXE_eb-stack")) + .env_remove("EASYBUILD_FILTER_DEPS") + .args(["stack", "shard", "--shards", "4", "--build-list"]) + .arg(&list) + .arg("--easyconfigs") + .arg(fixture("easyconfigs")) + .arg("--out-dir") + .arg(&out) + .output() + .unwrap(); + let stderr = String::from_utf8_lossy(&run.stderr); + assert!(!run.status.success(), "{option}{source}: {stderr}"); + assert!(stderr.contains("line 2: Alpha.eb"), "{stderr}"); + assert!(stderr.contains("dependency-filter override"), "{stderr}"); + assert!(!out.exists(), "a refused list wrote {}", out.display()); + } + } +} + +#[test] +fn other_line_filters_do_not_count_as_dependency_overrides() { + for option in ["--filter-ecs=example", "--filter-env-vars=TMP"] { + let line = eb_stack::shard::parse_lines(&format!("Alpha.eb {option}")); + assert_eq!(line[0].filter_deps_option(), None, "{option}"); + } +} diff --git a/tests/stack_shard_fixtures/fetched.list b/tests/stack_shard_fixtures/fetched.list new file mode 100644 index 00000000..a60e8c67 --- /dev/null +++ b/tests/stack_shard_fixtures/fetched.list @@ -0,0 +1,36 @@ +# easybuilders/easybuild-easyconfigs 0d6b80c34f3553774e93162cf89088e2423ecf89 +# easybuilders/easybuild-easyblocks 9607addb10e02da5e3f44916a0be6222f9795287 (easybuild-easyblocks-v5.4.1) +PETSc-3.25.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +petsc4py-3.25.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +OpenFOAM-14-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +ParaView-6.1.1-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +Hypre-3.1.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +SuperLU_DIST-9.2.1-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +h5py-3.16.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +AmberTools-26.1-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +Wannier90-3.1.0-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +RDKit-2026.03.4-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +scanpy-1.12.1-foss-2026.1.eb --from-commit=0d6b80c34f3553774e93162cf89088e2423ecf89 +phonopy-4.5.0-foss-2026.1.eb --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +PLUMED-2.10.0-foss-2026.1.eb --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +PyFR-3.1-foss-2026.1.eb --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +pymatgen-2026.5.4-foss-2026.1.eb --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +MDAnalysis-2.10.0-foss-2026.1.eb --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +MDTraj-1.11.1-foss-2026.1.eb --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +numba-0.65.1-foss-2026.1.eb --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +netcdf4-python-1.7.4-foss-2026.1.eb --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +geopandas-1.1.4-foss-2026.1.eb --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +GDAL-3.13.0-foss-2026.1.eb --include-easyblocks-from-commit=9607addb10e02da5e3f44916a0be6222f9795287 +aiida-core-2.9.2-foss-2026.1.eb +anndata-0.13.3.post0-foss-2026.1.eb +Armadillo-15.2.7-foss-2026.1.eb +arpack-ng-3.9.1-foss-2026.1.eb +HPL-2.3-foss-2026.1.eb +HeFFTe-2.4.1-foss-2026.1.eb +Kokkos-Kernels-5.1.1-foss-2026.1.eb +OpenMM-8.5.2-foss-2026.1.eb +Fiona-1.10.1-foss-2026.1.eb +Cartopy-0.25.0-foss-2026.1.eb +cooler-0.10.4-foss-2026.1.eb +CheMPS2-1.8.12-foss-2026.1.eb +Zoltan-3.901-foss-2026.1.eb