Repository navigation
feat(shard): resolve from-commit and from-pr lines against their trees - #22
Conversation
`ledger append`, `ledger head`, `ledger show`, and `ledger verify` keep one JSON Lines file per stack. Each record is written as canonical JSON (keys sorted, no whitespace, integers only) with `index` and `id` filled in, and its leaf is rfc9162::leaf_hash over the line. Heads are signed with Ed25519 under the domain eb-stack-ledger-v1; the key is a PEM path on the command line and is refused inside the log's directory. `verify --record N --head HEAD` checks an inclusion proof and `verify --from HEAD1 --to HEAD2` a consistency proof. rfc9162 is a git dependency on the deedar branch until it is published.
stack shard resolves a build-list line that carries --from-commit or --from-pr against that commit's or pull request's easyconfigs first, then the robot path, the order EasyBuild uses. Trees come from --commit-tree COMMIT=DIR, --pr-tree N=DIR, or --commit-cache DIR, which downloads a missing commit from codeload.github.com (its .eb files only) and a pull request's changed easyconfigs at its head. A shared module that is not installed goes to the prelude as for any other line; one only a fetched tree provides keeps the line's source options there. A fetched line that still cannot be resolved stays opaque, with the reason in plan.json and on stderr.
A line whose dependency only another line's commit tree provides resolves through that line: the module is installed by that line when it builds, so the lines share a shard and that line builds first. Each fetched line's own easyconfig sits under every universe at the lowest precedence.
Without a tree for its --from-commit, the petsc4py line was planned opaque, without a closure, and the planner warned about it on stderr. The fixture reads that commit from a cache, so PETSc and petsc4py share a shard. A cache download that never gets an HTTP status exits 1 and writes nothing.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
The extract_easyconfigs comment puts `<repo>-<sha>/` in a backtick span on the same line as `dest`. The docs postprocessor skips a span that contains angle brackets, so that closing backtick is paired with the next span and Sphinx -W reports an inline literal that never closes. The archive prefix is now a double-backtick literal, which the postprocessor leaves alone.
EasyBuild's check_checksums_for treats postinstallpatches as artifacts that need a digest, and a component dict may spell that list post_install_patches. Bundle component checksum tests accept a recipe when every source, patch, and postinstall patch has a digest, and packaging_gate rejects it for missing checksums when one is absent.
…b/shard-commit-trees-v2
The shard howto and the CLI reference showed a codeload archive address whose SHA was empty or the literal SHA. Lychee requests those addresses and they 404. The examples now give the host and the path without a single https URL, and the shell example still downloads that commit.
79f7f8a to
07787c1
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 07787c1. Configure here.
| } | ||
| let url = raw_url(repo, &pull.head.sha, &file.filename); | ||
| let text = get_raw(http, &url)? | ||
| .ok_or_else(|| anyhow::anyhow!("pull request {number}: {url}: HTTP 404"))?; |
There was a problem hiding this comment.
Fork PR files fetch from wrong host
High Severity
--from-pr trees load file text from raw.githubusercontent.com on the base repo plus the pull head SHA. ListedHead keeps only sha, so a fork commit is requested from easybuilders/easybuild-easyconfigs and typically 404s. Those lines then plan as opaque and never share a shard with the modules they need.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 07787c1. Configure here.
| let entry = cache.join(format!("pr-{number}-{}", head.to_ascii_lowercase())); | ||
| if entry.is_dir() { | ||
| return Ok(entry); | ||
| } |
There was a problem hiding this comment.
PR cache unused until live fetch
Medium Severity
pr_tree always calls pull_head_easyconfigs (API plus every raw file) before it looks at --commit-cache. A complete pr-N-HEAD directory is ignored unless that live fetch succeeds, so an outage or rate limit makes --from-pr lines opaque even when the tree is already on disk. Commit trees check the cache first.
Reviewed by Cursor Bugbot for commit 07787c1. Configure here.
| let expected = sources + patches + postinstall; | ||
| let Ok(checksums) = value_list_as_slice(fields.get("checksums")) else { | ||
| return false; | ||
| }; |
There was a problem hiding this comment.
Component checksums reject string sources
Medium Severity
component_checksums_complete counts sources, patches, and checksums only when each value is a list. EasyBuild treats a bare string as one entry. A Bundle component that writes sources or checksums as a string is marked incomplete, so packaging_gate still reports missing checksums for a fully hashed recipe.
Reviewed by Cursor Bugbot for commit 07787c1. Configure here.
What and why
stack shardplanned a--from-commitline with no tree as its own shard.petsc4py-3.25.0-foss-2026.1.ebthen sat apart from the PETSc line it depends on.The fixture list reads easybuild-easyconfigs
0d6b80c34f3553774e93162cf89088e2423ecf89through--commit-cache.PETSc and petsc4py share a shard, and PETSc is first.
A cache download that never receives an HTTP status exits 1 and writes nothing.
Before and after
34 lines. 11
--from-commit, 10--include-easyblocks-from-commitonly, 13 plain.The robot path is that commit with
p/petsc4py/petsc4py-3.25.0-foss-2026.1.ebremoved.Before, no
--commit-cache:PETSc is shard-03. petsc4py is shard-04 and opaque.
After,
--commit-cacheset:PETSc and petsc4py are both shard-03. PETSc is before petsc4py.
No module is in two shards.
A second run with the same
--generated-atwrites the same files.Tests
cargo +stable fmt --all --checkexited 0.cargo +stable clippy --locked --all-targets -- -D warningsexited 0.cargo +stable test --locked --all-targetsexited 0.1529 passed, 0 failed, 10 ignored.
uvx prek run -aexited 0.python3 scripts/gen-llms-txt.py --checkexited 0.The local mirror is the codeload tar.gz of commit
0d6b80c34f3553774e93162cf89088e2423ecf89(8436966 bytes).The shard command still downloads that URL into the cache.
Note
High Risk
Introduces cryptographic logging (Ed25519, Merkle proofs, file locking) and network-backed shard planning with shared caches—both are security- and correctness-sensitive operational paths.
Overview
Adds a per-stack software ledger (
ledger append,head,show,verify): canonical JSON Lines, RFC 9162 Merkle leaves, Ed25519 signed heads undereb-stack-ledger-v1, POSIX writer locks, idempotent retries onbuild.event_id, and inclusion/consistency proofs. The same surface is exposed as four MCP tools; CI runs a dedicated ledger job.stack shardnow resolves--from-commit/--from-prlines like EasyBuild: fetched easyconfig trees via--commit-cache,--commit-tree, and--pr-tree(codeload + GitHub API), places shared dependencies in the prelude with fetch flags preserved, and plans still-unresolved fetched lines as opaque withplan.jsonmetadata and stderr warnings. Transport failures before HTTP status exit 1 without writing outputs.Smaller fixes: packaging gate accepts Bundle recipes whose
componentscarry complete checksums;max_parallelis recognized in recipe lint vocab;.gitignoreignores all hidden dirs except.github/; docs/indexes point at CONTRIBUTING.md instead of AGENTS.md; workspace path resolution handles symlinked roots.Reviewed by Cursor Bugbot for commit 07787c1. Bugbot is set up for automated code reviews on this repo. Configure here.