Repository navigation
Conversation
80cd2ad to
8f6ccc7
Compare
|
@ 8f6ccc7 I still think we should not have the separate armhf shasum file since everyone can cross compile it now. |
|
Agreed, and I am working onusing the head commits timestamp as the mtime so we have one fewer potential mismatches, and it's reusing a datetime we already agree on anyway. |
|
Cross-machine reproduction: the armhf divergence is gone, and what remained was the calendar. @daywalker90 I reproduced your run exactly — dropped your two manifest files into
That run shape explains the earlier "armhf differs across machines" data points: at Which is exactly what the
The day-rollover rerun's manifest header says One more machine is queued: an Apple-Silicon Mac, where the run currently dies in Docker Desktop's virtiofs layer refusing the Consequence, as you suggest: the separate |
|
Edit (2026-10-26): the fetch-noise suppression described in the first half of this comment was pulled off the branch — the shim was too much machinery for the noise it hides. The branch head is back at Two follow-ups on the build hygiene front, both landing on top of the merged-manifest state. The build log got quiet.
Full hashes for anyone cross-checking this head: Incidentally this doubles as a cross-version check — the two routes ran different CI on |
5ff6b25 to
2da5932
Compare
|
Closing the loop on the fetch-noise experiment: the suppression was pulled off the branch ( What it took to work: a The finding worth keeping is in the comment above: the container route and |
|
|
Thanks @daywalker90 and sorry for not following up earlier. I was able to verify your SHA256SUMS file without issue, on both Linux (Ubuntu 26.04) and MacOS (via |
Three changes a static or cross build needs, none of which affects a native dynamic one: - the configurator itself is compiled with CONFIGURATOR_CC. It runs on the build machine, so when cross-compiling it must not be built for the host; its test programs are still compiled with CC and are now run through CONFIGURATOR_WRAPPER (e.g. qemu-aarch64), which makes the five places the build runs its own output explicit instead of relying on the kernel's binfmt registration. - POSTGRES_LDLIBS and POSTGRES_INCLUDE can come from the environment. A static libpq.a needs archives pg_config does not report (-lpgcommon -lpgport -lssl -lcrypto), and the no-pg_config branch used to *assign* these empty, discarding what the caller set -- which silently produced a lightningd with no PostgreSQL support at all.
have_function_sections() links its test program with -o, so without a main() the link dies on "undefined reference to `main'" -- on every platform, not just a fully static one against musl's crt1.o. The probe therefore answered "no" however capable the toolchain was, and HAVE_FUNCTION_SECTIONS has been 0 everywhere: -ffunction-sections and -Wl,--gc-sections were silently off for every build this project shipped. Adding the main() turns them on wherever the toolchain supports them, which is a change to native builds too: smaller binaries through section gc, and nothing else -- the flags do not alter semantics, and warnings are unaffected.
The configurator's loop sets every variable config.vars mentions, so
afterwards a stored empty value is indistinguishable from an environment
setting, and `: ${POSTGRES_LDLIBS=...}` skipped the pg_config probe. A
--reconfigure whose stored value was empty therefore built a lightningd
with no PostgreSQL support at all, without a word about it.
Whether the value came from the environment is now recorded before
config.vars is read; anything else is probed again, as a first run would.
The build runs some of its own output, which a cross build cannot do: - cdump-enumstr is compiled with CC_FOR_BUILD and taken out of ALL_PROGRAMS: it is run to generate source, so it must be a build-machine program, while libccan.a is host code. - tools/headerversions and plugins/sql --print-docs run through CONFIGURATOR_WRAPPER. The sql one had its failure masked by a pipe to jq, which shipped a lightning-sql.7 with no TABLES section. - external/Makefile passes BUILD/MAKE_HOST so libwally and libbacktrace configure as cross rather than native, and libbacktrace's generated backtrace-supported.h is declared as a dependency of the objects -- a -j race that only showed up where gcc did not happen to provide its own copy. - cln-rpc honours RUST_TARGET_DIR so the Rust artifacts land per target. That last dependency is an *order-only* prerequisite, and the header is deliberately not a member of EXTERNAL_HEADERS, which is subtler than it looks. EXTERNAL_HEADERS depends on the phony submodcheck, so every member of it is permanently out of date; that is harmless for headers we merely check out, because a target with no recipe and no missing file is not propagated as updated. A generated header behaves differently at both ends of that rule: as a member with no recipe it is a nonexistent file, which make imagines to have been updated and so invalidates every object; as a member with a recipe it re-runs libbacktrace's configure on every single make. Order-only says what is actually meant -- it must exist before anything compiles, and its timestamp is not a reason to recompile. A grouped target then says the archive and the header come from one recipe, and the header joins TESTPACK_EXTRAS for the same reason $(EXTERNAL_LIBS) already had to: the testpack prunes external/, so otherwise it never reaches the phase that only checks, and make would build it there with a compiler stubbed out to refuse.
…ation The release artifacts come from the same derivation as packages.default, evaluated under pkgsStatic (and pkgsCross for the arm rows), rather than from a second release-only expression: the thing tested hardest each release is then the thing Nix users install. release = true switches on what the reprobuild contract needs and nothing the ordinary build wants: version and mtime from the driver, one `make install` (install's own prerequisite graph already covers everything shipped), an offline cargo vendor dir, --prefix=/usr into a DESTDIR, strip --strip-all with an in-derivation assertion that no shipped ELF keeps .symtab/.debug_* or loses its build-id, and dontPatchELF so fixupPhase cannot rewrite signed bytes. static-pie.nix carries the link recipe, including two gaps nixpkgs does not cover: rustc still links crt-static musl targets non-PIE and self-contained, so every row rewrites its link line; and gcc cannot static-pie on armv7 at all, so that row links through an explicit-crt wrapper. Both are needed to keep ASLR on every shipped executable. Three changes stop the release bytes tracking the source tree: a constant unpacked source directory, a remap for the cargo vendor dir, and a pinned NIX_OUTPATH_USED_AS_RANDOM_SEED -- without the last one, nixpkgs seeds every C compile from $out and --build-id=sha1 hashes the unstripped output, so a tracked file that is never compiled moved every build-id.
…y consume nix/release-rows is the matrix as data: arch, package-set path, Docker platform. The flake generates cln-release-static-<arch> and vls-release-static-<arch> from it and the driver reads it with a plain `while read`, so there is one source of truth and no tooling on either side. Two guards, both only on the release path: - the rows record the nixpkgs revision the release is certified against. A `nix flake check` entry compares it with flake.lock and the release derivations refuse to evaluate if they differ, so a contributor's `nix flake update` cannot move gcc, rustc and openssl under three signed tarballs without a word. The error names the three ways out. - each row commits a short input manifest -- direct inputs with versions, toolchain versions, build flags, the release environment -- checked at eval. A distribution-motivated change that reaches the release then shows up as a diff a reviewer who does not read Nix can read.
Every signed artifact is a Nix output, so `nix build --rebuild` covers the bytes that ship. The zip was the last one assembled on the host, out of `git ls-files` of the *working tree* -- which is not the commit a manifest names, so a co-signer standing on a different branch rebuilt a different zip. vls.nix builds VLS's remote_hsmd_socket over the same rows. It is not rustPlatform.buildRustPackage: that would resolve rustPlatform to the row's toolchain and compile a musl rustc from source inside the inputs of a signed artifact. The crate closure is one fixed-output vendor derivation, and a git shim answers the one `git describe` build.rs asks for, since a fixed-output source fetch has no .git and every shipped binary would otherwise report git-desc-failed.
The manifests describe the build's inputs -- which package versions, which flags -- and said nothing about the files that decide what is done with them. So an edit to static-pie.nix could change how every shipped binary is linked, move the output bytes, and leave the manifest byte-identical: guard 2 would not fire, and the promise that nothing reaches a signed release without a reviewable diff did not hold where the most consequential edits happen. It is not an oversight in the flags section but a structural gap. The compiler wrapper reaches the build through configureFlagsArray inside the build phase and the Rust linker through an exported variable; neither is in the configureFlags attribute the flags section is generated from, and neither could be without duplicating the recipe into its own description. So name the recipe by content hash instead: default.nix, static-pie.nix and flake-module.nix for a CLN row, vls.nix and static-pie.nix for a VLS one. Any edit to them now moves the manifest, trips the guard, and has to go through certify like a nixpkgs bump.
Replaces build-release.sh and repro-build.sh. The host needs docker, coreutils and python3 (3.10+): every nix invocation runs inside a digest-pinned nixos/nix container against a store under XDG_CACHE_HOME, so no host nix.conf -- sandbox, substituters, filter-syscalls -- is an input to a signed artifact. --host-nix is an opt-in for whoever has Nix, and both routes produce identical bytes. Python rather than shell, importing only the standard library and run through plain python3 rather than uv: a co-signer reproducing a release should need an interpreter, not a package manager. The verbs that emit signed bytes were held to that -- the seven release derivations instantiate to the same .drv paths as the shell driver produced, and both manifests come out byte-identical -- so the rewrite is a change of language, not of output. It also drops the hand-rolled OCI JSON parsing the shell version needed to avoid a jq dependency; `json` is in the standard library. Verbs: fetch, build, vls, zip, docker, sign, verify, certify, publish, disclose, accept, all. `all` stops at sign; nothing pushes without publish or disclose, and certify is deliberately outside `all` so a release day cannot quietly re-certify a toolchain nobody examined. What the driver refuses to do is most of its value: - build a release from a dirty tree: nix caches the worktree it fetched, so the artifacts could be labelled with a commit they were not built from. - build unsandboxed. Under default docker nix cannot set up the sandbox and, with sandbox-fallback at its default, builds without one *silently* -- the build directory then ends up baked into every locally compiled Rust artifact, and no such build reproduces any other. - sign a manifest whose files came from different trees, or publish artifacts that did not come from the tag being published. - start a build that would end in a signature it cannot make: the signing key is exercised on a throwaway file first. Manifests follow decisions 19 and 45: SHA256SUMS-<ver> for what a co-signer is expected to reproduce and SHA256SUMS-<ver>-armhf for the row that does not reproduce across machines, each with a header recording rev, narHash, submodules, mtime and the certified nixpkgs revision -- so a hash is never reported without the tree it came from. verify takes the rev and the mtime from the manifest it is handed, not from the verifier's checkout.
check.sh asserts the strip policy on the shipped bytes -- stripped, static-pie, build-id present, no embedded library search path -- and is a precondition of sign, not advice. portability.sh runs the distro matrix (three Ubuntus and Fedora) with smoke.sh inside each container: --version, a regtest start and a getmanifest handshake with all 27 plugins. `all` runs the newest Ubuntu only; the full matrix belongs to the acceptance suite. two-tree-probe.sh builds one row from two trees differing by a single tracked, unshipped, uncompiled file and requires identical bytes. It replaces a "zero /nix/store references" invariant, which was both unreachable (lightningd keeps OpenSSL's store paths deliberately) and insufficient: a variant whose bytes tracked the tree carried no offending string at all. It runs per row, because the defect it last caught was armhf-only. vls-regtest.sh runs the CLN-VLS pair on regtest against a test-only vlsd, since remote_hsmd_socket only listens for a signer to connect and proves nothing alone. fetch-bitcoin.sh downloads bitcoin-cli and counts good signatures from the vendored keys rather than trusting gpg's exit status. pins carries the container digests and the Bitcoin Core pin; publish.vars the public bucket, its endpoint and the prefix the source archive is served from.
The image consumes the static tarball for its platform instead of compiling anything: debian:trixie-slim pinned by index digest, the tarball entering through a mount so it is never stored in a layer, bitcoin-cli fetched and GPG-verified on the host beforehand. The -vls overlay also gets the environment its signer cannot start without. remote_hsmd_socket reads BITCOIND_RPC_URL, VLS_NETWORK and VLS_CLN_VERSION and panics if any is missing, and lightningd then waits forever on an hsmd that never answers, with no error of its own. The network follows lightningd's; the bitcoind URL cannot be guessed, so it is refused with an explanation instead.
The Ubuntu and Fedora builder images, the Alpine/apk static builder, their pin lists and hooks, the VLS pseudo-distro and both old driver scripts all go: the matrix is one static build per architecture now, produced by a Nix derivation, so there is no image of ours to build and no per-distribution directory to keep in step. Keeping them as a fallback was considered and rejected -- they produce Ubuntu tarballs, a matrix that no longer exists, so invoking them would ship a different set of artifacts than the one promised. Any historical release is reproduced from its own tag, where these files still are. Leftovers from the same route, all found in review: - contrib/cl-repro.sh built the cl-repro-jammy/noble/resolute builder images from Dockerfiles that no longer exist. Nothing referenced it. - rust-toolchain.toml only ever affected contributors: its own header says it is not what the release is built with, since the toolchain comes from the certified nixpkgs revision. What it did do was install a 1.85 toolchain plus six cross targets nobody building CLN needs, under the `minimal` profile, so a rustup user lost rustfmt and never had rust-analyzer. A pin that helps nothing and degrades every contributor's setup is worse than no pin. - .dockerignore still explained itself in terms of builder images that COPY their pins and rust-toolchain.toml, and kept both visible for them. The runtime Dockerfile's only main-context input is the entrypoint; the release tarball, bitcoin-cli and VLS all arrive as named build contexts. - release.yml had a `gate` job whose entire purpose was to join several parallel builds. One build is left, so it joined nothing. Changelog-Changed: release binaries are now one static build per architecture (amd64, arm64, armhf) instead of non-static builds per distribution (Ubuntu, Fedora).
The release binaries and the images are produced and signed locally by tools/reprobuild, from one static build per architecture, and published from the release captain's machine. So release-build.yml and release-publish.yml go with the scripts they called, repro.yml with the builder images it rebuilt nightly, and docker-build.yml/docker-publish.yml with the CI image path -- docker-publish was also the job that pushed by digest and applied `latest` to any non-rc tag, neither of which a release under embargo can afford. "Release 🚀" keeps the tag check and PyPI. tools/reprobuild disclose refuses to push a tag while it is enabled, because a tag push would publish the pyln modules on its own schedule rather than the captain's.
The reproducible-builds page loses the per-distribution builder images and gains the driver: what the artifacts are, the two guards on what the build consumes, per-release manifest names with the identity header, and co-signing as a single `verify` that takes the commit and the timestamp from the manifest it was handed -- including how to fetch that commit when it is only on the private mirror. The release checklist becomes one sequence for rc, final and point releases: all, three signatures, publish at T0, disclose when the source may go out. The docker-images page points at the driver instead of the builder images. Both note what the release does not currently contain: no VLS signer binary and no -vls image, because Core Lightning offers channel splicing by default and refuses a signer that cannot sign splices, which no released VLS does yet.
…t up front `all` preflights the signing key before it builds anything, so that an hour of building cannot end on a key nobody can reach. For a key on a smartcard that has a cost the preflight was not written for: it means being at the machine at the start, an hour before there is anything to sign, and again at the end. --wait-for-key inverts that trade. `all` skips the up-front check, builds unattended, and waits when it reaches sign -- 30 minutes by default. Absence stops being an error and becomes something to wait for; what is given up, and only under the flag, is failing early on a key that is broken rather than merely absent. It polls the key rather than reading a keypress. Runs this long are started detached, where there is no stdin to read from but there is a log to tail, and a poll answers "is the key ready" where a keypress only claims it is. Each poll is a real signature of a throwaway file, which is also what unlocks a smartcard, so success is proof rather than inference. `sign` now checks the tree before the key: the tree check is instant, and summoning someone to unlock a smartcard only to refuse the run for a dirty tree is the wrong order.
The release derivation used --prefix=/usr, so the tarballs unpacked over `/` and the docs had to tell people to do that. A release tarball is not the place to move a project's install directory: every earlier release put its binaries in /usr/local, `make install` still does, and someone upgrading would have ended up with two copies in two prefixes. So back to --prefix=/usr/local, and the documented command is the one it always was, `-C /usr/local --strip-components=2`. The harnesses stop hardcoding the prefix rather than swapping one spelling for another: check.sh and vls-regtest.sh locate lightningd in the extracted tree, and smoke.sh takes it from PATH and derives the plugin directory from it. Which prefix the release installs under is the derivation's business, and three scripts that each had to know it were three things to keep in step.
Binaries and source now go out together, so the two-verb split goes: `publish`
uploads the artifacts and the source zip, reads them back and checks them
against the signed manifest, pushes the images, pushes the tag and creates the
GitHub release. Least-reversible last, and nothing left that can half-publish a
release. The runbook loses the T0 / T0+14 structure with it.
Stripping stays, re-justified: it was argued for as a security property that
only held while source lagged binaries. The real reasons are that it keeps the
tarballs small, matches every earlier release, and costs nothing because the
build-id survives for symbolising crashes against the unstripped twin.
The comments are rewritten throughout to stand on their own. They referred to
ticket numbers and decision dates that a reader of this diff has no access to,
and an earlier pass at removing them had left duplicated sentences
(default.nix's header said the same thing twice) and clauses that stopped
mid-thought ("rustc's static-pie gap, which is"). static-pie.nix now opens by
saying what it is -- the shared half of one recipe, imported by both CLN and
VLS, holding workarounds for two upstream gaps -- which is the question it kept
being asked.
publish.vars loses a provisioning narrative that named an internal ticket, the
cloud project, IAM role details and the password-manager vault and item. None
of it is needed to run the release, and a public repository is the wrong place
for it.
Two bugs fell out of writing the above:
- publish_vars() and pins() matched keys with `[A-Z_]+`, which does not match
a digit, so REPRO_S3_ENDPOINT, REPRO_S3_BUCKET and REPRO_S3_PROVIDER were
silently absent and `publish` would have died on a KeyError at the upload.
The shell driver looked each key up by name and did not have this.
- `verify`'s advice on a missing tag pointed at a private mirror that is not
part of this flow.
One more review step survives the collapse: the GitHub release page is
created as a draft, so the captain has the hand check the flow has always
had before anyone reads it. Everything else is public by then.
The flake inputs were 11 to 15 months old; all six move to today. nixpkgs goes from 2025-10-25 to 2026-09-29, which carries the toolchain with it: gcc 14.3.0 -> 16.2.0 rustc 1.89.0 -> 1.98.1 python 3.13.8 -> 3.14.7 The container pins move too. The Nix builder goes to 2.35.2, still by index digest. skopeo goes to v1.22.3 rather than the v1.24.1 that exists as a GitHub release: quay.io/skopeo/stable does not publish 1.24.x, and the container track is what this pin names. One thing the bump broke, and it is the guard working: the 32-bit ARM row carried `hardeningDisable = [ "pie" ]` on libpq, because postgres could not configure there with nixpkgs' pie hardening and pkgsStatic's -static together. `pie` is no longer one of nixpkgs' hardening flags, so naming it now aborts evaluation rather than being ignored. Dropped rather than translated to a neighbouring flag that is not obviously the same thing; whether the original problem is back is a question only an armhf build answers, and that build is still running. Guard 1 refused the first build after the bump, before anything was compiled, which is exactly what it is for: the certified revision in nix/release-rows no longer matched flake.lock. Re-pinned and all six manifests regenerated through `certify --no-verify`. The bump to current nixpkgs failed to evaluate with "expected a set but found a function", always on some package's `src`. The cause was our own overlay: it replaced `fetchurl` with a plain lambda, and `fetchurl` is no longer a bare function. It is a set carrying __functor, override, extendDrvArgs and friends, so flattening it broke every nixpkgs call site that overrides it -- which is most `src` attributes, via packages we never name. The overlay existed because crates.io used to reject a User-Agent starting with "curl/". Removed rather than rewritten against the new interface, because it is no longer needed: a full amd64 release build fetches everything it needs without it. If that regresses, the way back is `extendDrvArgs`, not another lambda. flake.nix joins the recipe hashes in the input manifests while we are here. Its overlays reach every package in every row, so an edit there can move the shipped bytes -- which is not hypothetical, since this is exactly what the fetchurl overlay did -- and until now guard 2 could not see it. GNU autogen goes too, as an unused build input rather than as part of the fix: it was the *first* thing to fail (through guile 2.2.7 and its boehm-gc, broken under pkgsStatic), but removing it only moved the failure along. Nothing in this build invokes autogen -- libwally's tools/autogen.sh is a shell script that runs autoreconf -- so it was dragging a compiler and a garbage collector into the closure for nothing. Verified on the amd64 row: builds clean under gcc 16.2.0, rustc 1.98.1 and python 3.14.7; 40 ELFs all static-pie, stripped, with build-id; the tarball installs under /usr/local; and the portability smoke test passes on ubuntu:26.04 with 27 of 27 plugins answering getmanifest.
CLN's configure defaults COPTFLAGS to -Og, which is gcc's "optimise without getting in the debugger's way" -- roughly -O1. Nothing overrode it, so every C file in every release this project has shipped was compiled that way; the Rust half was fine, since it builds with RUST_PROFILE=release. Measured before changing it, three interleaved pairs of runs on one tree differing only in this flag, medians of three: rpc_listpays_c8 +17.0% ranges separate grpc_getinfo_c8 +17.0% ranges separate grpc_listpeers_c1 +8.1% ranges separate rpc_listinvoices_b_c8 +6.7% ranges separate grpc_listpeers_c8 -3.3% ranges separate Eight further phases were inconclusive -- their per-run ranges overlap -- and the clnrest ones varied 2.75x within a single side, so they say nothing at this sample size. The wins are on list operations, which is where sustained C-side iteration and serialisation happens, and that is the shape one would predict. The one separated regression is small and unexplained; it is not worth 17% twice. RSS is unchanged: lightningd 53028 vs 52960 kB, every plugin within noise. So this is not a size-for-speed trade. Warnings are unaffected. default_cwarnflags only drops -Wno-maybe-uninitialized at -O3, and -Werror is added for debug builds only, so -O2 changes neither. Verified on the built artifact: 40 ELFs all static-pie, stripped, with build-id, and 27 of 27 plugins answering getmanifest on ubuntu:26.04. This moves the shipped bytes, so the reproduction and portability measurements are retaken against it, together with the toolchain bump.
The release used `pkgs.buildPackages.{rustc,cargo}` from each row's
package set. Hydra does not build `pkgsCross.<t>.pkgsStatic`, so on
the arm64 and armhf rows cache.nixos.org had no rustc and every build
compiled rustc from source first (~4.5 min on 24 threads, per row, per
fresh store, and again for every co-signer). It is still true at the
current nixpkgs pin: the 1.98.1 static cross rustc is not cached for
either row, while the plain cross one is.
nix/pkgs/rust-bin.nix instead fetches rustc, cargo and rust-std from
static.rust-lang.org, pinned by the sha256s in the signed
channel-rust-1.98.1.toml -- the same version nixpkgs carries. One host
toolchain serves all three rows and nothing Rust-related is compiled.
nixpkgs bootstraps its own rustc from the same binaries, so this adds no
trust the release did not already extend.
The musl objects rustc links self-contained (libc.a, crt1.o, rcrt1.o,
...) are replaced with the row's own musl, as nixpkgs' static rustc
does, so the C and Rust code link one libc. crtbegin/crtend and
libunwind stay upstream's.
rust-bin.nix joins the recipe hashes, and the input manifests record
each component's sha256. This moves the release bytes, but only the
Rust half: built with the same version and mtime, the amd64 tarball's
33 C binaries are byte-identical to the parent commit's and only the 7
Rust plugins differ. Verified on all three rows: check.sh passes
(static-pie, stripped, build-id, same ARM attributes), portability.sh on
ubuntu:26.04 passes 27/27, and `nix build --rebuild` reproduces all
three.
…argo run `tools/reprobuild all` built each artifact with its own `nix build`: the zip, then amd64, then arm64, then armhf, then --rebuild each of them the same way. The rows do not depend on each other, so nix could overlap them, but was never asked to. The driver now puts every artifact a verb needs into one attribute set and builds it with one `nix build`, reading the output paths back in order. `all` and `verify` build the zip, the rows and (when it ships) VLS together; `build`, `vls` and `zip` still work alone. --rebuild checks all of them in one run too. nix's default `cores = 0` gives every derivation all CPUs, so three rows at once would each run make -j$(nproc). The driver passes --max-jobs <rows+1> --cores <jobs/rows> instead: the rows share the budget, and the +1 lets small derivations (the zip, a wrapper script) run beside them. --cores does not reach the output bytes: xz is pinned to one thread, and the three tarballs came out byte-identical to a serial build of the same commit. plugins/Makefile built the seven Rust plugins with seven `cargo build --bin X` rules. With make -j they started at once and queued on cargo's target-dir lock, so after the shared dependencies each final crate and link ran alone. One grouped rule now builds them all with a single `cargo build --bin a --bin b ...`, prefixed with `+` so cargo takes its job slots from make's jobserver rather than starting nproc rustc processes beside make's own jobs. A developer's `make` rebuilds the same things as before: nothing on a no-op, all seven plugins when a plugin source changes. Measured on 24 threads, host nix, all three rows with their toolchains already in the store: serial driver, one row at a time 10m46s one nix build, rows in parallel 5m37s Running build threads (sampled every second across every nixbld process, rustc codegen threads included) peaked at 26, and were above 24 for 6 of 294 seconds -- when a row's cargo and its C link steps meet. The serial driver never got past an average of 3 busy CPUs. Re-measured after rebasing onto gcc 16.2.0 / rustc 1.98.1 / -O2: --rebuild of all three rows took 6m00s, peak 26, above 24 for 5 of 324 seconds.
…ered plugins The smoke test at the end of `all` took 83-88 s per row, run one row after another: 4m19s for the three. Nearly all of it was smoke.sh's getmanifest handshake, which fed each plugin its request, held stdin open with a fixed `sleep 3`, and did that for 27 plugins in turn -- 81 s of sleeping per distro, and four times that per row in `accept`. Measured, the wait is not needed: every plugin answers in under 0.25 s natively and under 0.9 s under qemu-arm. smoke.sh now starts all of them at once, keeps each one's stdin open only until its reply arrives (polling for "result", 20 s cap), and reports them afterwards in a stable order. portability.sh runs its distro images concurrently -- each container only reads the shared unpacked tree -- and prints them in the listed order; a failed image now shows its FAIL lines first, since they come last in the log and the 40-line head used to cut them off. The driver runs portability.sh for every row at once in `all` and `accept`, printing each row's output whole. The two-tree probe in `accept` stays serial: it builds, and a build already uses every core it is given. Timings, three rows, 24 threads: all, ubuntu:26.04 per row 259 s -> 7 s accept, the full 11-image matrix ~17 min -> 11 s (portability part) Still caught, checked with a tarball carrying three bad plugins on two images: one that never answers, one that answers with an error, and one that exits at once. All three are reported FAIL on both images and the run fails, in 26 s.
The static musl lightningd was 16% slower than a glibc build, most of it musl's malloc and memcpy under sqlite3_expanded_sql in db/db_sqlite3.c. Replace the allocator in every shipped ELF -- the C binaries, the Rust plugins (whose System allocator is malloc) and VLS -- by linking nixpkgs' mimalloc.o on all three link paths in static-pie.nix. The object rather than libmimalloc.a: an archive member is only pulled in for an undefined symbol, so libc.a could satisfy malloc first and the release would ship musl's allocator without any error. mimalloc's version is now listed in each row's input manifest. musl's x86_64 memcpy is byte copies to alignment, `rep movsq`, then a byte-by-byte tail. For the short copies sqlite3_expanded_sql() makes on every database write it is about 5x slower than LLVM libc's, and with mimalloc in place it was most of the remaining gap to a glibc build. Build LLVM libc's memcpy.cpp and memmove.cpp (21.1.8, from nixpkgs' own source) for baseline x86-64 and link them with mimalloc on every link path. memmove has to be replaced too: musl's calls __memcpy_fwd, which would pull in musl's memcpy.o next to ours. clang builds them because the row's gcc produced a memmove 3-4x slower on large copies. The arm rows already get an optimised memcpy from musl and are unchanged. 3-node line benchmark, 3000 payments, 5 interleaved rounds (medians): build payments/s CPU s, all nodes glibc, -O2 (same commit) 614 7.7 musl, mimalloc 575 10.0 musl, mimalloc, LLVM mem 597 8.3
…ce it
The armhf row used nixpkgs' armv7l-hf-multiplatform, which is Debian's
baseline: VFPv3-D16, no NEON. The Rust plugins did not even hold to
that. rustc ignores the row's gcc.fpu, and its armv7 musl target leaves
D16-D31 enabled, so cln-lsps-client, cln-lsps-service, clnrest and
wss-proxy died with SIGILL on a VFPv3-D16 CPU. lightningd, cln-grpc
and the others also carried NEON code from OpenSSL's assembly.
check.sh only printed lightningd's attributes, so none of this failed
a build.
The arm tarballs are for the Raspberry Pi. Every Pi that runs ARMv7
code (Pi 2 and later, Zero 2 W) has NEON and VFPv4 with 32 D
registers, so build armhf for armv7-a with neon-vfpv4: C through a
package set with that gcc.fpu (pkgsCrossRaspberryPi), and Rust with
`-C target-feature=+vfp4,+neon`, taken from the same fpu. The fpu is
listed in the manifests. amd64 and arm64 build recipes are unchanged.
check.sh now enforces the target instead of only printing it:
* armhf: every ELF must be v7, at most VFPv4 and NEONv1, hard-float;
* arm64 and armhf: every ELF is run under the pinned qemu on the
oldest Pi's CPU (cortex-a53, cortex-a7) and must not die by
signal. This catches code the attributes do not declare, like the
D16-D31 use above.
vls-check.sh runs VLS on the same CPUs.
The official release tarballs store usr/local/..., and the documented install is `tar -xf ... -C /usr/local --strip-components=2`. With the leading ./ that command strips only . and usr, so everything lands in /usr/local/local/ and the installed binaries are silently left alone. Drop the ./ and make check.sh reject any member outside usr/local/.
The release manifests carried DIGESTS-<ver> and DIGESTS-<ver>-armhf, the
per-architecture and index digests of the published image, and three people
signed them. Nobody can check that signature: the image has one unpinned,
network-touching input -- the apt-get install of the entrypoint's runtime
dependencies -- so a third party rebuilding later gets different bytes and a
different digest.
The tree already said so in two places. verify's DIGESTS branch was explicitly
informational and never blocked, because "images are optional for a co-signer,
and the layer can drift between sittings". The Dockerfile calls the apt step
"the one unpinned, network-touching input". So the manifest required three
signatures over a hash we had twice written down as unverifiable. ("Bit-identical
across two --no-cache runs" was one machine at one moment, not a third party
later against a moved Debian package set.)
What is attested instead is stronger and one level down: the tarballs are built
from source under Nix, reproduce byte for byte, and carry three signatures -- and
the image is built from those signed bytes.
publish loses a guard with them, so it gets a better one. It used to refuse when
the layout's index digest was not the recorded one, which was really a staleness
check wearing an integrity costume. It now reads the layout's own OCI labels and
refuses when image.version is not this release or image.revision is not the
commit the manifest names -- the same question, answered by something that does
not have to be reproducible to be true.
The digests are still computed and logged for an operator comparing what Docker
Hub ended up with; they are simply not artifacts and not signed.
`SKOPEO_IMAGE=quay.io/skopeo/stable:v1.22.3` looked like a pin and was not one. Quay's version tags float: the tag was re-pushed on 2026-10-02 and again on 2026-10-07. Worse, quay garbage-collects a manifest once no tag points at it, so a digest taken from a floating tag does not merely stop being current -- it stops existing. A digest pinned from v1.22.3 was verified working on 2026-09-30 and was `manifest unknown` by 2026-10-07, which broke the `docker` verb outright. Now pinned by *index* digest of upstream's `v1.22.3-immutable` tag, exactly as NIX_IMAGE is and for the same reason, with the version kept beside it as SKOPEO_IMAGE_TAG so the diff stays readable. The immutable tag has not moved since 2026-09-21, and because a tag references it the manifest stays alive. The pins file's claim that every pin on the route is "bumped deliberately, reviewed as a diff" now holds. The comment also records why 1.22.3 and not 1.24.1, because the version numbers invite exactly that question: v1.23.0, v1.24.0 and v1.24.1 exist as *source* releases, but no 1.23.x or 1.24.x container is published -- not on quay.io/skopeo/stable (all 77 tags enumerated), not on quay.io/containers/skopeo, not on ghcr.io, not on Docker Hub. skopeo maintains several stable branches at once and backports to them; v1.22.3 shipped the same day as v1.24.1, and the image track follows one of those branches. The note says not to "fix" the pin upward without checking that an image exists first. Verified: the same skopeo 1.22.3 (commit 92397d5b), a four-platform OCI index, that it runs, and that it still has --preserve-digests -- which is the reason the pinned container exists at all, Ubuntu 22.04's skopeo 1.4.1 lacking it. The `docker` verb's smoke test passes all three platforms again: amd64, arm64 and arm/v7 each run `lightningd --version` under binfmt.
The default was today's date, which made the release bytes a function of when the build ran: two hosts on either side of UTC midnight, or a co-signer a week late, would build byte-different tarballs from the same commit. Pin the baked timestamp to the committer date of the commit the flake actually builds, which every build path now passes through prepare_tree; verify keeps reading the manifest's own '# mtime' line and --mtime still overrides. The date-only shape is unchanged, which is what tar's --mtime="$MTIME 00:00Z", configure's MTIME and docker's SOURCE_DATE_EPOCH all consume.
armhf was scoped out of the required co-signed set while its Rust binaries diverged across machines -- the rustc-1.89-era reordered ARM EHABI string table -- and its artifacts lived in a captain-only SHA256SUMS-<ver>-armhf. That divergence is gone: with the mtime pinned (now the commit-date default) and the current toolchain, two x86-64 hosts of different CPU vendors and an outside co-signer produce byte-identical armhf tarballs. So the split is ritual now, and ritual in the signing path is where you do not want ritual: sign writes one manifest with every row and the source zip, publish expects that one, and the @cosign-optional directive and its reader are deleted. verify keeps handling the legacy -armhf manifests from split-era releases -- it reads whatever manifest it is handed -- with their withhold-on-mismatch path intact, since that is exactly how those releases were signed. The docs' and release notes' reproducibility caveat retires with it.
Rebasing onto v26.12rc1 brought in master's switch to nixpkgs' lowdown-unsandboxed and the removal of the unconditional `export PYTHON=python3` (configure now falls back to python3 itself). Both touch nix/pkgs/default.nix, so its recipe hash moved. The nixpkgs pin and the toolchain list are unchanged; this is `tools/reprobuild certify --no-verify`.
2da5932 to
8ec987d
Compare
|
Rebased onto The branch conflicted with
Head is now
|
Summary
Rebuild the release pipeline so a release is produced outside GitHub and published in one act: from a tagged commit,
tools/reprobuildbuilds the three static tarballs (amd64, arm64, armhf), the source zip, the multi-arch Docker image and the signed manifests, and publishes all of it together — artifacts and source to the download host, images to Docker Hub, the tag and the (draft) GitHub release. Nothing leads and nothing lags: a user can upgrade the moment they learn there is something to upgrade to.The build is a Nix derivation — the same in-tree derivation
nix buildproduces, as a static/cross variant — so "reproducible" is enforced by data and guards, not by hope:Two guards make the toolchain an auditable input: the certified rev in
nix/release-rowsmust equalflake.lock(nix flake checktrips otherwise), and everycertifyre-generates the per-row input manifests, whose recipe hashes make an edit to the linking recipe show up as a manifest diff.What goes away: the per-distribution builder images (Ubuntu/Fedora/Alpine route), their pin lists and hooks, and the GitHub Actions release/image workflows.
release.ymlkeeps PyPI only; the driver refuses to push a tag while that workflow would race it. Native builds are unaffected except that aconfigureprobe fix (a probe that always answered "no" for lack of amain()) re-enables-ffunction-sections/--gc-sections, i.e. slightly smaller native binaries.The VLS signer ships in no release here: CLN refuses a signer without splice support and no released VLS has it, so the binaries could only ship unable to start. The derivations and harnesses stay; bringing it back is a pin bump, a green regtest, one flag.
Evidence
allbuilds three tarballs that pass the ELF gate — 40 ELFs per row, all static-pie,--strip-all, build-id present — and every ELF boots under pinned qemu on the oldest supported Pi CPUs (cortex-a53,cortex-a7): 0 signal deaths.nix build --rebuildreproduces all three rows from one tree; cross-machine byte-equality (two machines) was established on the previous toolchain and is being re-run against this tree's toolchain bump (gcc 16.2 / rustc 1.98) as we speak; the armhf Rust rows are carried in a separate manifest pending that divergence, which never blocks a release.lightningd --versionon all three platforms under binfmt.Merge Danger
Door: two-way until the first
publishthrough it — no tag moves and nothing publishes from CI by merging this; rollback is re-enabling the old workflows. The one-way door is the first release made this way: the reproducibility promise those tarballs make is forever.Blast Radius: release tooling, release/publish CI, the runtime Dockerfile, docs and
configure. The developer path (./configure && make,make installto/usr/local) and packaging are untouched; theconfigureprobe fix changes native binaries' size/sections, not semantics.