ci: exercise and measure the real Next.js app in the Linux proof - #12
Merged
Conversation
`resource_benchmark` produces every RSS/PSS/private-dirty/cgroup number Coop
publishes, and it could only ever measure one fixture: `locate_prepared_app`
hardcoded `target/dynamic-smoke/compiled/test1`, the tiny dependency-free app.
That is the wrong app to draw conclusions from. A tiny handler shares almost
everything through the providers, so its marginal cost per deployment is close
to the floor. A real application does not, and the density claim -- the whole
premise of the architecture -- is about real applications.
`COOP_BENCH_APP_LIBRARY` now points the benchmark at any published package,
with the tiny app unchanged as the default so Linux CI keeps measuring what it
has always measured:
scripts/prepare-next-benchmark.sh
COOP_BENCH_APP_LIBRARY=target/next-benchmark/coop-run/compiled/next-bench/<pkg>/app.so \
cargo test --release -p coop-daemon --test resource_benchmark \
measure_in_process_startup_and_rss -- --ignored --nocapture
Replicating the Next fixture measures something real, and it is worth saying
why rather than assuming it: Perry compiles the Next route INTO the application
dylib, so at load time a copy needs the shared providers and nothing else -- no
`node_modules`, no build output. N copies therefore measure the genuine
marginal cost of an Nth Next deployment on one box, which is exactly the
question the architecture exists to answer.
The override asserts the path is a file rather than failing later inside the
fixture copy, because a typo'd path would otherwise surface as a confusing
manifest error several steps downstream.
The Linux proof publishes RSS/PSS/private-dirty/cgroup matrices for 1, 10 and 100 applications, and every one of them measured the tiny dependency-free app. That app shares almost everything through the providers, so its marginal cost per deployment sits near the floor. The architecture's density claim is about real applications, so this measures one. Replicating the Next fixture is meaningful rather than a trick: Perry compiles the Next route INTO the application dylib, so a loaded copy needs the shared providers and nothing else -- no `node_modules`, no build output. N copies are the genuine marginal cost of an Nth Next deployment on one box. Node is required to BUILD the fixture (`npm ci` plus the `next build` that `prepare-next-benchmark.sh` performs), never to run it. The build is not optional: a committed `.next-production-bundle/` silently drifted from its source once already, and measuring Coop against different code than the Node comparison compiles is not a comparison. The step resolves the published package explicitly and fails when there is none, rather than letting `resource_benchmark` fall back to its default and report the tiny app's numbers under a heading that says Next. That fallback is the failure this arm would be least able to notice about itself.
`binary_http_roundtrip` is the only test that exercises the Next.js fixture -- it loads the published `next-bench` package and dispatches a real request through it. It ran nowhere. It is excluded from `fast-check` as one of three provider suites that need built provider images, and it was never added to the Linux proof. So the only test covering a real framework route was gated by nothing, and a green proof said nothing about Next. I nearly drew a conclusion from that. #10 made the fixture drive Next's real `AppRouteRouteModule.handle`, its proof went green, and the obvious reading -- "the real Next route works under Coop" -- is not supported by that run, which never touched the route. A proof that skips its most interesting case is the failure mode this repository keeps finding in its own gates, and this is one more instance of it. Worth noting what the test asserts, because #10 changed its meaning without changing its text: status 200, exactly one `content-type: application/json` header, and `checksum == 3726872593`. Those were precisely the constants the old handler fabricated, so the assertions were self-satisfying. They now come from Next, so the same three lines finally test something.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (2)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe change stages the Next.js server bundle under ChangesLinux runtime validation and Next.js bundle staging
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: ⚪ Minimal · up to This PR expands Linux proof coverage and measures the real Next.js application; no actionable merge-blocking risk remains beyond normal checks and review. Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant NextJsFixture
participant DelegatedRuntime
participant EvidenceUpload
GitHubActions->>NextJsFixture: install dependencies and build fixture
NextJsFixture-->>GitHubActions: publish benchmark fixture
GitHubActions->>DelegatedRuntime: run integration and resource benchmarks
DelegatedRuntime-->>GitHubActions: return test and resource results
GitHubActions->>EvidenceUpload: upload linux-next-resource-results.txt
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/settings/billing. Comment |
`binary_http_roundtrip` runs in the loader/lifecycle step, and when no
published `next-bench` package exists it rebuilds one through
prepare-next-benchmark.sh -- which needs the dependency tree installed. The
`npm ci` and fixture-build steps were placed AFTER that, alongside the density
matrices, so the test ran first and failed with:
cannot rebuild the Next benchmark fixture: the Next dependency tree is
not installed under .../benchmarks/next-small
My ordering mistake, found by the test I had just added in the previous commit
-- which is the argument for adding it. Setup now precedes the tests, and the
step carries the reason so it is not "tidied" back later.
…ee it
The fixture compile failed in CI with Perry reporting no resolution for
`../.next/server/app/api/benchmark/route.js`. The import form was fine; the
file was simply not there.
Coop stages compiler inputs two ways, and neither carried it.
`copy_source_snapshot` copies the DECLARED source files and refuses symlinks
and non-files outright ("compiler input is not a plain file"). Separately,
`node_modules` is dereferenced wholesale into the private snapshot. A
deployment has no way to say "this build output is also a compiler input" --
`[[handlers]]`, `[[static]]`, `[[crons]]` and `[[queues]]` are the entire
vocabulary. So the `.next` symlink beside the handler never reached
`.coop-source/`, and the daemon fed Perry exactly two files.
The bundle now travels through `node_modules`, which is the mechanism Coop
actually has for dereferencing a tree of non-source inputs. That is defensible
rather than merely expedient -- the build output genuinely is a dependency of
the handler -- but a first-class declaration would be better, and the comment
in the script says which one this is so the next reader is not misled.
Also asserts the route reached the staged bundle before the daemon runs. The
failure it guards is silent and costs a full CI cycle to diagnose, and the
check is three lines.
Verified locally rather than in CI this time, since the previous two failures
here were both plumbing found only after a 40-minute run: built the fixture,
staged it exactly as the script does, compiled the handler to CommonJS and
drove it with a real COOP frame. Status 200, `content-type: application/json`,
and the route's own body -- all read from the NextResponse.
Perry's diagnostic for the underlying case is filed as PerryTS/perry#8348: a
missing relative module is reported as a missing stdlib binding, which points
at compilePackages and cannot fix a file that is not there.
Nothing was failing to compile. Perry never attempted it, and both reasons
were self-inflicted.
`collect_modules.rs` decides AOT vs runtime-JS by LOCATION, not extension: a
`.js`/`.cjs`/`.mjs` file that is project source goes through the native
pipeline exactly like a `.ts` file, and only a file under `node_modules` keeps
the runtime-JS classification. There is no V8 fallback any more, so that
classification is a refusal.
The previous commit staged the bundle at `node_modules/.coop-next-bundle/`,
which got it into the compiler snapshot but was wrong twice over:
* under `node_modules` -> runtime-JS classification, refused by the
V8-free gate.
* leading dot -> `collect_packages_in_node_modules` skips dot entries
(`if name.starts_with('.') { continue }`), so it was never enumerated,
and the automatic `compilePackages` `"*"` expansion -- the DEFAULT when
no package.json pins the key -- never covered it either. The name chosen
to avoid colliding with real packages is exactly what opted it out of the
mechanism that would have compiled it.
That also corrects the conclusion in the previous commit message. Coop does
not need a way to declare `perry.compilePackages`, and the daemon does not need
to emit a package.json: with no key present Perry already compiles everything
reachable, and project source is compiled natively regardless.
So the bundle is now a plain `next-build/` directory in the deployment: no
dot, outside node_modules. `collect_source_files` walks it (it skips dots,
`node_modules`, `migrations`, `static`, `coop.toml`, and refuses symlinks) and
collects every `.js`, and Perry compiles the whole bundle natively with no
opt-in required.
Verified locally again before pushing: staged exactly as the script does,
compiled the handler to CommonJS, drove it with a real COOP frame. Status 200,
`content-type: application/json`, the route's own body.
Perry compiled the entire Next.js production bundle natively -- 60 codegen
units, 77 s, 620 MB peak RSS -- published the package, and loaded the app.so in
6 ms. The step still failed, with:
Coop exited before publishing the Next benchmark fixture
three lines below the daemon's own:
compilation succeeded and immutable package was published
The wait loop keyed on a single log line, "application library preloaded on
dedicated Perry thread", which only the dedicated-worker path emits. This
fixture's isolation resolves to "trusted", so it loads IN the daemon and logs
"preloading application library in daemon" instead. The condition never
matched, the one-shot daemon exited, and the script reported a publication
failure for something it had already published.
Now matches either line. That is also the more honest condition: what this
loop is waiting for is the application being loaded, not the particular thread
it landed on, and the published-package check immediately below is what
actually validates the result.
Proven end to end on the bench mini against the pinned Perry (fc6b3378): the
driver builds a COOP frame, calls the handler, and Next's real
`AppRouteRouteModule.handle` returns
dispatching...
returned bytes: 72
status: 200
That is Next's work-store setup, method resolution and NextResponse
construction running as native code. No Coop, no dylib, no V8.
Three blockers, none of them a Perry defect.
**Chunk loading.** Both bundlers load chunks with a computed require --
webpack `require("./chunks/" + id + ".js")`, turbopack
`require(path.resolve(RUNTIME_ROOT, chunkPath))`. An ahead-of-time compiler
cannot statically resolve either, so the chunk never enters the binary and the
route dies on first dispatch. `splitChunks: false` + `runtimeChunk: false` for
the server build makes `route.js` self-contained (209 KB, zero computed chunk
requires). That is close to a serverless bundle, not an exotic setting.
**The bundler had been silently swapped -- by me.** The fixture's script is
`next build --webpack`; invoking `next build` directly gets turbopack under
Next 16. That is why the bundle previously committed here was webpack, which I
had read as staleness rather than intent. The prepare script now passes
`--webpack` and says why.
**CJS interop shape.** Measured on the mini rather than assumed:
ns keys : default, module.exports
ns.routeModule : undefined
ns.default.routeModule.handle : function
The old nested-ternary-plus-`??` resolution did not land on that. It is now
plain statements with the measured shape recorded in the comment, so the next
reader does not have to rediscover it.
Perry warned about the AOT-unsupported sites at compile time and named the
failure precisely at runtime; every message was accurate and the sequence of
failures was mine to fix, not the compiler's.
NOT claimed here: that this explains the CI SIGSEGV. Locally this is a clean
200 where CI segfaulted. The same bundler mistake underlies both, but a crash
and a throw are different failure modes, and that stays open until CI is green.
The fixture build now fails with
perry compile terminated for deployment next-bench:
wall time exceeded 300 seconds
which is the daemon's default `compile_timeout_seconds`, sized for ordinary
application code.
This fixture is not that. It compiles Next's entire production server surface
natively: the App Route runtime alone is 15-21 MB of IR across 400-535
functions, wide enough that Perry drops it to -Os because LLVM's -O1+ pipeline
will not survive functions that size (#4880). Measured at roughly 8 minutes on
a quiet M1; a 2-vCPU CI runner is slower again.
The slowdown is the previous fix working, not a regression. Disabling server
chunk splitting made route.js self-contained, so there is more to compile. The
earlier 77-second compile was the split build -- which compiled fast and then
died at first dispatch because its chunks were loaded through a computed
require Perry cannot resolve ahead of time.
Raised to 1800 s for this fixture only, in the runtime.toml the prepare script
writes, with the reasoning inline so nobody trims it back wondering why a
benchmark needs half an hour.
The density harness copies the app image once per deployment (`std::fs::copy(source_app, &app)` in resource_benchmark.rs), so 100 apps means 100 physical copies. That is free for the tiny dependency-free fixture and is not for this one, which carries Next's entire compiled server surface -- the turbopack build was already 6.9 MB and the self-contained build compiles far more in. Running out of disk here would not say "disk full". ENOSPC surfaces as unrelated-looking failures: PerryTS/perry#8228 was filed as a codegen bug because `clang -c` fails with EMPTY stderr under ENOSPC, and it cost real time before anyone looked at `df`. So the step now prints the app size, the projected requirement and the free space, and refuses with a specific message if the arithmetic does not work. Cheap, and it turns a confusing 20-minute failure into one line.
With the timeout raised, the compile got further and hit the next wall:
perry compile terminated for deployment next-bench:
process-group RSS 4208620 KiB exceeded 4194304 KiB
4.2 GB against the daemon's 4 GB cap, on a runner with 7.75 GB total.
Raising the cap alone would be the wrong fix. Peak compile RSS scales with
CONCURRENT LLVM units, and this fixture's units are 15-21 MB of IR apiece. The
default on this runner is 2 module jobs x 2 unit workers -- four at once -- on
a box with 2 vCPUs, so that concurrency was buying nothing and costing four
times the memory.
So the workflow pins `PERRY_MODULE_JOBS=2` / `PERRY_CODEGEN_UNIT_JOBS=1` (two
concurrent units), and the cap moves to 6144 MB as headroom above that rather
than as the mechanism. Going near 7.75 GB would hand the job to the kernel OOM
killer, which replaces a clean refusal naming the limit with a mysterious
death -- strictly worse to debug.
Also prints `free -m` before the build, so the next memory failure can be read
against what the runner actually had rather than what it was assumed to have.
The Next fixture cannot be built on a GitHub-hosted runner. Its compile peaks above 8.3 GB RSS; the runner has 7.75 GB. Both earlier measurements were the CAP, not the peak. The first run reported 4.2 GB because the daemon's default limit stopped it there; raising the limit to 6 GB produced 8.3 GB, and the true figure is higher still. Treating 4.2 GB as the requirement was my error, and it cost an iteration. Reducing Perry's codegen concurrency is the right lever and is not available: Coop calls `env_clear()` before spawning the compiler so ambient `PERRY_*` switches cannot silently change emitted code without changing build identity, and `COMPILER_ENV_ALLOWLIST` is toolchain paths only. My earlier attempt to set `PERRY_MODULE_JOBS` in the workflow was a silent no-op for exactly that reason -- the env vars were visible in the step and stripped before the compiler saw them. The guard is correct; widening it to make CI convenient is a trade worth more deliberation than a debugging session affords. So the three Next steps move behind a `next_fixture` workflow-dispatch input, off by default, and `binary_http_roundtrip` comes out of the unconditional test list. Leaving either in would fail every run for a reason unrelated to the change under test -- a gate that is always red teaches people to ignore it. The fixture itself is unchanged and works: on an 8-core/16 GB M1 against the pinned Perry, a COOP frame in, `AppRouteRouteModule.handle` executed natively, `status: 200` out. What moves is where it is measured, not whether it runs.
I wrote "8-core/16 GB M1" in the previous commit and in the README without checking. The mini has 8 GB, barely more than the 7.75 GB runner that could not complete this compile. That makes the earlier explanation incomplete rather than merely imprecise. Memory alone does not separate the two hosts: macOS compresses and swaps under pressure, so a compile that peaks above physical RAM still finishes, while the Linux path is stopped dead by the daemon's `compile_max_rss_mb` cap the moment the process group crosses it. So the honest statement is that the mini succeeds despite similar RAM, not because of more of it, and a host with genuinely more memory remains the safer choice for this fixture.
Five fixes to `resource_benchmark.rs`, each found by a measurement that was
silently wrong.
**Daemon stderr was `Stdio::null()`.** A daemon that failed to start reported
only "exit status: 1" and threw its reason away. Six runs were diagnosed as
memory limits, timeouts and contention before capturing stderr produced the
actual cause in one run:
Error [ERR_PERRY_PATH_MODULE_THREAD]: Perry path-module initializer
registration must run on the runtime thread that owns the JavaScript heap
**RSS sampled only the daemon's pid.** In `in_process` mode that is the whole
system, so those figures were right. In `worker` mode every app runs in its own
process, none of which the sampler looked at -- ten apps read ~18 MiB,
indistinguishable from one, making worker mode look 6x DENSER than in-process
when it is actually ~6x heavier. Now sums the daemon's whole process tree, via
one `ps` snapshot walked by ppid rather than repeated `pgrep -P` calls, which
race against worker restarts. Verified against the in_process arm, whose
numbers are unchanged within noise.
**The request contract was the tiny fixture's.** Path `/` and body exactly
`ok`, asserted literally. Pointing the harness at any real application gave
`left: 404` before it measured anything. Path and expected body are now
`COOP_BENCH_REQUEST_PATH` / `COOP_BENCH_EXPECT_BODY`, and the body is a
substring: a real response carries incidental detail, and proving the request
was served is the job here -- correctness belongs to binary_http_roundtrip.
**Workload concurrency and timeout were hardcoded** at 50 and 10 s around a
handler that answers in microseconds. A Next.js route is orders of magnitude
heavier per dispatch, and 50 concurrent on one executor exceeded the timeout on
a QUIET host, so this was the workload's shape and not contention. Both are now
knobs.
**Execution mode was hardcoded** to `in_process`, so the harness could not
compare against process-per-app at all.
Together these produced the comparison that motivated them (tiny fixture, M1
mini, 2 trials, marginal cost per app): in_process ~1.1 MiB, worker ~17.7 MiB.
The in-process model is worth ~16x on marginal density, which is the opposite
of what I expected before measuring -- shared `.dylib` text is real but small
next to a per-process JS heap, arena, GC state and stacks.
Also corrects a claim I published earlier: the 20 MB -> 1.7 MB per-app falloff
between 10 and 100 apps does NOT reproduce here. Marginal cost is flat at
~1.1 MiB. That earlier shape was an artefact of the environment or of PSS vs
RSS, not a property of Coop.
fast-check's rustfmt gate failed on two hunks in resource_benchmark.rs. No behavioural change. Claude-Session: https://claude.ai/code/session_01UZJbhb2FTuakurTHPAKQgd
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes, both about the Linux proof measuring and exercising a real application rather than only the tiny fixture.
1. The Next.js roundtrip ran in no workflow at all
binary_http_roundtripis the only test that exercises the Next fixture — it loads the publishednext-benchpackage and dispatches a real request through it. It ran nowhere. Excluded fromfast-checkas one of three provider suites (they need built provider images), and never added to the proof. The only test covering a real framework route was gated by nothing.I nearly drew a conclusion from that. #10 made the fixture drive Next's real
AppRouteRouteModule.handle; its proof went green; and the obvious reading — "the real Next route works under Coop" — is not supported by that run, which never touched the route.Worth noting what the test asserts, because #10 changed its meaning without changing its text:
Those were exactly the constants the old handler fabricated, so the assertions were self-satisfying. They now come from Next, so the same three lines finally test something.
2. Density for the Next app, not just the tiny one
The proof publishes RSS/PSS/private-dirty/cgroup matrices for 1/10/100 apps, and every one measured the tiny dependency-free app — which shares almost everything through the providers, so its marginal cost per deployment sits near the floor. The architecture's density claim is about real applications.
This is meaningful rather than a trick: Perry compiles the Next route into the application dylib, so a loaded copy needs the shared providers and nothing else — no
node_modules, no build output. N copies are the genuine marginal cost of an Nth Next deployment on one box.Node is required to build the fixture (
npm ci+ thenext buildthatprepare-next-benchmark.shperforms), never to run it. The build isn't optional: a committed.next-production-bundle/silently drifted from its source once already, and measuring Coop against different code than the Node comparison compiles is not a comparison.The step resolves the published package explicitly and fails when there is none, rather than letting
resource_benchmarkfall back to its default and report the tiny app's numbers under a heading that says Next. That fallback is the failure this arm would be least able to notice about itself.Verified prerequisite
build-perry-libraries.sh:84builds the Perry compiler under theperry-devprofile, so the fixture can be compiled in the proof job — checked rather than assumed.Summary by CodeRabbit