Skip to content

fix(tests): keep the stuck-launcher survivor clear of the job's CPU limit - #12260

Merged
MarkusNeusinger merged 2 commits into
mainfrom
fix/executor-stuck-test
Oct 23, 2026
Merged

MarkusNeusinger merged 2 commits into
mainfrom
fix/executor-stuck-test

Conversation

@MarkusNeusinger

Copy link
Copy Markdown
Owner

Summary

  • Root cause of the TestRender::test_a_launcher_that_outlives_its_kill_blocks_renders_until_it_exits flake (two CI failures on 2026-10-23, runs 39932413037 and 39940278486). The fake launcher execs the harness, so the process the test keeps alive is the harness and carries the run's RLIMIT_CPU, one second for a one-second job (wire.run_cpu_seconds). The harness start (matplotlib import, a fresh font cache per test) costs about 0.33 CPU s locally and roughly 2.4 times that on the CI runner (the plain render test takes 0.54 s here and 1.31 s there). Whenever it spent the full second, the kernel's SIGKILL ended the "stuck" launcher about 50 ms after the timeout, inside the 0.2 s grace window, so stuck read 0. The failing run's wall_s=1.054 is that death: a survivor keeps _stop waiting the full grace, about 1.22 s.
  • The executor is correct; only the test raced. A real sandbox do runs on the host, outside the limit the harness sets inside the sandbox, and stuck counts only launchers still alive. The test now gives the run the host's full CPU cap and asserts that cap on the survivor with resource.prlimit, so it fails every time without the fix instead of once in a while.
  • Takes ruff's PEP 758 except form in two lines from feat(core): adaptability checker for Python implementations #12244 that main's Lint job flags since the 3.14 move. These are the same hunks as fix(js-render): bump Playwright to 1.64.1 for the Ubuntu 26.04 runner #12259, so whichever PR lands first, the other merges cleanly.

Ruled out: the fake binary's delete --force only logs and exits 0. The sleeper writes nothing, so it gets no SIGPIPE. Asyncio cancellation of the readers does not close the transport, and PYTHONASYNCIODEBUG=1 with -X dev reports no "close running child" or unclosed-transport warning. Coverage does not follow subprocesses (pytest-cov 7, no patch = subprocess). Python 3.14 makes the harness start only about 5 % heavier than 3.13 (0.33 against 0.32 CPU s, same harness, same matplotlib), so it narrowed the margin but did not cause the flake.

Plan

N/A

Test plan

  • Deterministic reproduction of the CI signature (assert (True and 0 == 1), wall_s about 1.02, launcher exit -9): the old test body with code that spends more than one CPU second before it sleeps.
  • Adjusted test with the CPU-cap line removed: 31 of 31 failures, 16 idle and 15 under 48 busy loops on 32 threads.
  • Adjusted test as committed: 50 of 50 passes, 25 idle and 25 under 48 busy loops.
  • uv run pytest tests/unit tests/integration -n 4 --dist worksteal: 12,715 passed, 3 skipped.
  • tests/unit/agents/renderer under python -X dev with PYTHONASYNCIODEBUG=1: 179 passed, no asyncio warnings.
  • ruff check ., ruff format --check ., mypy api core agents and tools.changelog check --base origin/main are clean.

…imit

The executor test for a launcher that outlives its kill failed twice in CI
on 2026-10-23 with `stuck == 0` and `wall_s` just over the 1.0 s timeout.
The fake launcher execs the harness, so the process the test keeps alive
is the harness itself and carries the run's RLIMIT_CPU: one second for a
one-second job (`run_cpu_seconds`). The harness start (matplotlib, a fresh
font cache per test) costs about 0.33 CPU seconds locally and about 2.4
times that on the CI runner, so once it spent the full second the kernel's
SIGKILL ended the "stuck" launcher inside the grace window. A real
`sandbox do` runs on the host, outside that limit, so the executor's
`stuck` count is right and only the test raced.

The test now gives the run the host's full CPU cap and asserts that limit
on the survivor with `resource.prlimit`, so the test fails every time
without the fix rather than once in a while: 31 of 31 failures with the
fix removed, idle and under 48 busy loops; 50 of 50 passes with it.

Also takes ruff's PEP 758 `except` form in two lines from #12244 that
main's Lint job flags since the 3.14 move (the same hunks as #12259).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0175hA62juZoajMcqKxWFGza
Copilot AI lite review requested due to automatic review settings October 23, 2026 11:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The reviewed changes have no unresolved blocking issues and the test stabilization is covered by the supplied validation.

No open issues 🎉

What changed in this PR

This pull request makes the stuck-launcher renderer test deterministic by preventing the fake survivor process from inheriting the job’s one-second CPU limit. It also updates the adaptability checker syntax for Python 3.14 and documents the fix.

Changes:

  • Applies and verifies the configured CPU limit for the test survivor.
  • Updates both adaptability checker copies to use PEP 758 exception syntax.
  • Adds a changelog fragment for the test reliability fix.
File Description
tests/​unit/​agents/​renderer/​test_executor.py Makes the stuck-launcher test deterministic and verifies the survivor’s CPU limit.
core/​adaptability.py Applies the Python 3.14-compatible exception syntax.
changelog.d/​executor-stuck-test.md Documents the renderer test flake fix.
automation/​scripts/​adaptability_check.py Keeps the generated checker mirror synchronized with the source checker.

Review effort: Lite

@codecov

codecov Bot commented Oct 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
automation/scripts/adaptability_check.py 0.00% 1 Missing ⚠️
core/adaptability.py 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@MarkusNeusinger
MarkusNeusinger merged commit 3da9c1b into main Oct 23, 2026
15 of 16 checks passed
@MarkusNeusinger
MarkusNeusinger deleted the fix/executor-stuck-test branch October 23, 2026 11:57
MarkusNeusinger added a commit that referenced this pull request Oct 23, 2026
MarkusNeusinger added a commit that referenced this pull request Oct 23, 2026
…#12259)

## Summary
- **Playwright 1.55.1 → 1.64.1** in the root `package.json` and
`package-lock.json`. GitHub's `ubuntu-latest` now resolves to Ubuntu
26.04 (image 20261012.163.1). Playwright 1.55.1 has no `ubuntu26.04-x64`
host platform, so `npx playwright install --with-deps chromium` in
`.github/actions/setup-node` failed (run 39938127853). Every
`impl-generate`/`impl-repair` run for chartjs, d3, echarts, highcharts
and muix died in setup. 1.64.1 ships the 26.04 download paths and apt
dependency list. The lock diff is playwright + playwright-core plus the
darwin-only `fsevents` optional entry that 1.64 no longer declares.
esbuild stays at 0.28.1. Integrity hashes match the registry. This also
aligns with the `playwright-core@1.64.1` pin that
`scripts/site-smoke.mjs` and `ci-image.yml` already use.
- **Ruff format fix no longer part of this PR.** This branch first
carried the two PEP 758 `except` lines that main's lint needed. #12260
landed the identical lines, and after merging main they collapsed, so
the diff is Playwright only.
- Supersedes Dependabot #12251 for the Playwright half; its esbuild
0.28.2 half is left to Dependabot.

## Plan
N/A

## Test plan
- [x] **The action's own steps on Ubuntu 26.04, in a container.**
`ubuntu:26.04` (26.04.1 LTS, resolute) with Node 22.23.3 (the action's
default `node-version: "22"`). Every `run:` step of
`.github/actions/setup-node/action.yml` ran verbatim against a minimal
checkout:
- **1.64.1:** `npm ci`, `npx playwright install --with-deps chromium`
(apt deps + Chrome for Testing 156.0.8078.4) and both smoke renders all
pass. The chartjs PNG and the MUI X React-path PNG are both 3200×1800.
- **1.55.1 (negative control):** the install fails with exactly the CI
error, `Error: ERROR: Playwright does not support chromium on
ubuntu26.04-x64`.
- [x] Render comparison: 1.55.1 (Chromium 140.0.7339.186) against 1.64.1
(Chromium 156.0.8078.4). Each harness was confirmed to launch its own
build. The sample is 17 catalogue snippets × 2 themes = 34 pairs,
rendered the way the pipeline calls the harness:
- chartjs: 4 snippets. d3: 3. echarts: 4. highcharts: 3. muix: 3. Six of
them are square-orientation.
- **All 34 pairs are pixel-identical:** 0.000 % of pixels differ and the
content bounding boxes match. Only the PNG encoding bytes differ.
- [x] **MUI X sample excluded two snippets.** Two sampled muix snippets
(`spiral-timeseries`, `biplot-pca`) import
`@mui/x-charts/ChartContainer`. MUI X 9.16 (#12243) no longer exports
that path, so both fail identically under BOTH Playwright versions. This
is the known #12243 blast radius, and the muix bulk regeneration covers
it. The muix sample therefore uses three snippets that still bundle.
- [x] `ruff check`, `ruff format --check`, `mypy api core`, `pytest
tests/unit tests/integration` (12715 passed) and `tools.changelog check`
are green locally.

## Known gaps
- **No PR check runs `.github/actions/setup-node`.** Only
`impl-generate.yml` and `impl-repair.yml` use it, and neither runs on
pull requests. A green PR therefore does not prove the install on the
real runner. The 26.04 container run above is the closest local proof:
same OS release, same Node major, same verbatim steps. The GitHub runner
image adds preinstalled software on top of that.
- **The real test is the first `impl-generate` run for a JavaScript
library after the merge.** The muix bulk-regeneration pilot will be that
run.
- **Steps after setup are also untested on 26.04.** No `impl-generate`
run has yet passed setup on the new image. Python, uv and the pngquant
apt install already pass there (run 39938127853, steps 9–13). Claude,
image processing and the GCS upload have not run on 26.04 yet.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants