Skip to content

test: split slow storage activation command cases - #750

Closed
roodboi wants to merge 4 commits into
nextfrom
test/split-storage-activation-commands
Closed

roodboi wants to merge 4 commits into
nextfrom
test/split-storage-activation-commands

Conversation

@roodboi

@roodboi roodboi commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #748 so this test-only follow-up uses light CI immediately. After #748 lands, target next; a squash merge of the base can make its commits appear in history, but this PR's authored changes are only the seven test/planning paths below. No freshness-only push is needed for the lead's combined-tree check.

The storage activation file exceeded the 12-minute per-file limit on macOS shard 1, run 43313402845. Seven cases consumed about 637 seconds before the finalization case could finish. Split it into five files so the existing three macOS lanes can schedule the independent heavy cases separately.

Group Cases Planning seconds
Original guard file 4 33.7
Enrollment 1 52.1
Recovery 1 188.9
Proof counts 1 362.2
Finalization 1 390 estimate

The first four weights are completed case sums from that hosted Mac job. Finalization's 390s is its unchanged per-test timeout upper bound, not a measured completion. Startup/hook overhead is unmeasured. Every group stays in the macOS pool; no coverage moves to Linux. This is a scheduling change, not a lower total-work or measured Mac speedup claim. The existing pooled plan covers all 730 files exactly, with Mac estimates around 19m19s under the 30-minute shard deadline.

All seven registration token streams (eight expanded cases) are identical: assertions, command order, proof counts, test.each data and timeouts are preserved. The shared helper keeps private fixture environments, capture budgets, retention on uncertainty and refusal of later calls in an uncertain file. Each file registers its own cold-lifetime fence; between files the per-file runner owns process lifetime, and fixtures remain separate. The only relocated helper path adjustment still targets the same source CLI. Each heavy file has one test with a 390s or 500s timeout, below the normal 720s per-file cap. #748's temporary 20-minute original-file exception is harmless but now stale; its removal remains a post-base cleanup.

Validation: all eight cases pass on isolated Linux (98 assertions; 567.53s); this preserves behavior but does not qualify Mac timing. Pinned typecheck, repository check/privacy, changed-file lint, gitleaks and exact pooled coverage pass. Pre-stack runner/hermeticity checks: 63 passed; post-stack result recorded in the readiness comment. Persistent reviewer clean on the authored patch. Lead's combined Mac run or next integration supplies actual Mac qualification; these files are deferred on an unlabeled light PR. No product runtime implementation or proof boundary changed. HACK-2111.


Get evidence
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

@vercel

vercel Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hack-docs Ready Ready Preview Nov 12, 2026 11:55pm UTC

Request Review

@roodboi

roodboi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T00:04:05.497632Z c56f367 Manual request
🔒 Security Review ✅ Completed 2026-11-13T00:06:37.538236Z c56f367 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: c56f367047

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: c56f367047

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@roodboi

roodboi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Ready for lead: Linux CI run 43332910450 passed all five unit shards, typecheck/lint/privacy, secret scan, docs and test. Persistent review clean at this head; Codex code review clean, no open findings, and computer pr review-status CLEAR at c56f367.

All eight storage cases also passed in isolated Linux verification; all seven registration token streams, assertions, timeouts and proof counts are preserved. The runner/hermeticity checks passed 67 tests after stacking. Pending hosted jobs: none. The five storage files remain in the macOS pool and are deferred under the light tier, so Mac duration/qualification remains for your combined run.

Stacked on #748; merge that base first and retarget this to next. Its only authored changes are the seven test/helper/planning paths. No freshness-only push and no merge during the prerelease freeze. The temporary #748 timeout exception can be removed once this split is integrated.
Real-VM check: no native implementation changed; run the five storage test files on the combined Mac tree. Their Docker/compiler transports are synthetic, so this split does not require a new real-VM behavior check.

c56f367

@roodboi

roodboi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Integration note for the combined tree: the newly updated #748 base contains #749, commit b21c3b189841c1a3cbb8af2b76414fa347dc7057. A read-only merge check of #748 head 56075e67 with this PR head c56f3670 reports one conflict, in tests/native-compose-storage-activation-command.test.ts.

#749 adds the fixture-local Unix /info engine and its teardown to the cold-case harness, sets DOCKER_HOST for those calls, and adds direct-read/exact CLI-identity-count assertions to the proof-count case. This split predates that optimization. Preserve those changes when resolving the combined tree: the engine/harness logic belongs in tests/helpers/native-compose-storage-activation.ts, and the new assertions belong in the extracted proof-count test. Ordinary guard cases must retain their CLI-only fallback coverage. The current #750 qualification predates that combination.

No freshness-only push made, per the batch policy. The split's cases, timeouts and lifetime fences remain as reviewed; the combined-tree qualification should exercise the preserved #749 assertions too.

roodboi added a commit that referenced this pull request Nov 13, 2026
Small PRs currently queue native builds ahead of other work. This makes
PR CI light by default and moves platform/Docker qualification to the
selected integration run on `next`. Superseded `next` pushes cancel
themselves; `main` keeps its full, non-cancelling push behavior.

A contributor sees secret scan, typecheck, lint/privacy, relevant docs
checks and selected Linux unit files on shards 4–8. A light run skips
macOS, Docker E2E, state models and CLI/release build smoke.
Candidate/compiler host qualification is skipped unless the Rust input
rule below applies. **Ready PRs changing protected CI inputs
automatically run full ordinary PR tiers, as `ci:full` would. Ready PRs
into `main` keep the full tier, including Intel.** Missing/empty diffs
or unavailable base policy also fall back to full tiers. Ready PRs
touching non-Markdown runtime-core/config-compiler inputs (including
pins, fixtures and generated data), or any relay-guest input, also keep
Linux Candidate core and both Linux compiler hosts with Rust checks.
Their macOS jobs stay deferred. Linux unit preflight still prepares its
matching compiler. On a light run, shard 8 names selected macOS paths as
**“deferred to the integration run”** before typecheck; deferred files
run nowhere on that PR. `test` and `docs site` always report. The
required `test` fan-in now also requires the active Linux
process-lifetime job to succeed; only that job's planned out-of-scope
skip passes. Branch rules are unchanged.

- `ci:mac`: opt into selected macOS unit/native tiers under their usual
input gates.
- `ci:e2e`: opt into Docker/runtime-image/lifetime jobs under their
usual gates. Labels combine.
- `ci:full`: every unit file and ordinary native/Docker tier, state
models and build smoke. PRs into `next` and ordinary `next` integration
runs omit Intel; `main` PRs/pushes and explicit full dispatches retain
it.

Drafts retain the light deferrals even when labeled, and both fan-ins
fail with `skipped: draft`. They no longer move macOS-pool files onto
Linux. The independent shell guard enables heavy tiers for protected
inputs, including this PR's workflow/selector changes. An ordinary
selector fallback widens files within the chosen tier. State models
follow the same full-tier admission.

![CI selection and release
evidence](https://raw.githubusercontent.com/hack-dance/hack/fix/ci-macos-throughput/docs/reference/ci-test-selection.svg)

`next` selection uses trusted policy/dependencies from `event.before`
against the new tree as data. Missing/zero/non-ancestor bases, protected
inputs, old policy, failed/cancelled/running/missing predecessor CI or
API errors force full integration selection. This keeps cancelled
changes covered. `next` runs selected platform and Docker jobs after
batch merges; failures are fixed forward. Only `next` pushes share the
cancelling group; full manual runs and `main` pushes stay independent.

For a release, freeze `next` at the release head, run `gh workflow run
ci.yml --ref next -f full=true`, wait for success, then dispatch the
prerelease. Every prerelease now needs this explicit full run. The plan
refuses subset-only evidence and prints that command. If a newer scoped
push at the same SHA is running, failed or cancelled, it also names that
run and says to wait for success or `gh run rerun RUN_ID`; older full
evidence cannot override it, while a new full dispatch can supersede it
once the dispatch succeeds. Plan and publish require executed
full-selection/release-full markers, all four compiler hosts, every
shard/job successful, and required CI checks from the selected suite and
exact attempt jobs. Newer failed/cancelled/unfinished evidence cannot
expose older green. Partial reruns missing current-attempt jobs refuse
safely; use an entire workflow rerun. A moved head refuses. Stable
release preparation remains main-push-only; cancelled runs cannot
qualify.

The storage activation command file gets a temporary 20-minute per-file
budget, runs first in its shard, and remains subject to the 30-minute
shard deadline. Other files retain 12 minutes; explicit CLI limits
override the exception. This addresses run
[43313402845](https://github.com/hack-dance/hack/actions/runs/43313402845)'s
12-minute timeout with bounded headroom, not a speedup claim.
Splitting/cutting the eight-case file and refreshing its planning weight
remain the next cost follow-up.

Ten successful six-macOS-job runs on Nov 12 gave these median job
minutes (queue excluded; sums of medians are descriptive):

| Job | Before minutes |
| --- | ---: |
| Unit 1 / 2 / 3 | 17.93 / 17.67 / 14.00 |
| Candidate ARM | 19.25 |
| Compiler ARM / Intel | 5.97 / 12.22 |

The total is about 87 macOS runner-minutes. Ordinary light PRs schedule
**zero** macOS jobs. Protected-input/full-tier PRs and ordinary full
`next` runs use five instead of six, deferring Intel to
`main`/release-full. This intentionally moves Intel-only regressions on
`next` to release qualification. Bun install is only 5–6s on unit lanes,
19s on Intel; unit compiler prep 48–55s. Three candidate logs show
10.3–14.3min Cargo compilation and 3.3–4.1min Cargo test execution.
Compiler ARM transport/bundle work alone is 169s median; folding it into
the 19min candidate job saves under a minute of setup while extending
that lane, so they stay parallel.

Bun downloads, Cargo registry/target trees and Zig data are cached by
OS/arch, image OS, toolchain pins and locks, with separate
compiler/candidate targets. PR native caches restore only; trusted
push/dispatch caches save. Image updates do not create another multi-GB
entry. Builds/tests still execute; Zig is checksum-verified and
re-extracted. Structured step timings from the first full PR run show
native-cache helper steps at 0–1s, Bun cache restore at 7–11s, unit
installs at 1–2s and unit compiler preparation at 34–60s. Candidate Zig
setup was 1s plus 12s post-step; compiler post-step was 16s. Warm
hit/miss and native-save benefit remain unqualified; PR native caches do
not save. This PR's protected-input guard now schedules its own full
ordinary PR run, which the lead requires before merging this gating
change.

Runner options, checked Nov 12: the account is Team; its published macOS
cap is five, shared by standard/larger runners. The sample observed five
concurrent Mac jobs; custom support overrides are not exposed by the
limits API and remain unverified. Larger Intel: 12 CPU/30GB, $0.077/min.
M2 Pro xlarge: 5 CPU (+8 GPU)/14GB, $0.102/min. Larger runners are paid
even for public repos; current standard public jobs are free.
[Limits](https://docs.github.com/en/actions/reference/limits),
[specs](https://docs.github.com/en/actions/reference/runners/larger-runners),
[prices](https://docs.github.com/en/billing/reference/actions-runner-pricing).

A spare Apple Silicon Mac buys queue capacity, not a lower current
hosted bill. Use ephemeral isolated VM/JIT workers labeled `self-hosted,
macOS, ARM64, trusted-ci`, admitted externally for trusted next/main
pushes and manual runs only. Never fork PRs, other PR events or
pull_request_target. Labels/editable workflow conditions are not access
control: restrict runner-group/workflow routing and event/ref admission,
discard each VM/checkout, and retain no host credentials. No
registration, paid runner or account changes here.

Validation on the Batch A integrated tree: 289 focused CI/shard/release
tests passed (1,734 assertions). The 65-file source-contract gate:
passed 1,315 tests with 26 conditional skips and zero failures (13,731
assertions). No daemon before/after. Pinned typecheck, lint/privacy,
explicit script/test lint, actionlint 1.7.12 and commit-range gitleaks
8.30.1 pass; existing complexity warnings remain. The persistent
reviewer is clean on exact tree
`98ab31ac29bfcb971058590d7cb750b906b0e69b`.

After next.16 published and #747 landed, normal merge
`ae94e5fb602240ee381d4f112edeb4f76813fd30` brought `next` at
`e7a70393e1bae3d3e453ac82bf20fad6273277b3` into this PR. The
model-coverage conflict retains this PR's CI-evidence row and #747's
updated log-follow row verbatim. Batch A implementation is unchanged.
The existing startup model README points to the three merged regression
files. The source-only selector regression requires a proper subset from
the real tree.

For comparison, the prior full PR run
[43371306048](https://github.com/hack-dance/hack/actions/runs/43371306048)
passed all 26 jobs at `56075e67`. Its five Mac jobs took
15.28/15.52/14.28 minutes for unit shards, 20.00 for Candidate ARM and
6.38 for compiler ARM: **71.47 Mac runner-minutes**, versus the earlier
six-job baseline's sum of medians, 87.03. These are different trees and
unmatched runs; they do not establish a cache speedup. Mac shard 1
waited 109.53 minutes after scope, so queue pressure remained. The
latest revision still needs its own hosted result.

This protected-input PR must run its full ordinary PR tier at the new
head. Ready follows green Linux jobs, clean reviews and handled bot
threads, naming any pending heavy jobs. The lead requires this PR's full
hosted run before merge. Manual dispatch/partial-rerun and warm-cache
qualification remain post-merge work; this PR adds no native runtime
implementation changes relative to `next`. After this PR merges, #688,
#691, #750 and #751 will be retargeted to `next`; #750 must preserve
#749's optimized identity engine and assertions. HACK-2111.

---------

Co-authored-by: Kit <344842+roodboi@users.noreply.github.com>
Base automatically changed from fix/ci-macos-throughput to next November 13, 2026 10:48

roodboi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Retarget audit after #748 merged: this PR now targets next; head remains c56f36704713ae3beb54ac661fc8cb7a470527d1.

Keeping it open. Authored split e644608f still contributes seven paths absent as a split from next 021b16bcfd13c48b14bf2677eba2cffe62453ea2: the extracted helper, four heavy-case files, trimmed original and duration hints. The helper and four new files do not exist on next; #748's temporary per-file timeout is not a replacement for the split.

The integration constraint remains: preserve #749 (b21c3b189841c1a3cbb8af2b76414fa347dc7057)'s private Unix /info engine, cleanup and DOCKER_HOST in the extracted helper, and its direct-read/exact CLI identity assertions in the proof-count file. Retain ordinary CLI-only fallback guards. All five split files remain in the macOS pool. No freshness-only push or superseded closure.

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Cleanup audit against next at a262d42762ae7b395921564203192b75169733d0: retain as draft; the storage-test split has not landed.

Commit/change mapping:

  • 573a4171 and ecacb456 are included in merged PR ci: make PR checks light and coalesce next integration runs #748's commit list; their CI-tiering/replay changes landed on next in e2c73c6122a545c13fdee2c89a6ce987d9e98b4b. The old inherited policy diff is therefore covered, including subsequent ci: make PR checks light and coalesce next integration runs #748 fixes.
  • e644608f carries the actual split. Its seven paths relative to ecacb456 are scripts/test-shard-durations.json, tests/helpers/native-compose-storage-activation.ts, the trimmed tests/native-compose-storage-activation-command.test.ts, and the four new enrollment/finalization/proof-count/recovery command test files. The helper and all four new test files are absent from next. The cases still residing in the original next file do not mean the scheduling split or duration provenance has landed.
  • c56f3670 is the normal merge of the light-CI base into that authored split. Its incoming ecacb456 policy is mapped to ci: make PR checks light and coalesce next integration runs #748 above. git diff ecacb456 c56f3670 --name-only contains exactly the seven split paths, so there is no unmapped extra authored change.

Reason to keep open: no verified successor provides this split, its per-file lifetime-fence boundary/documentation and updated planning weights. The original performance motivation and current macOS qualification must be rechecked after refreshing from next; Linux component evidence alone does not qualify Mac scheduling. Marking draft removes the stale ready signal while retaining the reviewed work. Tracking context: HACK-2111.

@roodboi
roodboi marked this pull request as draft November 14, 2026 01:13
@linear-code

linear-code Bot commented Nov 14, 2026

Copy link
Copy Markdown

HACK-2232

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Retained unlanded work is now explicitly tracked in HACK-2232, under HACK-2111, with acceptance criteria and links to the full commit-by-commit mapping above. This PR stays open as draft because its unique changes have no verified complete successor on next; no patch is represented as landed. The ticket records the refresh/qualification or explicit-retirement decision needed before closure.

@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Closing unmerged as requested by the lead; retained work is transferred to HACK-2232, which is still Backlog. This closes the stale PR, not that ticket or its acceptance criteria. The branch is preserved; no unlanded patch is represented as merged or retired.

Reverified against next at 5965b23: #748 includes both inherited policy commits, and its squash e2c73c6 is an ancestor. Unique net paths and missing artifacts remain as listed below. This supersedes the earlier keep-open decision now that the lead has explicitly requested closure and each gap has a tracking ticket.

Commit/change mapping:

  • 573a4171 and ecacb456 are included in merged PR ci: make PR checks light and coalesce next integration runs #748's commit list; their CI-tiering/replay changes landed on next in e2c73c6122a545c13fdee2c89a6ce987d9e98b4b. The old inherited policy diff is therefore covered, including subsequent ci: make PR checks light and coalesce next integration runs #748 fixes.
  • e644608f carries the actual split. Its seven paths relative to ecacb456 are scripts/test-shard-durations.json, tests/helpers/native-compose-storage-activation.ts, the trimmed tests/native-compose-storage-activation-command.test.ts, and the four new enrollment/finalization/proof-count/recovery command test files. The helper and all four new test files are absent from next. The cases still residing in the original next file do not mean the scheduling split or duration provenance has landed.
  • c56f3670 is the normal merge of the light-CI base into that authored split. Its incoming ecacb456 policy is mapped to ci: make PR checks light and coalesce next integration runs #748 above. git diff ecacb456 c56f3670 --name-only contains exactly the seven split paths, so there is no unmapped extra authored change.

Uncovered work above is assigned to HACK-2232; the mapping does not claim a complete successor PR for it.

HACK-2146 remains Done: #749 / b21c3b1 fixed the original timeout. HACK-2232 tracks only whether the separate scheduling split is still useful and how to qualify it.

@roodboi roodboi closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant