Skip to content

docs: clarify E2E CI scope and result guarantees - #691

Closed
roodboi wants to merge 12 commits into
nextfrom
kit/hack-688-followups-1111
Closed

roodboi wants to merge 12 commits into
nextfrom
kit/hack-688-followups-1111

Conversation

@roodboi

@roodboi roodboi commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

The E2E README described focused doc tests, full push runs and cancellation passing fan-ins. Update its CI section and result diagram to the policy from #733 and #748: light Linux PRs with explicit macOS deferrals, independent ci:mac/ci:e2e labels, ci:full, failed draft fan-ins, selected next integration with safe predecessor fallback, and explicit exact-head release-full qualification. Failure, cancellation and unexpected skips cannot count as green.

The latest correction includes the lead's requested #748 exceptions: ready PRs changing protected CI inputs automatically run full ordinary tiers; ready PRs into main retain full tiers including Intel. Ordinary next integrations omit Intel. Light runs explicitly skip macOS, Docker and build smoke. The diagram separates draft failure, light deferral and full-tier qualification; release guidance names the scoped-push wait/rerun path. These docs must land after #748's corresponding fix, whose push is held until next.16 publishes.

Normal merges bring next and #748 into the existing branch; no history rewrite. Temporarily stacked on #748 to use light CI. Merge that base first, then target next; its squash merge can make inherited commits appear in history, but the authored diff remains only these three files under tests/e2e. This is independent of #688's Docker experiment.

CI results

Validation: the earlier 145 CI/shard/release contract tests passed (1,107 assertions). For this documentation correction, four CI-section links resolve, the diagram is regenerated with Pele 1.9.0, privacy/commit-range gitleaks/whitespace checks pass, and the persistent reviewer is clean against the reviewed #748 fix. Hosted Linux and current-head bot review are separate readiness gates. No workflow, scenario, runtime or release behavior changes. HACK-2111.

@vercel

vercel Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hack-docs Ready Ready Preview Nov 13, 2026 12:51am UTC

Request Review

@linear-code

linear-code Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

HACK-688

HACK-2231

@roodboi

roodboi commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T00:54:19.384595Z f741104 Manual request
🔒 Security Review ✅ Completed 2026-11-13T00:55:17.356745Z f741104 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 6cabd1489a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6cabd1489a

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@roodboi

roodboi commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 80ca76f471

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 80ca76f471

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

roodboi added a commit that referenced this pull request Nov 12, 2026
The per-file CI runner's tests checked the suite working directory but
did not prove that Bun loaded that suite's test preload. Add four
subprocess regressions covering the root and workspace suites: each must
load its own preload before the test, run its teardown hook, and report
a failing hook as a failed shard file even after the test assertion
passes.

Only `tests/test-shard.test.ts` changes. This closes a separate coverage
gap from #681, #688, #689, #691 and #700. Release signal: test coverage
only.

Validation: Bun 1.4.4 focused tests pass (48 tests, 360 assertions);
explicitly bypassing suite configuration makes all four new cases fail.
Changed-test lint, repository typecheck/check, staged privacy scan,
compiler build and CLI build pass. The additional monolithic suite
attempt has no terminal receipt and is not counted as passing.

Persistent HD Codex review: CLEAN at
`af967a55470f1d3ce67a69421d7fc0b3188fb585`; independently ran the four
new cases (24 assertions). Synthetic hooks qualify runner integration,
not production guard internals. An additional container attempt passed
the new cases but failed four existing checkout-discovery checks because
linked Git metadata was unavailable in the mount; no whole-file
container pass is claimed.

Exact-head [hosted CI run
43023939316](https://github.com/hack-dance/hack/actions/runs/43023939316)
completed successfully at `af967a55470f1d3ce67a69421d7fc0b3188fb585`.
All eight unit shards, all four Docker E2E shards, both aggregates, and
both required branch contexts (`test` and `docs site`) succeeded.
Scope-selected Candidate core and advisory Proof were skipped as
configured.

GitHub Codex code review completed without findings at the same head.
Bot conversation comments have been read; `computer pr review-status`
reports `CLEAR at af967a5` with no unresolved bot threads or change
requests. The base subsequently advanced on unrelated files; the signed
head is unchanged.

Keep draft and unmerged, targeting `next`.

Co-authored-by: Dimitri Kennedy <344842+roodboi@users.noreply.github.com>
roodboi added a commit that referenced this pull request Nov 12, 2026
The stable release gate's API tests previously returned CI jobs in one
response, so they could not detect a regression that read only the first
page. Add twelve cases across CI-completion and manual-dispatch
triggers: required jobs on page two must be accepted, while failed,
cancelled or skipped supplemental jobs on page two must block
preparation. Missing later pages and the ten-page limit must fail
closed.

Only `tests/release-prepare-gate.test.ts` changes. Workflow behavior,
release policy and the required-job list are unchanged. This is
independent of #681, #688, #689 and #691.

Validation: Bun 1.4.4 focused tests pass (21 tests, 64 assertions);
changed-test lint, full lint/check, full typecheck, staged privacy scan
and diff whitespace checks pass. The config compiler build passes. The
isolated full-suite and mutation attempts stopped during toolchain
startup before tests executed; neither is claimed as passing. Hosted CI
run 43019614781 completed successfully on exact head
`58cc69744aa6604d50fa6dc714fda7d915281d94`; both required contexts
(`test` and `docs site`) passed. The first run timed out in
`native-authored-project-start.test.ts`; one failed-jobs rerun passed
without source changes. Codex code review completed with no findings,
and `computer pr review-status` reports CLEAR at this head with zero
unresolved bot threads.

Release signal: test coverage only; no product release. Keep draft and
unmerged.

Co-authored-by: Dimitri Kennedy <344842+roodboi@users.noreply.github.com>
@roodboi
roodboi changed the base branch from next to fix/ci-macos-throughput November 13, 2026 00:09
@roodboi
roodboi marked this pull request as ready for review November 13, 2026 00:10
@roodboi

roodboi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 2b5959ad90

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 2b5959ad90

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@roodboi

roodboi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f74110436b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/e2e/README.md Resolved
@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: f74110436b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@roodboi

roodboi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Blocked: current-head Linux CI is green in run 43340674654, including shards 4–8, typecheck, lint/privacy, secret scan, docs and the required test fan-in. The earlier cancelled run is superseded. Persistent review is clean; no heavy jobs are pending in this light run.

One Codex thread remains open because these docs describe the lead-requested #748 parent correction, which is committed and tested locally but not published. next.16 is still awaiting protected publication, so the lead's freeze prevents the parent's merge/push. I have answered the thread and will verify/cite the published parent SHA before resolving it and posting Ready. No further documentation push is needed for that correction.

f741104

@roodboi

roodboi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Ready for lead: current-head Linux CI passed in run 43340674654, including shards 4–8, typecheck, lint/privacy, secret scan, docs and test. Persistent review is clean. The Codex source/docs mismatch is now answered and resolved against the verified published parent fix, 56075e67b01de0f13e6fa4f0288d5b1d8945d986; computer pr review-status is CLEAR at f741104, with no unresolved bot threads or overlapping base changes.

The authored change remains three documentation/diagram files. Four CI-section links resolve, the diagram is generated with Pele 1.9.0, and privacy/secret/whitespace checks pass. No additional push was needed after the parent fix. Merge #748 first, then retarget this stack to next.

Pending hosted jobs: none in this light run; macOS/Docker/build-smoke evidence belongs to integration. The lead still requires #748's new full hosted run before merging that parent. Real-VM check: no native runtime implementation changes. No merge performed here.

f741104

roodboi added a commit that referenced this pull request Nov 13, 2026
Small PRs currently queue native builds ahead of other work. This makes
PR CI light by default and moves platform/Docker qualification to the
selected integration run on `next`. Superseded `next` pushes cancel
themselves; `main` keeps its full, non-cancelling push behavior.

A contributor sees secret scan, typecheck, lint/privacy, relevant docs
checks and selected Linux unit files on shards 4–8. A light run skips
macOS, Docker E2E, state models and CLI/release build smoke.
Candidate/compiler host qualification is skipped unless the Rust input
rule below applies. **Ready PRs changing protected CI inputs
automatically run full ordinary PR tiers, as `ci:full` would. Ready PRs
into `main` keep the full tier, including Intel.** Missing/empty diffs
or unavailable base policy also fall back to full tiers. Ready PRs
touching non-Markdown runtime-core/config-compiler inputs (including
pins, fixtures and generated data), or any relay-guest input, also keep
Linux Candidate core and both Linux compiler hosts with Rust checks.
Their macOS jobs stay deferred. Linux unit preflight still prepares its
matching compiler. On a light run, shard 8 names selected macOS paths as
**“deferred to the integration run”** before typecheck; deferred files
run nowhere on that PR. `test` and `docs site` always report. The
required `test` fan-in now also requires the active Linux
process-lifetime job to succeed; only that job's planned out-of-scope
skip passes. Branch rules are unchanged.

- `ci:mac`: opt into selected macOS unit/native tiers under their usual
input gates.
- `ci:e2e`: opt into Docker/runtime-image/lifetime jobs under their
usual gates. Labels combine.
- `ci:full`: every unit file and ordinary native/Docker tier, state
models and build smoke. PRs into `next` and ordinary `next` integration
runs omit Intel; `main` PRs/pushes and explicit full dispatches retain
it.

Drafts retain the light deferrals even when labeled, and both fan-ins
fail with `skipped: draft`. They no longer move macOS-pool files onto
Linux. The independent shell guard enables heavy tiers for protected
inputs, including this PR's workflow/selector changes. An ordinary
selector fallback widens files within the chosen tier. State models
follow the same full-tier admission.

![CI selection and release
evidence](https://raw.githubusercontent.com/hack-dance/hack/fix/ci-macos-throughput/docs/reference/ci-test-selection.svg)

`next` selection uses trusted policy/dependencies from `event.before`
against the new tree as data. Missing/zero/non-ancestor bases, protected
inputs, old policy, failed/cancelled/running/missing predecessor CI or
API errors force full integration selection. This keeps cancelled
changes covered. `next` runs selected platform and Docker jobs after
batch merges; failures are fixed forward. Only `next` pushes share the
cancelling group; full manual runs and `main` pushes stay independent.

For a release, freeze `next` at the release head, run `gh workflow run
ci.yml --ref next -f full=true`, wait for success, then dispatch the
prerelease. Every prerelease now needs this explicit full run. The plan
refuses subset-only evidence and prints that command. If a newer scoped
push at the same SHA is running, failed or cancelled, it also names that
run and says to wait for success or `gh run rerun RUN_ID`; older full
evidence cannot override it, while a new full dispatch can supersede it
once the dispatch succeeds. Plan and publish require executed
full-selection/release-full markers, all four compiler hosts, every
shard/job successful, and required CI checks from the selected suite and
exact attempt jobs. Newer failed/cancelled/unfinished evidence cannot
expose older green. Partial reruns missing current-attempt jobs refuse
safely; use an entire workflow rerun. A moved head refuses. Stable
release preparation remains main-push-only; cancelled runs cannot
qualify.

The storage activation command file gets a temporary 20-minute per-file
budget, runs first in its shard, and remains subject to the 30-minute
shard deadline. Other files retain 12 minutes; explicit CLI limits
override the exception. This addresses run
[43313402845](https://github.com/hack-dance/hack/actions/runs/43313402845)'s
12-minute timeout with bounded headroom, not a speedup claim.
Splitting/cutting the eight-case file and refreshing its planning weight
remain the next cost follow-up.

Ten successful six-macOS-job runs on Nov 12 gave these median job
minutes (queue excluded; sums of medians are descriptive):

| Job | Before minutes |
| --- | ---: |
| Unit 1 / 2 / 3 | 17.93 / 17.67 / 14.00 |
| Candidate ARM | 19.25 |
| Compiler ARM / Intel | 5.97 / 12.22 |

The total is about 87 macOS runner-minutes. Ordinary light PRs schedule
**zero** macOS jobs. Protected-input/full-tier PRs and ordinary full
`next` runs use five instead of six, deferring Intel to
`main`/release-full. This intentionally moves Intel-only regressions on
`next` to release qualification. Bun install is only 5–6s on unit lanes,
19s on Intel; unit compiler prep 48–55s. Three candidate logs show
10.3–14.3min Cargo compilation and 3.3–4.1min Cargo test execution.
Compiler ARM transport/bundle work alone is 169s median; folding it into
the 19min candidate job saves under a minute of setup while extending
that lane, so they stay parallel.

Bun downloads, Cargo registry/target trees and Zig data are cached by
OS/arch, image OS, toolchain pins and locks, with separate
compiler/candidate targets. PR native caches restore only; trusted
push/dispatch caches save. Image updates do not create another multi-GB
entry. Builds/tests still execute; Zig is checksum-verified and
re-extracted. Structured step timings from the first full PR run show
native-cache helper steps at 0–1s, Bun cache restore at 7–11s, unit
installs at 1–2s and unit compiler preparation at 34–60s. Candidate Zig
setup was 1s plus 12s post-step; compiler post-step was 16s. Warm
hit/miss and native-save benefit remain unqualified; PR native caches do
not save. This PR's protected-input guard now schedules its own full
ordinary PR run, which the lead requires before merging this gating
change.

Runner options, checked Nov 12: the account is Team; its published macOS
cap is five, shared by standard/larger runners. The sample observed five
concurrent Mac jobs; custom support overrides are not exposed by the
limits API and remain unverified. Larger Intel: 12 CPU/30GB, $0.077/min.
M2 Pro xlarge: 5 CPU (+8 GPU)/14GB, $0.102/min. Larger runners are paid
even for public repos; current standard public jobs are free.
[Limits](https://docs.github.com/en/actions/reference/limits),
[specs](https://docs.github.com/en/actions/reference/runners/larger-runners),
[prices](https://docs.github.com/en/billing/reference/actions-runner-pricing).

A spare Apple Silicon Mac buys queue capacity, not a lower current
hosted bill. Use ephemeral isolated VM/JIT workers labeled `self-hosted,
macOS, ARM64, trusted-ci`, admitted externally for trusted next/main
pushes and manual runs only. Never fork PRs, other PR events or
pull_request_target. Labels/editable workflow conditions are not access
control: restrict runner-group/workflow routing and event/ref admission,
discard each VM/checkout, and retain no host credentials. No
registration, paid runner or account changes here.

Validation on the Batch A integrated tree: 289 focused CI/shard/release
tests passed (1,734 assertions). The 65-file source-contract gate:
passed 1,315 tests with 26 conditional skips and zero failures (13,731
assertions). No daemon before/after. Pinned typecheck, lint/privacy,
explicit script/test lint, actionlint 1.7.12 and commit-range gitleaks
8.30.1 pass; existing complexity warnings remain. The persistent
reviewer is clean on exact tree
`98ab31ac29bfcb971058590d7cb750b906b0e69b`.

After next.16 published and #747 landed, normal merge
`ae94e5fb602240ee381d4f112edeb4f76813fd30` brought `next` at
`e7a70393e1bae3d3e453ac82bf20fad6273277b3` into this PR. The
model-coverage conflict retains this PR's CI-evidence row and #747's
updated log-follow row verbatim. Batch A implementation is unchanged.
The existing startup model README points to the three merged regression
files. The source-only selector regression requires a proper subset from
the real tree.

For comparison, the prior full PR run
[43371306048](https://github.com/hack-dance/hack/actions/runs/43371306048)
passed all 26 jobs at `56075e67`. Its five Mac jobs took
15.28/15.52/14.28 minutes for unit shards, 20.00 for Candidate ARM and
6.38 for compiler ARM: **71.47 Mac runner-minutes**, versus the earlier
six-job baseline's sum of medians, 87.03. These are different trees and
unmatched runs; they do not establish a cache speedup. Mac shard 1
waited 109.53 minutes after scope, so queue pressure remained. The
latest revision still needs its own hosted result.

This protected-input PR must run its full ordinary PR tier at the new
head. Ready follows green Linux jobs, clean reviews and handled bot
threads, naming any pending heavy jobs. The lead requires this PR's full
hosted run before merge. Manual dispatch/partial-rerun and warm-cache
qualification remain post-merge work; this PR adds no native runtime
implementation changes relative to `next`. After this PR merges, #688,
#691, #750 and #751 will be retargeted to `next`; #750 must preserve
#749's optimized identity engine and assertions. HACK-2111.

---------

Co-authored-by: Kit <344842+roodboi@users.noreply.github.com>
Base automatically changed from fix/ci-macos-throughput to next November 13, 2026 10:48

roodboi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Retarget audit after #748 merged: this PR now targets next; head remains f74110436b67cfeb6c9dc29b6013402124cc68ac.

Keeping it open. The E2E CI contract rewrite and tests/e2e/ci-results.{mmd,svg} are not in next at 021b16bcfd13c48b14bf2677eba2cffe62453ea2; the diagrams are absent and the base README still describes four unit shards. #748 supplies the workflow, not these three documentation paths.

Before integration, update this branch's light-tier wording for the Linux Rust floor and active lifetime fan-in from e2c73c6122a545c13fdee2c89a6ce987d9e98b4b. Preserve the newer compose-short-oneshot README section from #758 (87a6df56) when combining the docs. No freshness-only push or superseded closure.

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Cleanup audit against next at a262d42762ae7b395921564203192b75169733d0: retain as draft; three documentation changes remain unlanded.

Commit/change mapping:

Reason to keep open: the README refresh and source/rendered diagram are still unique. They also need reconciliation with later policy changes, especially #777's active/pending next-run rule, before being advertised as ready. Marking draft preserves that work without falsely treating it as merged or current. Tracking context: HACK-2111.

@roodboi
roodboi marked this pull request as draft November 14, 2026 01:13

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Retained unlanded work is now explicitly tracked in HACK-2231, under HACK-2111, with acceptance criteria and links to the full commit-by-commit mapping above. This PR stays open as draft because its unique changes have no verified complete successor on next; no patch is represented as landed. The ticket records the refresh/qualification or explicit-retirement decision needed before closure.

@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Closing unmerged as requested by the lead; retained work is transferred to HACK-2231, which is still Backlog. This closes the stale PR, not that ticket or its acceptance criteria. The branch is preserved; no unlanded patch is represented as merged or retired.

Reverified against next at 5965b23: #748 includes both inherited policy commits, and its squash e2c73c6 is an ancestor. Unique net paths and missing artifacts remain as listed below. This supersedes the earlier keep-open decision now that the lead has explicitly requested closure and each gap has a tracking ticket.

Commit/change mapping:

Uncovered work above is assigned to HACK-2231; the mapping does not claim a complete successor PR for it.

@roodboi roodboi closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant