Repository navigation
Hosted Windows jobs intermittently hang to the 45-minute limit #842
Description
Describe the bug
Two automatic pull-request Windows jobs stopped making observable progress and were cancelled at their execution limits. Other jobs on the same heads passed. Treat this as a hosted-runner hang signature pending contrary log evidence, not a demonstrated assertion failure or a proven root cause.
The title uses the observed terminal-job elapsed time. The annotations and measured durations differ and are recorded separately below; they must not be conflated.
| PR / exact head | Workflow and failed job | Measured job duration | Execution-limit annotation | Last recorded active step |
|---|---|---|---|---|
#835 / 05720f40c38a223a69aa519928581ca138789ff7 |
Run 38841630690, integration job 116782866747 | 2026-10-16T02:40:43Z to 03:35:43Z: 55m00s |
Maximum execution time 45m0s | Verify shell build, pinned-provider smoke, and live UI Automation, active since 02:44:09Z |
#841 / b285440a52c56b0de5ae3fd3d38ca867e6188641 |
Run 38851473200, terminal/contracts job 116813450028 | 2026-10-16T04:36:28Z to 05:21:28Z: 45m00s |
Maximum execution time 40m0s | Run Windows port validation, active since 04:37:10Z |
Both job conclusions are cancelled; their required aggregates then failed. These are the last step names from the job metadata, not last internal log lines. Both cancelled-job log downloads returned HTTP 404 / unavailable. For #841, the response was BlobNotFound. The precise internal hang location, including whether UIA was responsible, remains unobserved.
To reproduce
- Inspect the automatic pull-request runs and exact heads above; there is no deterministic local reproduction established.
- Compare the timeout annotations, job start/end times and incomplete validation steps. Post-failure diagnostic-retention steps did not complete.
- Compare with unchanged-source successful runs: Add guarded Windows command execution wrapper #835's one authorized integration rerun 116800051462 passed in 13m11s, within its usual 12–14-minute range. Add offscreen paint evidence pipeline for #764 cards #841's same complete tested tree
80587bf0f9f433e9ff20a8bf0dc4a7018f4a251epreviously passed terminal/contracts in 3m16s on run 38848923251. The later publication merge changed commit metadata/parents, not that tree or its six reviewed source files.
Expected behavior
Hosted validation should finish in its normal bounded interval or fail promptly with the actual primary error and preserved diagnostics, rather than hang until cancellation with the terminal log unavailable.
Screenshots
Not applicable. This is a hosted CI execution/diagnostic-retention issue; no screenshots or private runtime artifacts are attached.
Environment
- GraphCode version: the two exact PR heads above; Windows validation only.
- macOS version: not applicable; these jobs run on GitHub-hosted Windows runners.
- Backend: no paid model/backend session involved in this report.
Additional context
- Bounded diagnostic policy: allow one diagnostic rerun per unchanged head for this timeout signature, citing this issue. Preserve the original failed run, annotation and log-unavailability receipt. Do not apply this policy to an assertion failure or unrelated signature. If the rerun hangs or fails again, stop and report; do not repeatedly rerun toward green.
- Add guarded Windows command execution wrapper #835's authorized rerun is already consumed and passed. One failed-job diagnostic rerun is authorized for Add offscreen paint evidence pipeline for #764 cards #841 on unchanged
b285440a; it has not been requested at filing time. - Add offscreen paint evidence pipeline for #764 cards #841's Swift job passed in 25m28s. Its shell, deterministic hardening and codec jobs passed, including actual codec 8/8 and scene 16/16 results with all 14 refusals retained. This does not erase the required port failure or admit the PR.
- Add guarded Windows command execution wrapper #835's older
0xC0000142startup failure is a distinct signature tracked in Scrubbed production shell startup intermittently fails with 0xC0000142 (STATUS_DLL_INIT_FAILED) on hosted runners #838; do not relabel it as this timeout. - Suggested future fix: review step-level timeouts, bounded progress/diagnostic flushing and durable log retention before the outer job deadline. This is a proposal only: no worker, workflow change or implementation is assigned.
- The elapsed-time versus annotation-limit discrepancy is retained rather than explained speculatively.
Activity
coneilen commentedon Oct 16, 2026
One operator-authorized diagnostic rerun is being requested for PR #841 on unchanged head b285440, targeting terminal/contracts job 116813450028 in run 38851473200 and its dependent aggregate. The original job ran 45m00s; its annotation says a 40m0s execution limit. Last recorded active step: Run Windows port validation. The cancelled-job log returned HTTP 404 BlobNotFound, so the internal last log line is unavailable. This consumes the one-per-head diagnostic allowance for this signature. If it hangs or fails again, stop and report; no repeated reruns toward green. Original artifacts and six reviewed source hashes remain unchanged.
coneilen commentedon Oct 16, 2026
The one authorized diagnostic on unchanged PR #841 head b285440 passed: terminal/contracts job 116826582621 ran from 2026-10-16T05:25:38Z for 3m32s; dependent aggregate 116827531925 passed. Other successful records retain their original timestamps and were not re-executed. The original 45m00s cancellation, its 40m0s annotation and HTTP404 log limitation remain preserved. Treat as hosted-runner hang under the operator disposition, not a proved internal root cause. The one-per-head diagnostic allowance is consumed; no further rerun is authorized.