Skip to content

Hosted Windows jobs intermittently hang to the 45-minute limit #842

Open

Description

@coneilen

Describe the bug
Two automatic pull-request Windows jobs stopped making observable progress and were cancelled at their execution limits. Other jobs on the same heads passed. Treat this as a hosted-runner hang signature pending contrary log evidence, not a demonstrated assertion failure or a proven root cause.

The title uses the observed terminal-job elapsed time. The annotations and measured durations differ and are recorded separately below; they must not be conflated.

PR / exact head Workflow and failed job Measured job duration Execution-limit annotation Last recorded active step
#835 / 05720f40c38a223a69aa519928581ca138789ff7 Run 38841630690, integration job 116782866747 2026-10-16T02:40:43Z to 03:35:43Z: 55m00s Maximum execution time 45m0s Verify shell build, pinned-provider smoke, and live UI Automation, active since 02:44:09Z
#841 / b285440a52c56b0de5ae3fd3d38ca867e6188641 Run 38851473200, terminal/contracts job 116813450028 2026-10-16T04:36:28Z to 05:21:28Z: 45m00s Maximum execution time 40m0s Run Windows port validation, active since 04:37:10Z

Both job conclusions are cancelled; their required aggregates then failed. These are the last step names from the job metadata, not last internal log lines. Both cancelled-job log downloads returned HTTP 404 / unavailable. For #841, the response was BlobNotFound. The precise internal hang location, including whether UIA was responsible, remains unobserved.

To reproduce

  1. Inspect the automatic pull-request runs and exact heads above; there is no deterministic local reproduction established.
  2. Compare the timeout annotations, job start/end times and incomplete validation steps. Post-failure diagnostic-retention steps did not complete.
  3. Compare with unchanged-source successful runs: Add guarded Windows command execution wrapper #835's one authorized integration rerun 116800051462 passed in 13m11s, within its usual 12–14-minute range. Add offscreen paint evidence pipeline for #764 cards #841's same complete tested tree 80587bf0f9f433e9ff20a8bf0dc4a7018f4a251e previously passed terminal/contracts in 3m16s on run 38848923251. The later publication merge changed commit metadata/parents, not that tree or its six reviewed source files.

Expected behavior
Hosted validation should finish in its normal bounded interval or fail promptly with the actual primary error and preserved diagnostics, rather than hang until cancellation with the terminal log unavailable.

Screenshots
Not applicable. This is a hosted CI execution/diagnostic-retention issue; no screenshots or private runtime artifacts are attached.

Environment

  • GraphCode version: the two exact PR heads above; Windows validation only.
  • macOS version: not applicable; these jobs run on GitHub-hosted Windows runners.
  • Backend: no paid model/backend session involved in this report.

Additional context

Activity

  1. coneilen commented on Oct 16, 2026

    @coneilen
    CollaboratorAuthor

    One operator-authorized diagnostic rerun is being requested for PR #841 on unchanged head b285440, targeting terminal/contracts job 116813450028 in run 38851473200 and its dependent aggregate. The original job ran 45m00s; its annotation says a 40m0s execution limit. Last recorded active step: Run Windows port validation. The cancelled-job log returned HTTP 404 BlobNotFound, so the internal last log line is unavailable. This consumes the one-per-head diagnostic allowance for this signature. If it hangs or fails again, stop and report; no repeated reruns toward green. Original artifacts and six reviewed source hashes remain unchanged.

  2. coneilen commented on Oct 16, 2026

    @coneilen
    CollaboratorAuthor

    The one authorized diagnostic on unchanged PR #841 head b285440 passed: terminal/contracts job 116826582621 ran from 2026-10-16T05:25:38Z for 3m32s; dependent aggregate 116827531925 passed. Other successful records retain their original timestamps and were not re-executed. The original 45m00s cancellation, its 40m0s annotation and HTTP404 log limitation remain preserved. Treat as hosted-runner hang under the operator disposition, not a proved internal root cause. The one-per-head diagnostic allowance is consumed; no further rerun is authorized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions