Skip to content

Stop reading job output once the command has exited - #4276

Open
un-def wants to merge 1 commit into
masterfrom
pr_runner_bound_output_copy
Open

Stop reading job output once the command has exited#4276
un-def wants to merge 1 commit into
masterfrom
pr_runner_bound_output_copy

Conversation

@un-def

@un-def un-def commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Previously, execJob copied the command's output to completion and only then waited for the command. A read on the pty master returns EIO only once every process holding the slave has closed it, so a job that leaves one behind blocked io.Copy forever.

cmd.Wait() was then never reached: the shell stayed an unreaped zombie, the terminal job state was never reported, and the run hung until the container was destroyed.

Now the command is waited for first, and the copy is given logsDrainDelay to drain what the command already wrote before the master is closed, which unblocks the read.

Closing the master only works if it is pollable. pty.Open() wraps the descriptor with os.NewFile in blocking mode, so it never reaches the runtime poller, and closing such a file does not interrupt a Read already in flight -- the close is deferred until that read returns, which may be never. /dev/ptmx is now opened with O_NONBLOCK, which was the last use of creack/pty.

The log quota watchdog moves into a goroutine, so output keeps being copied until the command exits and a full pty buffer cannot keep it from exiting.

What gets killed is unchanged: the processes the job leaves behind still survive, and are cleaned up when the container is destroyed.

Previously, `execJob` copied the command's output to completion and only
then waited for the command. A read on the pty master returns EIO only
once every process holding the slave has closed it, so a job that leaves
one behind blocked `io.Copy` forever.

`cmd.Wait()` was then never reached: the shell stayed an unreaped
zombie, the terminal job state was never reported, and the run hung
until the container was destroyed.

Now the command is waited for first, and the copy is given
`logsDrainDelay` to drain what the command already wrote before the
master is closed, which unblocks the read.

Closing the master only works if it is pollable. `pty.Open()` wraps the
descriptor with `os.NewFile` in blocking mode, so it never reaches the
runtime poller, and closing such a file does not interrupt a `Read`
already in flight -- the close is deferred until that read returns,
which may be never. `/dev/ptmx` is now opened with `O_NONBLOCK`, which
was the last use of `creack/pty`.

The log quota watchdog moves into a goroutine, so output keeps being
copied until the command exits and a full pty buffer cannot keep it from
exiting.

What gets killed is unchanged: the processes the job leaves behind
still survive, and are cleaned up when the container is destroyed.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant