Stop reading job output once the command has exited - #4276
Open
un-def wants to merge 1 commit into
Open
Conversation
Previously, `execJob` copied the command's output to completion and only then waited for the command. A read on the pty master returns EIO only once every process holding the slave has closed it, so a job that leaves one behind blocked `io.Copy` forever. `cmd.Wait()` was then never reached: the shell stayed an unreaped zombie, the terminal job state was never reported, and the run hung until the container was destroyed. Now the command is waited for first, and the copy is given `logsDrainDelay` to drain what the command already wrote before the master is closed, which unblocks the read. Closing the master only works if it is pollable. `pty.Open()` wraps the descriptor with `os.NewFile` in blocking mode, so it never reaches the runtime poller, and closing such a file does not interrupt a `Read` already in flight -- the close is deferred until that read returns, which may be never. `/dev/ptmx` is now opened with `O_NONBLOCK`, which was the last use of `creack/pty`. The log quota watchdog moves into a goroutine, so output keeps being copied until the command exits and a full pty buffer cannot keep it from exiting. What gets killed is unchanged: the processes the job leaves behind still survive, and are cleaned up when the container is destroyed. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously,
execJobcopied the command's output to completion and only then waited for the command. A read on the pty master returns EIO only once every process holding the slave has closed it, so a job that leaves one behind blockedio.Copyforever.cmd.Wait()was then never reached: the shell stayed an unreaped zombie, the terminal job state was never reported, and the run hung until the container was destroyed.Now the command is waited for first, and the copy is given
logsDrainDelayto drain what the command already wrote before the master is closed, which unblocks the read.Closing the master only works if it is pollable.
pty.Open()wraps the descriptor withos.NewFilein blocking mode, so it never reaches the runtime poller, and closing such a file does not interrupt aReadalready in flight -- the close is deferred until that read returns, which may be never./dev/ptmxis now opened withO_NONBLOCK, which was the last use ofcreack/pty.The log quota watchdog moves into a goroutine, so output keeps being copied until the command exits and a full pty buffer cannot keep it from exiting.
What gets killed is unchanged: the processes the job leaves behind still survive, and are cleaned up when the container is destroyed.