Skip to content

Close the logs-done channels only once - #4275

Open
un-def wants to merge 1 commit into
masterfrom
pr_runner_idempotent_logs_done_channels
Open

Close the logs-done channels only once#4275
un-def wants to merge 1 commit into
masterfrom
pr_runner_idempotent_logs_done_channels

Conversation

@un-def

@un-def un-def commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Previously, pullGetHandler closed pullDoneCh on every request it served in the WaitLogsFinished state, and streamJobLogs closed wsDoneCh on every stream that drained after shutdown. Both assumed they run exactly once per server lifetime.

Neither assumption holds. Server.stop() sets WaitLogsFinished as soon as the job is asked to stop, while the server keeps serving until the executor returns -- which does not happen while a process left behind by the job holds the pty slave open, since io.Copy from the master then never sees EOF. The dstack server goes on polling, and the second pull panics:

http: panic serving 127.0.0.1:50670: close of closed channel
...
api.(*Server).pullGetHandler.func1()
    runner/internal/runner/api/http.go:191

/logs_ws has the same defect without needing a stuck executor: nothing limits the number of connections, each is drained by its own goroutine, and two draining together close the channel twice. Both handlers hold only a read lock, so concurrent requests can observe the state together as well.

net/http recovers the panic per connection, so the runner survives it but serves a broken pull endpoint from then on: the job state stops being reported and the run hangs.

Now both channels are closed through sync.Once helpers. The api package had no tests; the ones added here cover the repeated and the concurrent case.

Why the executor fails to return is left alone here.

Previously, `pullGetHandler` closed `pullDoneCh` on every request it
served in the WaitLogsFinished state, and `streamJobLogs` closed
`wsDoneCh` on every stream that drained after shutdown. Both assumed
they run exactly once per server lifetime.

Neither assumption holds. `Server.stop()` sets WaitLogsFinished as soon
as the job is asked to stop, while the server keeps serving until the
executor returns -- which does not happen while a process left behind
by the job holds the pty slave open, since `io.Copy` from the master
then never sees EOF. The dstack server goes on polling, and the second
pull panics:

    http: panic serving 127.0.0.1:50670: close of closed channel
    ...
    api.(*Server).pullGetHandler.func1()
        runner/internal/runner/api/http.go:191

`/logs_ws` has the same defect without needing a stuck executor:
nothing limits the number of connections, each is drained by its own
goroutine, and two draining together close the channel twice. Both
handlers hold only a read lock, so concurrent requests can observe the
state together as well.

net/http recovers the panic per connection, so the runner survives it
but serves a broken pull endpoint from then on: the job state stops
being reported and the run hangs.

Now both channels are closed through `sync.Once` helpers. The api
package had no tests; the ones added here cover the repeated and the
concurrent case.

Why the executor fails to return is left alone here.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant