[Bug]: max_duration signals the wrapper shell, not the job (workload survives and the runner never reports the timeout)

[juancarlosm]

Steps to reproduce

Submit a service with a short max_duration on a backend where the runner execs the workload directly (reproduced on kubernetes). Any long-running process shows it; the image is incidental:

type: service
name: poc-maxdur
image: nginxinc/nginx-unprivileged
port: 8080
replicas: 1
commands:
  - nginx -g 'daemon off;'
max_duration: 2m
resources:
  cpu: 1..
  memory: 512MB..

Past the deadline, inspect the job container: ps -eo pid,ppid,stat,etime,args.

Actual behaviour

The wrapper shell is signalled and dies; the workload survives and keeps serving.

Before the deadline (t+101s):

  PID  PPID STAT ELAPSED COMMAND
  970   945 Ss     01:15 /bin/sh -i -c nginx -g 'daemon off;'
  971   970 S+     01:15 nginx: master process
  972   971 S+     01:15 nginx: worker process

After the deadline (t+161s, max_duration = 120s):

  PID  PPID STAT ELAPSED COMMAND
  970   945 Zs     02:16 [sh] <defunct>
  971     1 S      02:16 nginx: master process
 1206   971 S      00:06 nginx: worker process

The shell is an unreaped zombie; nginx is reparented to PID 1, has respawned its workers, and still answers HTTP 200 at t+235s.

Nothing reports a timeout. Runner /api/pull:

{"job_states":[{"state":"running","termination_reason":"","termination_message":""}]}

Server: status: running, max_duration: 120, termination_reason: None at t+235s. A 349-line runner log at --log-level 6 contains zero lines matching max.duration.

Root cause

1. The signal reaches the wrong process. src/dstack/_internal/server/services/jobs/configurators/base.py:264 wraps user commands as entrypoint = [self._shell(), "-i", "-c"]. The -i defeats the shell's single-command exec optimisation, so sh forks the workload rather than becoming it. runner/internal/runner/executor/executor.go:222 arms the timeout, and the cancel func at executor.go:505 signals cmd.Process — the shell — only. Nothing signals the process group, so the workload is untouched and gets reparented to PID 1.

2. The timeout is never reported. Termination is set inside if err := ex.execJob(timeoutCtx, ...); err != nil { select { case <-timeoutCtx.Done(): ... } }. The shell is left unreaped, so cmd.Wait() never returns, execJob never returns, and that select is never reached. The runner does not merely fail to stop the job — it never records that it tried, so the server sees a healthy running job indefinitely.

Expected behaviour

The workload is terminated at max_duration and the job ends as terminated / max_duration_exceeded.

Possible directions (untested):

• Signal the process group (Setpgid + Kill(-pgid, ...)), or drop -i so the shell execs and the signalled process is the workload.
• Don't gate the MAX_DURATION_EXCEEDED report on execJob returning — a stalled wait currently silences the whole feature.

dstack version

Server and runner 0.21.3, kubernetes backend.

Additional information

• Also seen with a longer limit: a service ran 104 minutes against max_duration: 3600. The minimal case above reached 4.96x its limit before being stopped by hand.
• Dockerized backends may be unaffected, since tearing down the container kills the whole tree regardless of which process was signalled. Not verified.
• Likely why it went unnoticed: since Set max_duration to off by default for all run configurations #2357 max_duration defaults to off, and ServiceConfigurator._default_max_duration() returns None, so only users who set it explicitly reach this path. stop_duration and normal dstack stop are unaffected — those tear the instance down.

[un-def]

Related: #2233