Skip to content

sites:list leaks ~10 zombie processes/second when stdout pipe closes early (exhausts cgroup PID limit) #8458

Description

@agencia-digd

Describe the bug

When the reader of netlify sites:list's stdout closes the pipe early (| head, | grep -m1, | less and quitting), the CLI does not exit. It enters a retry loop, and every iteration spawns npm -v, yarn -v and npm get prefix --global (the envinfo call in dist/commands/main.js) without ever reaping them.

The result is a zombie process leak at a steady ~10 zombies/second, indefinitely. Zombies consume no memory, only PIDs — so every memory dashboard stays green while the PID table fills up.

On our machine the CLI ran for 58 minutes before anyone noticed, accumulating 19,093 zombies. That exhausted pids.max of the systemd cgroup the command was running in, and from that point no process on that cgroup could fork at all:

pids.current = pids.max = 19144
pids.events: max 14737        # 14,737 forks refused

Unrelated services in the same cgroup started failing with EAGAIN on spawn, which is how we found it — the CLI itself gave no error output.

The leak is proportional to account size: it reproduces on an account with 321 sites, where the full listing is far longer than what a short pipe consumes. I did not manage to trigger it on a listing that fits within the reader's window.

Steps to reproduce

On an account with enough sites that the listing exceeds what the reader consumes (321 in our case):

netlify sites:list 2>&1 | head -30

head exits after 30 lines; the CLI keeps running. Measure from another shell:

while :; do
  echo "zombies=$(ps -eo stat= | grep -c Z) spawned=$(pgrep -cf 'npm -v|npm get prefix|yarn -v')"
  sleep 5
done

Observed (contained in a systemd-run --scope -p TasksMax=3000 so it could not take the host down again):

t+25s  zombies=15    spawned=27
t+35s  zombies=138   spawned=28
t+45s  zombies=284   spawned=23
t+55s  zombies=513   spawned=19

Left alone, this grows without bound until the PID limit of the cgroup (or the system) is reached.

The spawned npm/yarn children that are still alive sit in futex_wait with their stdio on socketpairs that nothing reads, and they do not exit when the CLI is killed — they have to be cleaned up by hand afterwards.

Writing to a file instead of a pipe is fine:

netlify sites:list > sites.txt 2>&1   # no leak

Expected: netlify sites:list receives EPIPE/SIGPIPE and exits, as most CLIs do when their output is piped to head.

Configuration

No netlify.toml involved — the command was run outside any project directory. Account has 321 sites across one team.

Environment

  • netlify-cli/26.0.0 linux-x64 node-v26.8.1
  • Node.js v26.8.1
  • Ubuntu, Linux 6.8.0-138-generic
  • Installed globally via npm (/usr/lib/node_modules/netlify-cli)

Sidenote for anyone who lands here mid-incident: don't clean up with pkill -f netlify. The -f pattern matches the full command line of the shell running the pkill itself, so it kills the caller before the target. Kill the PID directly.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions