[Bug]: Run metrics with `docker: true` fail until the first nested container starts

[r4victor]

Steps to reproduce

1. Start a run with docker: true that does some work before starting a nested container, then keeps a nested container running:

   type: task
   name: dind-metrics
   docker: true
   commands:
     - sleep 300
     - docker run -d --name stress polinux/stress stress --vm 1 --vm-bytes 4G --vm-keep
     - sleep infinity

2. Watch the dstack server logs during the first five minutes.

3. After the nested container is up, run dstack metrics dind-metrics.

Actual behaviour

Until the first nested container starts, the server logs the traceback below on every metrics collection cycle and stores no metrics points for the job.

Once a nested container is running, collection resumes, but the reported CPU and memory cover only the runner, dockerd and processes started directly by the job's shell. Everything running in nested containers is excluded, which is where the workload of a docker: true run usually lives.

The mechanism: start-dockerd moves every process in the container's root cgroup, including dstack-runner, into a child cgroup /sys/fs/cgroup/dind. In cgroup v2, a child cgroup only gets memory.* files when the memory controller is enabled in the parent's cgroup.subtree_control, which start-dockerd does not do, so /sys/fs/cgroup/dind/memory.current does not exist after the move. cpu.stat exists on every non-root cgroup regardless of controllers, which is why the CPU read succeeds and the memory read is the first to fail.

dstack/docker/dind/start-dockerd

Lines 95 to 99 in aaeb201

	if [[ -f /sys/fs/cgroup/cgroup.controllers ]]; then
	local group=/sys/fs/cgroup/dind
	mkdir -p ${group}
	xargs -rn1 < /sys/fs/cgroup/cgroup.procs > ${group}/cgroup.procs || true
	fi

When runc creates the cgroup for the first nested container under /sys/fs/cgroup/docker/, it enables the controllers in the ancestors' cgroup.subtree_control. From then on /sys/fs/cgroup/dind/memory.current exists, but it accounts for the dind subtree only. The nested containers live under the sibling /sys/fs/cgroup/docker/.

The runner resolves its cgroup from /proc/self/cgroup and reads from that directory with no fallback:

dstack/runner/internal/runner/metrics/metrics.go

Lines 41 to 57 in aaeb201

	// so we detect the current group each time.
	cgroupPathname, err := getProcessCgroupPathname(ctx, "/proc/self/cgroup")
	if err != nil {
	return nil, fmt.Errorf("get cgroup pathname: %w", err)
	}
	cgroupPath := path.Join(s.cgroupMountPoint, cgroupPathname)
	timestamp := time.Now()
	cpuUsage, err := s.GetCPUUsageMicroseconds(cgroupPath)
	if err != nil {
	return nil, err
	}
	memoryUsage, err := s.GetMemoryUsageBytes(cgroupPath)
	if err != nil {
	return nil, err
	}
	memoryCache, err := s.GetMemoryCacheBytes(cgroupPath)
	if err != nil {

Non-dind images are unaffected because the runner stays in the container's root cgroup, which is the container's own cgroup on the host and accounts for everything in the container.

Expected behaviour

Metrics for docker: true runs are collected from the start and include nested containers.

Suggested fix: make the runner prefer the cgroup mount root (/sys/fs/cgroup) whenever memory.current exists there, and fall back to the process cgroup otherwise. Inside a container with a cgroup namespace, the mount root is the container's own cgroup, so it covers the runner, dockerd and all nested containers. On a bare host or without a cgroup namespace the root has no memory.current, so the process cgroup is still used there.

Enabling the controllers in start-dockerd the way moby's hack/dind does would only remove the error window and would still exclude nested containers.

dstack version

master (aaeb201)

Server logs

ERROR    dstack._internal.server.background.scheduled_tasks.metrics:139 Failed to collect job dind-metrics-0-0 metrics
Traceback (most recent call last):
  ...
  File ".../dstack/_internal/server/services/runner/client.py", line 349, in _raise_for_status
    raise RunnerResponseStatusError(response)
dstack._internal.server.services.runner.client.RunnerResponseStatusError: GET /api/metrics: 500: could not read memory usage: open /sys/fs/cgroup/dind/memory.current: no such file or directory

Additional information

Related: #3398 has the same runner error on Kubernetes with privileged: true, but for a different reason (no cgroup namespace, so /sys/fs/cgroup is the host root). The fallback described above would not fix that case.