Skip to content

Collect DinD metrics from the container cgroup root - #4272

Closed
pentaoa wants to merge 1 commit into
dstackai:masterfrom
pentaoa:issue_4267_dind_metrics
Closed

Collect DinD metrics from the container cgroup root#4272
pentaoa wants to merge 1 commit into
dstackai:masterfrom
pentaoa:issue_4267_dind_metrics

Conversation

@pentaoa

@pentaoa pentaoa commented Sep 8, 2026

Copy link
Copy Markdown

Summary

Fixes #4267.

Prefer the cgroup mount root for runner CPU and memory accounting when its memory.current exists. In a container cgroup namespace this includes the whole run, including nested containers under the sibling docker/ group after start-dockerd moves the runner into dind/.

When the mount exposes the host root, which has no memory.current, retain the existing /proc/self/cgroup resolution. Resolve this on each collection, preserving support for process cgroup changes. A failure to inspect the root other than a missing file is reported rather than silently selecting a different accounting scope.

The change follows the approach suggested in the issue. It does not enable controllers, modify the Docker startup script, change GPU collection, or restore cgroup v1 support. The namespace behavior addressed by #3402 is retained; the separate privileged Kubernetes limitation described in #4267 is not claimed as fixed.

Validation

Filesystem-fixture regression cases cover:

  • ordinary containers whose process cgroup is /;
  • DinD before the first nested container, when child memory files are absent;
  • DinD after child memory files appear, with deliberately different root and child counters;
  • host-namespace fallback to the process cgroup;
  • missing proc metadata errors when fallback is required.

An additional test exercises GetSystemMetrics using root CPU, memory usage, and cache counters and checks the resulting working set.

With the original path-selection behavior extracted unchanged, both DinD cases failed because they selected /dind instead of the mount root. After the fix:

go test -race ./internal/runner/metrics -run '^(TestGetMetricsCgroupPath|TestGetProcessCgroup|TestGetSystemMetrics_ContainerRoot)' -count=1
ok github.com/dstackai/dstack/runner/internal/runner/metrics 2.199s

go vet ./internal/runner/metrics
# passed

Changed Go files are gofmt-formatted and git diff --check passes. The metrics test package also cross-compiles for Linux/amd64 (compile check only, not Linux execution).

Validation ran on macOS using temporary proc/cgroup fixtures. I did not launch nested Docker containers, run GPU workloads, execute the full runner suite (existing collector-constructor tests require Linux procfs), or run golangci-lint locally.

AI-assisted implementation and tests; I inspected the DinD startup code, previous cgroup-path fix, and current collector, and ran the focused validation above.

@peterschmidt85

Copy link
Copy Markdown
Contributor

@pentaoa we don't accept PRs if the implementation is not discussed first and the author that submits the PR is not actively using dstack.

@pentaoa
pentaoa deleted the issue_4267_dind_metrics branch September 8, 2026 11:33
@pentaoa

pentaoa commented Sep 8, 2026

Copy link
Copy Markdown
Author

Thanks for clarifying, and apologies for not discussing the implementation with you first. I’m not currently an active dstack user, so I understand these PRs don’t meet your contribution requirements. I’ll hold off on further submissions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Run metrics with docker: true fail until the first nested container starts

2 participants