Skip to content

feat(power): enable official dcgm energy lane for gb200/gb300 1p1d / 打开 gb200/gb300 官方 1P1D 能耗采集 - #2456

Draft
edwingao28 wants to merge 4 commits into
wenyao/power-pr3-multinode-consumptionfrom
wenyao/power-pr4-official-energy
Draft

feat(power): enable official dcgm energy lane for gb200/gb300 1p1d / 打开 gb200/gb300 官方 1P1D 能耗采集#2456
edwingao28 wants to merge 4 commits into
wenyao/power-pr3-multinode-consumptionfrom
wenyao/power-pr4-official-energy

Conversation

@edwingao28

Copy link
Copy Markdown
Collaborator

Draft on purpose: the producer pin points at the validated fork SHA (edwingao28/srt-slurm@6609d46). Before this leaves draft, the pin swaps to the upstream srt-slurm merge SHA (one constant per launcher) and both platforms re-run a c4 smoke.

Stacked on #2437 (multinode consumer), which stacks on #2323 (single-node).

What this adds

  • telemetry block in the two official 1P1D 8k1k recipes. gb300 uses port 19401: 9401 is already bound by the cluster-level exporter on im-gb300 nodes.
  • lane-scoped provisioning in the gb200/gb300 launchers: a run opts in only when its recipe carries an enabled dcgm-power block. Every other model/recipe keeps the exact same path as before.
  • producer pin contract: power lanes clone the pinned SHA, assert HEAD matches, and write power-producer-sha.txt. CI derives POWER_PRODUCER_SHA from that stamp; the workflow input stays as a manual override.
  • contract tests for the above (utils/test_gb200_power_official_contract.py, utils/test_gb300_power_official_contract.py).

Scope: 1P1D only. No nvidia-master.yaml change. Dispatches for this PR use exact-key test-config, not full-sweep.

Validation so far (fork pin): GB200 run 30618706258 (340.5 W/GPU avg, 193,404 J), GB300 run 30663050396 (349.3 W/GPU avg, 172,800 J), both with power_valid=1 and stored==recomputed sidecars. Local: 124 tests pass.

中文:为 gb200/gb300 官方 1P1D lane 打开 DCGM 能耗采集。recipe 声明 telemetry(gb300 用 19401 端口),launcher 按 recipe 判定是否 provision exporter 与 pin producer(非 power lane 路径不变),CI 从 launcher stamp 读取 POWER_PRODUCER_SHA。当前 pin 指向已验证的 fork SHA,upstream merge 后换 pin 并重跑双平台 c4 smoke,再转正式 review。

…in gb launchers

中文:launcher 按 recipe 判定 power lane,provision exporter 并 pin producer;非 power lane 行为不变。
…de template

中文:CI 从 launcher stamp 读取 producer SHA,workflow input 保留为手动覆盖。
中文:两个官方 1P1D recipe 声明 telemetry;gb300 用 19401 端口避开集群级 exporter。
中文:契约测试覆盖 lane 检测、恰好两个 power recipe、pin 单一来源与 stamp 一致性。
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant