Skip to content

Feat: show_metrics() and stream_logs() helper functions#6002

Merged
zhaoqizqwang merged 2 commits into
aws:master-nova-follow-upsfrom
ehsu3:job-monitoring
Jul 9, 2026
Merged

Feat: show_metrics() and stream_logs() helper functions#6002
zhaoqizqwang merged 2 commits into
aws:master-nova-follow-upsfrom
ehsu3:job-monitoring

Conversation

@ehsu3

@ehsu3 ehsu3 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Issue #, if available: 'Monitoring training/eval jobs with CW logs' task

Description of changes:

  • Added cloudwatch_metrics.py with CW log parsing for training_loss, lr, reward_score
  • Supports SMTJ and SMHP
  • Added show_metrics() on BaseTrainer with start_time/end_time, step range filtering
  • Added stream_logs() on BaseTrainer (SMTJ auto-stops on completion, SMHP via Ctrl+C)
  • show_metrics() generates separate subplots per metric
  • Added pandas to pyproject.toml for generating graphs
  • Added 18 unit tests covering parsing, fetching, end-to-end, and streaming
  • The two functions can be used on a 'trainer' object that has a job name attached. If the session ends, the user can make a placeholder trainer object and add the job name so it's able to use the functions again.
  • These functions only work training jobs since eval jobs don't have metrics stored in the CW logs and the logs don't have enough information for stream_logs() to provide information on progress. However, the latter can be added if we want.
  • Added changes requested from initial PR (clean up code, use logger > print, notebook example, update parameters)

Tests:

  • Ran both methods on some completed and in-progress jobs to make sure they're able to capture the metrics and logs properly.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.


return final_hyperparameters

def show_metrics(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something we need to gate to the Nova models?
The log streaming should still work for OSS models, but Nova models metric parsing is Nova model specific

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good catch! I'm not familiar with the OSS model logs but they'd likely be different from Nova. I'll update the logic there, thanks!

@ehsu3 ehsu3 force-pushed the job-monitoring branch from 328ae42 to 7317c54 Compare July 9, 2026 15:57
@zhaoqizqwang zhaoqizqwang merged commit c0aead5 into aws:master-nova-follow-ups Jul 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants