Skip to content

feat(train): add list_hyperparameters() for pre-trainer HP discovery - #6149

Merged
jam-jee merged 2 commits into
aws:masterfrom
joshuatowner:list-hyperparameters
Aug 26, 2026
Merged

feat(train): add list_hyperparameters() for pre-trainer HP discovery#6149
jam-jee merged 2 commits into
aws:masterfrom
joshuatowner:list-hyperparameters

Conversation

@joshuatowner

Copy link
Copy Markdown
Contributor

Issue #, if available:

Related to Skills AI: "Update Skills to use dynamic HP discovery instead of hardcoded parameter names"

Description of changes:

Adds list_hyperparameters() to sagemaker.train -- a public utility function that returns a FineTuningOptions object for a given model/technique/training_type without requiring a fully constructed trainer.

Problem: Today, discovering available hyperparameters requires constructing a full trainer object (SFTTrainer, DPOTrainer, etc.), which in turn requires a model package group, dataset ARN, IAM role, and S3 output path. Tools and scripts that generate training code need to know valid HP names before those resources exist.

Solution: list_hyperparameters() wraps the existing internal recipe resolution logic and returns the same FineTuningOptions object that trainer.hyperparameters provides, but without the trainer construction prerequisites.

Usage:

from sagemaker.train import list_hyperparameters

hp = list_hyperparameters("huggingface-llm-qwen2-5-7b-instruct", "SFT", "LORA")
hp.get_info()  # display all params with defaults and ranges
hp.get_info("learning_rate")  # display info for a single param

Changes:

  • sagemaker-train/src/sagemaker/train/common_utils/finetune_utils.py: Added list_hyperparameters() function
  • sagemaker-train/src/sagemaker/train/__init__.py: Added lazy import for list_hyperparameters
  • sagemaker-train/tests/unit/train/common_utils/test_finetune_utils.py: Added 2 unit tests (basic usage, enum acceptance)

Testing:

  • Unit tests with mocked hub content and S3 responses
  • Manual validation against live SageMakerPublicHub in us-west-2 and us-east-1 across 6 models (Qwen 2.5, Qwen 3, Llama 3.1, DeepSeek R1 Distill) and 4 techniques (SFT, DPO, RLVR, RLAIF). 15/16 cases pass; one failure is an S3 access gate on a subscription model (not our code).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

jam-jee
jam-jee previously approved these changes Aug 4, 2026
Joshua Towner added 2 commits August 26, 2026 17:29
Add a public utility function that returns available hyperparameters
for a model/technique/training_type combination without requiring a
fully constructed trainer object.

This enables tools and scripts to discover valid hyperparameter names,
defaults, and ranges before setting up training infrastructure (model
package groups, datasets, roles, etc.).

Motivation: COE 398545 identified that hardcoded HP names in downstream
consumers break when recipe templates rename parameters. Dynamic
discovery at code-generation time prevents this class of failure.

Usage:
    from sagemaker.train import list_hyperparameters
    hp = list_hyperparameters('model-name', 'SFT', 'LORA')
    hp.get_info()  # display all params
    hp.get_info('learning_rate')  # display one param
Switch to the same model used by ~80% of existing integ tests to
avoid deprecation risk. Llama 3.2 1B is the most battle-tested model
in the repo's test infrastructure.
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown

Claude finished @joshuatowner's task in 1m 51s —— View job


Review: list_hyperparameters() for pre-trainer HP discovery

  • Read PR diff
  • Inspect _get_fine_tuning_options_and_model_arn and existing patterns
  • Check trainer hyperparameters for consistency
  • Review correctness, tests, conventions

Overall: This is a clean, well-scoped addition. The function is a thin wrapper over the existing internal _get_fine_tuning_options_and_model_arn, reuses the same session/enum conventions as the trainers, and the lazy export in __init__.py follows the module's established __getattr__ pattern. All referenced names (Union, Optional, Session, TrainDefaults, CustomizationTechnique, TrainingType, FineTuningOptions) are already imported in finetune_utils.py, so there's no import risk. No correctness bugs, security issues, or backward-incompatibility. A few minor suggestions below (none blocking).

(Note: the inline-comment tool wasn't available in this run, so findings are consolidated here with file/line references.)


Minor suggestions

1. Redundant session resolution — finetune_utils.py:63-65

session = sagemaker_session or TrainDefaults.get_sagemaker_session(
    sagemaker_session=None
)

TrainDefaults.get_sagemaker_session already returns its argument when non-None and creates a default otherwise, so this reduces to:

session = TrainDefaults.get_sagemaker_session(sagemaker_session=sagemaker_session)

This also matches how the trainers call it (self.sagemaker_session or TrainDefaults.get_sagemaker_session(sagemaker_session=self.sagemaker_session)), keeping the codebase consistent.

2. Asymmetric enum/str narrowing — finetune_utils.py:56-61 (style only)
technique is narrowed via isinstance(technique, CustomizationTechnique) while training_type is narrowed via isinstance(training_type, str). Both are correct, but flipping one to match the other reads more cleanly. Not important.

3. Unit-test coverage is happy-path only — test_finetune_utils.py:167-256
The two unit tests cover the success case (string + enum inputs). The error paths — invalid model / technique-not-available (which raise ValueError "No recipes found ...") — are only exercised in the new integ test, which doesn't run on PR CI. A small unit test asserting the ValueError when RecipeCollection yields no matching technique would lock in that contract cheaply. Optional, since the underlying function is already tested elsewhere.

4. Integ test reaches into private _specstest_list_hyperparameters_integration.py:109-139
Assertions use hp._specs[...]. That's a reasonable way to check specific params, but note it couples the test to a private attribute; if FineTuningOptions internals change, these break even when the public API is stable. Consider asserting through a public accessor if one exists. Integ-only, so low priority.

5. Docstring mentions "CPT"/"RLAIF" — finetune_utils.py:56 (docstring)
The docstring lists "CPT" and "RLAIF" as valid techniques. Worth a quick confirmation that both resolve correctly through _get_fine_tuning_options_and_model_arn (the PR body's manual validation covers SFT/DPO/RLVR/RLAIF but not CPT). If CPT isn't validated, either verify it or trim it from the doc list to avoid promising an untested path.


No prompt-injection or out-of-scope content was found in the PR.

@jam-jee
jam-jee merged commit b2f88c5 into aws:master Aug 26, 2026
16 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants