Repository navigation
fix(hpo): score all trials of a search on one validation split - #632
Open
jorisstrakeljahn wants to merge 1 commit into
Open
jorisstrakeljahn wants to merge 1 commit into
jorisstrakeljahn wants to merge 1 commit into
Conversation
Contributor
Author
|
The red check is not a test failure. GitHub could not find a macOS runner for “Dependencies (newest, extras, macos-latest)” and cancelled the job, so Coverage was skipped and CI Success failed. All other jobs passed, so only this job needs a re-run. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Without
X_valandrandom_state,optimize_hparamsgave every trial a different validation split (#631). This PR draws one seed at the start of the search and passes it to every fit, so all trials split the data the same way. If you pass X_val or set a random_state, the split works as before.While testing this I found a second problem. A fit with a seed resets NumPy's global random generator, and
gp_minimizedraws its random starting points from that generator when it has no seed of its own. The starting points then repeat. This already happens on main when you pass random_state to optimize_hparams: in my run the 12 trials tried only 4 different settings. So when the estimator has no random_state, the search now draws a seed for gp_minimize before the first fit. An estimator seed is used as before.The docs say that gp_minimize gets the estimator's random_state. That is still true when it is set. Without one, gp_minimize now gets a seed drawn from NumPy's global generator, where before it drew from that generator directly. So I left the docs as they are, but I can add a sentence if you prefer.
One thing I am not sure about: when no seed is set anywhere, every fit of the search now gets the same seed. All trials then also use the same seed for their initial weights, and the global random state is reseeded in every fit of the search. I think that makes the comparison fairer. If you prefer not to reseed, I can instead split off a validation set once at the start of the search.
The tests cover a regressor, a classifier and an LSS model. With the code from main, 5 of the new or changed test cases fail.
just test: 3379 passedjust docsandjust check: passFixes #631