Skip to content

fix(hpo): score all trials of a search on one validation split - #632

Open
jorisstrakeljahn wants to merge 1 commit into
OpenTabular:mainfrom
jorisstrakeljahn:fix/hpo-one-validation-split
Open

jorisstrakeljahn wants to merge 1 commit into
OpenTabular:mainfrom
jorisstrakeljahn:fix/hpo-one-validation-split

Conversation

@jorisstrakeljahn

Copy link
Copy Markdown
Contributor

Without X_val and random_state, optimize_hparams gave every trial a different validation split (#631). This PR draws one seed at the start of the search and passes it to every fit, so all trials split the data the same way. If you pass X_val or set a random_state, the split works as before.

While testing this I found a second problem. A fit with a seed resets NumPy's global random generator, and gp_minimize draws its random starting points from that generator when it has no seed of its own. The starting points then repeat. This already happens on main when you pass random_state to optimize_hparams: in my run the 12 trials tried only 4 different settings. So when the estimator has no random_state, the search now draws a seed for gp_minimize before the first fit. An estimator seed is used as before.

The docs say that gp_minimize gets the estimator's random_state. That is still true when it is set. Without one, gp_minimize now gets a seed drawn from NumPy's global generator, where before it drew from that generator directly. So I left the docs as they are, but I can add a sentence if you prefer.

One thing I am not sure about: when no seed is set anywhere, every fit of the search now gets the same seed. All trials then also use the same seed for their initial weights, and the global random state is reseeded in every fit of the search. I think that makes the comparison fairer. If you prefer not to reseed, I can instead split off a validation set once at the start of the search.

The tests cover a regressor, a classifier and an LSS model. With the code from main, 5 of the new or changed test cases fail.

just test: 3379 passed
just docs and just check: pass

Fixes #631

@jorisstrakeljahn

Copy link
Copy Markdown
Contributor Author

The red check is not a test failure. GitHub could not find a macOS runner for “Dependencies (newest, extras, macos-latest)” and cancelled the job, so Coverage was skipped and CI Success failed. All other jobs passed, so only this job needs a re-run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] optimize_hparams without X_val scores every trial on a different validation split

1 participant