Skip to content

Preserve missing metrics when merging repeated measurements - #5313

Open
DiadkoShmek wants to merge 1 commit into
meta:mainfrom
DiadkoShmek:lezo3-preserve-missing-metrics
Open

DiadkoShmek wants to merge 1 commit into
meta:mainfrom
DiadkoShmek:lezo3-preserve-missing-metrics

Conversation

@DiadkoShmek

Copy link
Copy Markdown

When an arm has no observations for a metric that is measured on another arm, MergeRepeatedMeasurements still constructs an empty inverse-variance aggregation. This replaces the missing mean with zero and the SEM with infinity. Converting the transformed experiment data to observations then includes a metric that was never measured for that arm.

Skip empty arm/metric groups so their original missing values remain missing. Regression coverage checks either metric being absent, the weighted mean and variance of the measured metric, preservation of other arms, and a metric first observed in a later repeat.

Validation:

  • Native Ax experiment/data extraction and observation conversion reproduce the failure on the base revision and pass with the change.
  • python -m pytest -q ax/adapter/transforms/tests/test_merge_repeated_measurements_transform.py ax/adapter/transforms/tests/test_search_space_to_choice_transform.py ax/adapter/tests/test_discrete_adapter.py — 14 passed, 2 subtests passed.
  • An independent exact-rational inverse-variance oracle passes 24 synthetic experiments, including reversed trial order (288 metric checks).
  • Project-pinned ufmt and flake8 checks pass for both changed files.
  • A separate public Experiment/Data check carried the result through TorchAdapter into actual BoTorch SupervisedDataset objects: the base fabricates three absent rows; the candidate retains the six measured rows, including a genuine zero.

AI assistance was used to identify, implement, and test this change.

@meta-cla meta-cla Bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Nov 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant