Skip to content

Select configs among the configurations of the current tasks, and give the taskgraph their labels - #7598

Draft
marco-c wants to merge 9 commits into
mozilla:masterfrom
marco-c:config-selection-task-configs
Draft

marco-c wants to merge 9 commits into
mozilla:masterfrom
marco-c:config-selection-task-configs

Conversation

@marco-c

@marco-c marco-c commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

The config selection models are trained with the configurations of the push data, as mozci names them from the task definitions (renamed by rename_task, e.g. shippable configurations are the ones of their non-shippable equivalents). But the candidates were obtained by matching the labels of the tasks in a decision task's manifests-by-task.json with the configs seen in training, and the taskgraph matches the selected configs with the labels of its tasks. Those aren't the same:

  • the configs of tasks whose label doesn't have their Treeherder platform (e.g. Windows ASan: test-windows11-64-25h2-asan/opt-… vs test-windows11-64-25h2/asan-*) and of shippable tasks were unknown to the models, and the taskgraph couldn't run the Windows ASan ones when selected;
  • suites and variants named alike were confused (e.g. mochitest-webgpu and the webgpu variant), and the taskgraph also ran all the variants of a selected config (test-linux2404-64/opt-* matches the -swr, -standalone… tasks);
  • tasks which test selection doesn't optimize (e.g. standalone and repeat, which only run on backstop pushes) were candidates.

On a backstop push, about 8–12% of the target test tasks got the wrong config this way, and 36–64 candidate configs were unknown to the model. With this change, the configs of all the current tasks are known to the model, except those of brand-new variants.

What changes

  • The HTTP service retrieves the test tasks from the task graph of the latest backstop push (which runs all of them, with all their manifests), and computes their configurations with mozci's get_configuration, like the push data. Configs are selected among them.
  • config_groups in the replies now contains the exact labels (without chunk number) of the tasks of the selected configs, and config_group_configs contains the selected configs. The taskgraph matches labels with fnmatch, so exact labels work without any Gecko change. Selecting test-linux2404-64/opt-* no longer runs its variants, so the config selection strategies schedule fewer tasks.
  • Shippable tasks are never selected (they're too costly); their non-shippable equivalents are.
  • Because the labels must be the ones of the push's tasks, the tasks are updated every 10 minutes in the refresh process: from a new backstop push as soon as there is one, and with the tasks of the pushes after it that aren't known yet (e.g. renamed). Those are found among the pushes' target tasks (before optimization), with their definitions from the full task graph, which is only fetched (and only partly parsed) when there are unknown test tasks.
  • The configurations are computed with each push's own variants.yml, since mozci memoizes mozilla-central's, which lacks variants added on autoland until they're merged. Tasks whose configuration can't be computed are retried later rather than ignored.
  • Requires mozci 2.6.1 (Bump mozci to 2.6.1 #7599), whose get_configuration reads the test variants from test-setting. Adds pyyaml to the HTTP service's dependencies.

Checks

  • On live autoland data: boot retrieval takes about 40 s and peaks at about 700 MB. A 10-minute update with no new pushes makes no requests. A full-graph fetch (rare) takes about 1 minute and peaks at about 1 GB.
  • Offline evaluation (risk-weighted harness, production setting): training with the configs test selection can't control, while selecting only among the others, is better than training without them (−1.7 points without them, 95% CI −3.2 to −0.2, in each of three windows), so training still uses them. Keeping shippable configs separate instead of renamed made no measurable difference, so the rename is kept.
  • On backstop pushes (complete labels), the model catches 78.8 / 86.0 / 89.3% of failing pushes at 60 / 100 / 150 config/groups per push, against 71.4 / 80.2 / 84.8% for pooled statistics and 62.7 / 77.4 / 80.4% for a design with nothing learned.

Known limitation

A push that renames its own tasks (and pushes racing with it) can still lose the groups that only ran on the old labels, until its decision task finishes and the next update picks up the new labels. Fixing that needs the taskgraph to leave a group unrestricted when none of its labels match, which needs the global view discussed in bug 1927227.

The commits are meant to be reviewed one by one.

@marco-c
marco-c force-pushed the config-selection-task-configs branch 2 times, most recently from 7643f34 to cffcefd Compare November 19, 2026 23:03
… as the models name them

The config selection models are trained with the configurations of the push data, as mozci names
them from the task definitions (renamed, e.g. shippable configurations are the ones of their
non-shippable equivalents). The candidates were instead obtained by matching the labels of the
current tasks with the configs seen in training, which isn't the same:
- configs of tasks whose label doesn't have their Treeherder platform (e.g. the Windows ASan ones)
  and shippable tasks were unknown to the models;
- suites and variants named alike were confused (e.g. mochitest-webgpu and the webgpu variant);
- tasks which test selection doesn't optimize (e.g. the standalone ones, which only run on backstop
  pushes) were candidates.

The HTTP service now retrieves the test tasks that test selection optimizes from the task graph of
the latest backstop push (which runs all of them, with all their manifests), with their configs
computed with mozci's get_configuration, like the ones of the push data. Configs are selected among
them. Shippable tasks are left out, as they are too costly: the models are trained with the configs
of their non-shippable equivalents, which are selected instead.
The taskgraph matched the selected configs with the labels of its tasks, so it couldn't run the
configs of tasks whose label doesn't have their Treeherder platform (e.g. the Windows ASan ones),
and it also ran the variants of a selected config (e.g. test-linux2404-64/opt-* matching the swr
tasks). The replies now give it the labels (without chunk number) of the tasks of the selected
configs, and the selected configs themselves in "config_group_configs".
…ection models

They were only used to name the tasks of the current task graph, which are now named after their
configurations.
… parsing all of it

A decision task's full-task-graph.json is hundreds of MB (and much more once parsed), so only the
definitions of the given tasks are parsed, where they are in the text. If one isn't found this way
(e.g. if the format of the file changed), the whole graph is parsed.
… their configs

Tasks can be renamed after the task graph the config selection tasks come from. The tasks whose
manifests aren't known, with their configs, are given for all the groups selected on their configs,
as the taskgraph only runs a task for a group if the task runs it.
As the taskgraph matches the labels, they must be the ones of the push's tasks: the test tasks of
the pushes after the backstop push which weren't in its task graph (e.g. renamed since then) are
stored too. Those are found among the target tasks of the pushes (before optimization, which could
remove them as they match none of the labels in the replies), with their definitions from the full
task graph, which is only retrieved when there are unknown test tasks: the ones which aren't
eligible are remembered.
The tasks of the pushes after the backstop push (e.g. renamed since then) are only useful if they
are found soon after they appear, so the test tasks among which configs are selected are updated
every 10 minutes in the refresh process, rather than once per day: from the latest backstop push as
soon as there is a new one, and with the pushes after it which weren't processed yet. The pushes
known not to be backstop pushes aren't checked again, and the file is only written when it changes.
…t variants of their push

mozci memoizes the test variants of mozilla-central for the lifetime of the process (the refresh runs
for weeks), and those don't have the variants added on autoland until they are merged, so the
configurations of their tasks couldn't be computed. The HTTP service gives mozci the test variants
of each push it reads tasks from (from the push's taskcluster/test_configs/variants.yml), or makes it
retrieve the current ones of mozilla-central again if they can't be retrieved.
… which failed

Tasks whose configuration can't be computed (e.g. if retrieving the push's test variants failed)
were ignored, or skipped and left behind once their push was processed. They are now kept pending,
with the parts of their definitions needed to try again, and resolved by a later update: the ones
of the backstop push become candidates (with their manifests), the ones of the later pushes new
tasks. Other pushes targeting them don't need their full task graph meanwhile.
@marco-c
marco-c force-pushed the config-selection-task-configs branch from cffcefd to 3d6a2d2 Compare November 19, 2026 23:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant