[Better Tests] Add GCE metadata emulator worker/uworker configs by IvanBM18 · Pull Request #5485 · google/clusterfuzz

IvanBM18 · 2026-09-23T18:13:20Z

Bug: b/555370359

Split out of #5480, which had grown to 821 lines across the fixtures, the emulator that serves them, and the tests that exercise it.

This PR is the data half: the JSON configs the GCE metadata emulator seeds itself from. Each config describes one worker identity, so a test can pick whether it runs trusted or untrusted by picking which fixture answers on the default address.

The shape of metadata mirrors the real metadata server's hierarchy, so a config can be retrieved by reading a live instance:

curl -H 'Metadata-Flavor: Google' \
  'http://metadata.google.internal/computeMetadata/v1/?recursive=true' | jq

Changes

Add local/emulators/configs/tworker.json: the trusted worker identity, with its service account email, scopes, a fake access token, and the project and instance metadata a tworker reads.
Add local/emulators/configs/uworker.json: the untrusted counterpart, so both can run side by side and be shown to stay distinct.
Add local/emulators/configs/template.json: an annotated reference for writing a new config. It is documentation rather than a loadable fixture, so it keeps its // comments; nothing parses it. tworker.json and uworker.json are plain JSON and are the only two loaded.

Nothing reads these yet. The emulator that serves them is #5480, next in the stack.

📚 Stacked PRs

Step	PR	Title	Base Branch
0	#5475	`[Better Tests] Adds new test dependencies`	`master`
1	#5479	`[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata`	`better-test/dependencies`
2	#5485	`[Better Tests] Add GCE metadata emulator worker fixtures` (👉 This PR)	`better-tests/01-compute-metadata-host`
3	#5480	`[Better Tests] Add the WireMock GCE metadata emulator`	`better-tests/01a-metadata-emulator-configs`
4	#5486	`[Better Tests] Bootstrap and cover the GCE metadata emulator`	`better-tests/02-metadata-emulator-core`
5	#5481	`[Better Tests] Add fault injection and runtime overrides to metadata emulator`	`better-tests/02a-metadata-emulator-tests`

dylanjew

LGTM

decoNR · 2026-09-30T19:08:11Z

+      "preempted": "FALSE",
+      "scheduling": {
+        "preemptible": "FALSE",
+        "provisioning-model": "SPOT"


Is preemptible = False and provisioning-model = SPOT correct?

I would try to match what we have on prod. Is that different than SPOT?

The metadata server does return FALSE & TRUE, don't know why but thats what it does.

MB "SPOT" is used with preemptible VMs, so if we want to match what a non pre-emptible bot has this should be STANDARD, Changing it!

Adds the JSON fixtures the WireMock metadata emulator serves: a tworker and a uworker identity, plus an annotated template describing the shape of a fixture. The shape mirrors the real metadata server's hierarchy, so a fixture can be written by reading a live instance: curl -H 'Metadata-Flavor: Google' \ 'http://metadata.google.internal/computeMetadata/v1/?recursive=true' template.json is documentation rather than a loadable fixture: it keeps its // comments, and nothing parses it.

Bug: b/555370359 We can redirect `google-auth` metadata requests to a local emulator via the `GCE_METADATA_HOST` environment variable, but ClusterFuzz's own `compute_metadata.py` previously hardcoded `metadata.google.internal` and port `80`. Reading `GCE_METADATA_HOST` and parsing optional ports allows local integration tests to run a metadata emulator on an ephemeral unprivileged port without root privileges or `/etc/hosts` changes. ## Changes - Update `_METADATA_SERVER` in `compute_metadata.py` to read `GCE_METADATA_HOST` from the environment, falling back to `metadata.google.internal`. - Add unit tests in covering `_metadata_host_port()`, `is_gce()` on non-default ports, and `GCE_METADATA_HOST` overrides. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | **1** | **#5479** | **`[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata`** *(👉 This PR)* | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Creates a GCE metadata emulator for our future integration test suite, built on Testcontainers and WireMock. It lets tests resolve credentials and read instance and project metadata without a real GCE host or a local `gcloud` login, and lets them distinguish a trusted from an untrusted environment. `google-auth` and our vendored `oauth2client` evaluate the `GCE_METADATA_*` environment variables once, at import time, and bake the resulting URL in. Importing this module therefore claims two loopback addresses up front; the addresses never move, only what listens on them does. A test picks its in-process identity by picking which fixture answers on the default address: ```python with gce_metadata_emulator.trusted_as_default() as tworker: ... with gce_metadata_emulator.untrusted_as_default() as uworker: ... with gce_metadata_emulator.trusted_untrusted_pair() as (tworker, uworker): ... # uworker sits on the secondary address ``` Nothing is patched to make that work. An emulator on the secondary address is reachable through the client its context manager yields, or by handing its env to a child process: ```python subprocess.run(argv, env={**os.environ, **uworker.env}) ``` This emulator was based off the pre-existing [local emulator go wrap](https://github.com/google/clusterfuzz/blob/master/local/emulators/metadata.go) + what is detailed at the [GCP metadata server docs](https://docs.cloud.google.com/compute/docs/metadata/overview) ## Changes - Add `gce_metadata_emulator.py`, a WireMock Testcontainer fake GCE metadata server. Supports: - Serving fake service account credentials, so a test can tell a trusted from an untrusted environment. - Serving project- and instance-scoped metadata values, including attribute shadowing, seeded from the JSON configs added in #5485. - Rejecting requests that omit `Metadata-Flavor: Google`, the way the real server does. - Failing loudly when two emulators want the same address. The tests covering all of this land in #5486, next in the stack. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | **3** | **#5480** | **`[Better Tests] Add the WireMock GCE metadata emulator`** *(👉 This PR)* | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Split out of #5480. This PR wires the GCE metadata emulator added in #5480 into the test runner, and covers it. ## Note - This is the first of many integration test modules, for this purpose, it uses `pytest` library instead of `unittest`, this requires different syntax for asserting, but files & methods naming is basically the same. - This is a experimental integration test module, still a standalone, we require to wire a couple of things with a butler command to allow easy execution of this kind of tests like we do with unit test, this will be done in a subsequent bug: b/540876792 - Documentation will be created in b/555424439 once we have our first set of integration test suites & the butler command is ready to run them - I aim to create documentation & guidelines to be created based off this first tests and overall the discussion we have in the PRs for them, so feel free to push back on even the smallest details, nothing is set in stone! ## Changes - Adds `gce_metadata_emulator_test.py`, skipped right now because its experimental, just in case this ends in the CI process before we intended to. ## Tests - `GceMetadataEmulatorTest` an integration test for how some common workflows behave using the new emulator, runs a trusted and an untrusted emulator side by side and verifies: - `GceMetadataEmulatorUntrustedDefaultTest` puts a metadata server that returns uworker on the default address and verifies that all resolve to the untrusted identity. To run it: ``` PYTHONPATH=src:src/third_party pytest test_suites/fixtures/gce_metadata_server_test.py ``` We need to later include them in `butler.py` so that we can: - Run them smoothly - Don't require to do the imports manually But theres already a bug for that, and the purpose of this is not creating the butler setup but to create an emulator for other tests. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | **4** | **#5486** | **`[Better Tests] Bootstrap and cover the GCE metadata emulator`** *(👉 This PR)* | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

…data fake values (#5481) Bug: b/555370359 Following the parent PR, this one introduces changes for our emulator so that we can force it to fail either constantly, or on demand, this opens a ton of possibilities for integration tests, this is technically also a new feature, but to simplify reviews i moved this one to a separate PR. With this changes it now simulates transient GCE metadata server failures (such as HTTP 500s or latency spikes) and mutate instance/project attributes dynamically during a test without restarting the WireMock container. Using WireMock stateful scenarios allows injecting faults that automatically expire after N requests so retry decorators (such as `@retry.wrap` on `compute_metadata.get()`) can be tested deterministically. ## Changes - Adds an interface(for composition instead of inheritance) for so that we can inject fault mapping to any API that uses wiremock as its base starting with the metadata_emulator. - Track runtime instance attribute overrides via `_runtime_instance_attributes` so `set_project_attribute()` does not overwrite attributes shadowed at runtime, and reset them in `clear_faults()`. - Adds tests cases for this. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | **5** | **#5481** | **`[Better Tests] Add fault injection and runtime overrides to metadata emulator`** *(👉 This PR)* | `better-tests/02a-metadata-emulator-tests` |

IvanBM18 changed the title ~~[Better Tests] Add GCE metadata emulator worker fixtures~~ [Better Tests] Add GCE metadata emulator worker/uworker configs Sep 24, 2026

IvanBM18 added this pull request to stack #5497 September 25, 2026 01:44

IvanBM18 marked this pull request as ready for review September 25, 2026 01:45

IvanBM18 requested a review from a team as a code owner September 25, 2026 01:45

IvanBM18 requested a review from dylanjew September 25, 2026 02:12

IvanBM18 force-pushed the better-tests/01a-metadata-emulator-configs branch from 321e5bc to d26fa22 Compare September 28, 2026 16:54

dylanjew reviewed Sep 29, 2026

View reviewed changes

Comment thread local/emulators/configs/uworker.json Outdated

dylanjew reviewed Sep 29, 2026

View reviewed changes

IvanBM18 force-pushed the better-tests/01a-metadata-emulator-configs branch from ce97852 to 762e923 Compare September 30, 2026 18:00

IvanBM18 added ready for GOSST review Pull requests ready for GOSST team review Tests PRs focused merely on tests suite changes labels Sep 30, 2026

IvanBM18 force-pushed the better-tests/01a-metadata-emulator-configs branch from 8cd19f0 to 1c35e7d Compare September 30, 2026 18:27

decoNR approved these changes Sep 30, 2026

View reviewed changes

decoNR reviewed Sep 30, 2026

View reviewed changes

Comment thread test_suites/seed/gce_metadata_server/uworker.json

dylanjew approved these changes Sep 30, 2026

View reviewed changes

IvanBM18 force-pushed the better-tests/01a-metadata-emulator-configs branch from 32fca30 to dddad9c Compare October 2, 2026 17:12

IvanBM18 added 4 commits October 6, 2026 16:19

[Better tests] Defaults uworker to non pre-emtible bot

64c1ec5

[Better tests] Moves location of configs

580fe26

[Better tests] Scheduling is now STANDARD for uworkers

2c93536

IvanBM18 force-pushed the better-tests/01a-metadata-emulator-configs branch from dddad9c to 2c93536 Compare October 6, 2026 22:19

IvanBM18 merged commit b264aa7 into master Oct 6, 2026
14 checks passed

IvanBM18 deleted the better-tests/01a-metadata-emulator-configs branch October 6, 2026 22:23

Navigation Menu

Repository navigation

[Better Tests] Add GCE metadata emulator worker/uworker configs - #5485

[Better Tests] Add GCE metadata emulator worker/uworker configs#5485
IvanBM18 merged 4 commits into
better-tests/01-compute-metadata-hostfrom
better-tests/01a-metadata-emulator-configs

IvanBM18 commented Sep 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

dylanjew left a comment

Uh oh!

decoNR Sep 30, 2026 •

edited

Loading

Uh oh!

dylanjew Sep 30, 2026

Uh oh!

IvanBM18 Sep 30, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

IvanBM18 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

📚 Stacked PRs

Uh oh!

Uh oh!

dylanjew left a comment

Choose a reason for hiding this comment

Uh oh!

decoNR Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dylanjew Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

IvanBM18 commented Sep 23, 2026 •

edited

Loading

decoNR Sep 30, 2026 •

edited

Loading