[Better Tests] Support custom host in compute_metadata by IvanBM18 · Pull Request #5479 · google/clusterfuzz

IvanBM18 · 2026-09-22T21:36:59Z

Bug: b/555370359

We can redirect google-auth metadata requests to a local emulator via the GCE_METADATA_HOST environment variable, but ClusterFuzz's own compute_metadata.py previously hardcoded metadata.google.internal and port 80. Reading GCE_METADATA_HOST and parsing optional ports allows local integration tests to run a metadata emulator on an ephemeral unprivileged port without root privileges or /etc/hosts changes.

Changes

Update _METADATA_SERVER in compute_metadata.py to read GCE_METADATA_HOST from the environment, falling back to metadata.google.internal.
Add unit tests in covering _metadata_host_port(), is_gce() on non-default ports, and GCE_METADATA_HOST overrides.

📚 Stacked PRs

Step	PR	Title	Base Branch
0	#5475	`[Better Tests] Adds new test dependencies`	`master`
1	#5479	`[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` (👉 This PR)	`better-test/dependencies`
2	#5485	`[Better Tests] Add GCE metadata emulator worker fixtures`	`better-tests/01-compute-metadata-host`
3	#5480	`[Better Tests] Add the WireMock GCE metadata emulator`	`better-tests/01a-metadata-emulator-configs`
4	#5486	`[Better Tests] Bootstrap and cover the GCE metadata emulator`	`better-tests/02-metadata-emulator-core`
5	#5481	`[Better Tests] Add fault injection and runtime overrides to metadata emulator`	`better-tests/02a-metadata-emulator-tests`

decoNR · 2026-09-28T19:35:25Z

Hi @IvanBM18 , it seems that tests are failing (probably unrelated), can you check it? (maybe re-run them or add a comment mentioning why it is unrelated if this is the case).

decoNR · 2026-09-28T19:57:18Z

-_METADATA_SERVER = 'metadata.google.internal'
+# GCE_METADATA_HOST may point to a metadata emulator instead of the
+# real server. It is the same variable google-auth honours.
+_METADATA_SERVER = os.getenv('GCE_METADATA_HOST', 'metadata.google.internal')


Just to confirm, are we sure that GCE_METADATA_HOST is not used by ClusterFuzz? In this case it could change the value of _METADATA_SERVER.

I didn't found any usages of the GCE_METADATA_HOST env var inside of CF:

Only the ones introduces by this changes.

And a job that defines GCE_METADATA_HOST will not override it, since this resolves at import time, where as job env vars resolve afterwards.

Nice, thanks. I found one usage in the configuration repo (which I will not link here as it is private) that may be worth checking. Alternatively, a safer approach could be to add a log and monitor it after the deployment, with the option to delete it in the future.

Sure I'll add monitoring for this.
Regarding the CF-Config definition:

We shouldn't worry about that, its only usage is for the swarming bots, and that's were i got the inspiration for this :D
To not dive deep into details, Swarming bots also use a custom metadata server to provide auth credentials.

IvanBM18 · 2026-09-28T22:36:55Z

Hi @IvanBM18 , it seems that tests are failing (probably unrelated), can you check it? (maybe re-run them or add a comment mentioning why it is unrelated if this is the case).

By reading the logs, seems that this check is failing at this step:

clusterfuzz/docker/chromium/base/ubuntu-24-04.Dockerfile

Line 15 in e8ccb0b

    
           RUN apt-get update && echo ttf-mscorefonts-installer msttcorefonts/accepted-mscorefonts-eula select true | debconf-set-selections &&     curl 'https://chromium.googlesource.com/chromium/src/+/main/build/install-build-deps.py?format=TEXT' | base64 -d > /tmp/install-build-deps.py &&     sed -i s/snapcraft/doesnotexist/ /tmp/install-build-deps.py &&     sed -i "s/if requires_pinned_linux_libc():/if False:/" /tmp/install-build-deps.py &&     chmod u+x /tmp/install-build-deps.py &&     /tmp/install-build-deps.py --backwards-compatible --no-prompt --no-chromeos-fonts --syms --lib32

Which is not related to this changes :D

Is a curl to the chromium repo. I think that maybe the curl fails because as of recently, chromium no longer accepts unauthenticated traffic at its codesearch, still not 100% sure if this is the roout cause but might be a lead.

decoNR

LGTM but left a comment that needs to be resolved before merging.

dylanjew · 2026-09-29T19:30:42Z

+                       compute_metadata._metadata_host_port())  # pylint: disable=protected-access
+
+    with mock.patch.object(compute_metadata, '_METADATA_SERVER', '[::1]:41234'):
+      self.assertEqual(('::1', 41234), compute_metadata._metadata_host_port())  # pylint: disable=protected-access


nit: this would be slightly clearer if you used variables lik host, port = compute_metadata._metadata_host_port() and asserted against those

Bug: b/555370359 Split out of #5480, which had grown to 821 lines across the fixtures, the emulator that serves them, and the tests that exercise it. This PR is the data half: the JSON configs the GCE metadata emulator seeds itself from. Each config describes one worker identity, so a test can pick whether it runs trusted or untrusted by picking which fixture answers on the default address. The shape of `metadata` mirrors the real metadata server's hierarchy, so a config can be retrieved by reading a live instance: ``` curl -H 'Metadata-Flavor: Google' \ 'http://metadata.google.internal/computeMetadata/v1/?recursive=true' | jq ``` ## Changes - Add `local/emulators/configs/tworker.json`: the trusted worker identity, with its service account email, scopes, a fake access token, and the project and instance metadata a tworker reads. - Add `local/emulators/configs/uworker.json`: the untrusted counterpart, so both can run side by side and be shown to stay distinct. - Add `local/emulators/configs/template.json`: an annotated reference for writing a new config. It is documentation rather than a loadable fixture, so it keeps its `//` comments; nothing parses it. `tworker.json` and `uworker.json` are plain JSON and are the only two loaded. Nothing reads these yet. The emulator that serves them is #5480, next in the stack. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | **2** | **#5485** | **`[Better Tests] Add GCE metadata emulator worker fixtures`** *(👉 This PR)* | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Creates a GCE metadata emulator for our future integration test suite, built on Testcontainers and WireMock. It lets tests resolve credentials and read instance and project metadata without a real GCE host or a local `gcloud` login, and lets them distinguish a trusted from an untrusted environment. `google-auth` and our vendored `oauth2client` evaluate the `GCE_METADATA_*` environment variables once, at import time, and bake the resulting URL in. Importing this module therefore claims two loopback addresses up front; the addresses never move, only what listens on them does. A test picks its in-process identity by picking which fixture answers on the default address: ```python with gce_metadata_emulator.trusted_as_default() as tworker: ... with gce_metadata_emulator.untrusted_as_default() as uworker: ... with gce_metadata_emulator.trusted_untrusted_pair() as (tworker, uworker): ... # uworker sits on the secondary address ``` Nothing is patched to make that work. An emulator on the secondary address is reachable through the client its context manager yields, or by handing its env to a child process: ```python subprocess.run(argv, env={**os.environ, **uworker.env}) ``` This emulator was based off the pre-existing [local emulator go wrap](https://github.com/google/clusterfuzz/blob/master/local/emulators/metadata.go) + what is detailed at the [GCP metadata server docs](https://docs.cloud.google.com/compute/docs/metadata/overview) ## Changes - Add `gce_metadata_emulator.py`, a WireMock Testcontainer fake GCE metadata server. Supports: - Serving fake service account credentials, so a test can tell a trusted from an untrusted environment. - Serving project- and instance-scoped metadata values, including attribute shadowing, seeded from the JSON configs added in #5485. - Rejecting requests that omit `Metadata-Flavor: Google`, the way the real server does. - Failing loudly when two emulators want the same address. The tests covering all of this land in #5486, next in the stack. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | **3** | **#5480** | **`[Better Tests] Add the WireMock GCE metadata emulator`** *(👉 This PR)* | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Split out of #5480. This PR wires the GCE metadata emulator added in #5480 into the test runner, and covers it. ## Note - This is the first of many integration test modules, for this purpose, it uses `pytest` library instead of `unittest`, this requires different syntax for asserting, but files & methods naming is basically the same. - This is a experimental integration test module, still a standalone, we require to wire a couple of things with a butler command to allow easy execution of this kind of tests like we do with unit test, this will be done in a subsequent bug: b/540876792 - Documentation will be created in b/555424439 once we have our first set of integration test suites & the butler command is ready to run them - I aim to create documentation & guidelines to be created based off this first tests and overall the discussion we have in the PRs for them, so feel free to push back on even the smallest details, nothing is set in stone! ## Changes - Adds `gce_metadata_emulator_test.py`, skipped right now because its experimental, just in case this ends in the CI process before we intended to. ## Tests - `GceMetadataEmulatorTest` an integration test for how some common workflows behave using the new emulator, runs a trusted and an untrusted emulator side by side and verifies: - `GceMetadataEmulatorUntrustedDefaultTest` puts a metadata server that returns uworker on the default address and verifies that all resolve to the untrusted identity. To run it: ``` PYTHONPATH=src:src/third_party pytest test_suites/fixtures/gce_metadata_server_test.py ``` We need to later include them in `butler.py` so that we can: - Run them smoothly - Don't require to do the imports manually But theres already a bug for that, and the purpose of this is not creating the butler setup but to create an emulator for other tests. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | **4** | **#5486** | **`[Better Tests] Bootstrap and cover the GCE metadata emulator`** *(👉 This PR)* | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

…data fake values (#5481) Bug: b/555370359 Following the parent PR, this one introduces changes for our emulator so that we can force it to fail either constantly, or on demand, this opens a ton of possibilities for integration tests, this is technically also a new feature, but to simplify reviews i moved this one to a separate PR. With this changes it now simulates transient GCE metadata server failures (such as HTTP 500s or latency spikes) and mutate instance/project attributes dynamically during a test without restarting the WireMock container. Using WireMock stateful scenarios allows injecting faults that automatically expire after N requests so retry decorators (such as `@retry.wrap` on `compute_metadata.get()`) can be tested deterministically. ## Changes - Adds an interface(for composition instead of inheritance) for so that we can inject fault mapping to any API that uses wiremock as its base starting with the metadata_emulator. - Track runtime instance attribute overrides via `_runtime_instance_attributes` so `set_project_attribute()` does not overwrite attributes shadowed at runtime, and reset them in `clear_faults()`. - Adds tests cases for this. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | **5** | **#5481** | **`[Better Tests] Add fault injection and runtime overrides to metadata emulator`** *(👉 This PR)* | `better-tests/02a-metadata-emulator-tests` |

…ta emulators LUCI's local auth server (exported via GCE_METADATA_HOST by luci-auth context on Swarming) only serves tokens and a few project/instance values. Since #5479, is_gce() honours GCE_METADATA_HOST and only checks for a TCP connection, so Swarming bots were treated as GCE and crashed fetching instance/zone (404). Probe instance/id instead, which real GCE and the test emulator serve but LUCI's emulator does not. Bug: b/571127789

On Swarming, luci-auth context exports GCE_METADATA_HOST pointing to its local token-only metadata emulator. Since #5479, is_gce() honours that variable and only checks for a TCP connection, so Swarming bots were treated as GCE and crashed fetching instance/zone (404). Bug: b/571127789

Bug: b/571127789 ## Background Since #5479, `is_gce()` uses `GCE_METADATA_HOST` and treats any TCP connection to it as "on GCE". Swarming's uses a cli tool: `luci-auth context` which sets `GCE_METADATA_HOST` to a server that mints tokens so that swarming bots can connect to gcp services, but seems that luci-auth emulator is so good that it fools CF to believe that is running on a GCE bot, hence causing failures in swarming when: ``` requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://127.0.0.1:36755/computeMetadata/v1/instance/zone ``` This change only affects Swarming bots. A broader option (probe `instance/id` instead of a TCP connect) is on a separate branch, because it changes behaviour on every bot. ## Changes - `compute_metadata.is_gce()` now returns `False` right away when `environment.is_running_on_swarming()` is true. ## Test - Add `test_is_gce_on_swarming`: with `SWARMING_BOT=True`, `is_gce()` returns `False` even when the metadata host is reachable.

IvanBM18 changed the title ~~[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata~~ [Better Tests] Support custom host in compute_metadata Sep 24, 2026

IvanBM18 marked this pull request as ready for review September 25, 2026 01:23

IvanBM18 requested a review from a team as a code owner September 25, 2026 01:23

IvanBM18 added this pull request to stack #5497 September 25, 2026 01:44

IvanBM18 requested a review from dylanjew September 25, 2026 02:12

Base automatically changed from better-test/dependencies to master September 28, 2026 16:54

IvanBM18 force-pushed the better-tests/01-compute-metadata-host branch from a5e5ca2 to 4414308 Compare September 28, 2026 16:54

decoNR reviewed Sep 28, 2026

View reviewed changes

decoNR approved these changes Sep 29, 2026

View reviewed changes

dylanjew approved these changes Sep 29, 2026

View reviewed changes

IvanBM18 force-pushed the better-tests/01-compute-metadata-host branch 2 times, most recently from f971c0a to 6fd6e3b Compare October 2, 2026 17:12

IvanBM18 added 5 commits October 6, 2026 16:19

[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata

6246c14

[Better Tests] Refine comment

a4dadd2

[Better tests] Better parsing in compute_metadata

e2b677d

[Better tests] Adds logs for monitoring metadata server

b3a108b

[Better tests] Make assertions cleaner by asserting per host & port"

278548e

IvanBM18 force-pushed the better-tests/01-compute-metadata-host branch from 6fd6e3b to 278548e Compare October 6, 2026 22:19

IvanBM18 merged commit cde1b89 into master Oct 6, 2026
15 checks passed

IvanBM18 deleted the better-tests/01-compute-metadata-host branch October 6, 2026 22:23

IvanBM18 mentioned this pull request Oct 7, 2026

[Swarming] Skip GCE detection on Swarming bots #5535

Merged

Navigation Menu

Repository navigation

[Better Tests] Support custom host in compute_metadata - #5479

[Better Tests] Support custom host in compute_metadata#5479
IvanBM18 merged 5 commits into
masterfrom
better-tests/01-compute-metadata-host

IvanBM18 commented Sep 22, 2026 •

edited

Loading

Uh oh!

decoNR commented Sep 28, 2026

Uh oh!

decoNR Sep 28, 2026

Uh oh!

IvanBM18 Sep 28, 2026 •

edited

Loading

Uh oh!

decoNR Sep 29, 2026

Uh oh!

IvanBM18 Sep 29, 2026

Uh oh!

IvanBM18 commented Sep 28, 2026

Uh oh!

decoNR left a comment

Uh oh!

dylanjew Sep 29, 2026

Uh oh!

IvanBM18 Sep 29, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

IvanBM18 commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

📚 Stacked PRs

Uh oh!

decoNR commented Sep 28, 2026

Uh oh!

decoNR Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

decoNR Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 commented Sep 28, 2026

Uh oh!

decoNR left a comment

Choose a reason for hiding this comment

Uh oh!

dylanjew Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

IvanBM18 commented Sep 22, 2026 •

edited

Loading

IvanBM18 Sep 28, 2026 •

edited

Loading