[Better Tests] Integration tests for the GCE metadata fake by IvanBM18 · Pull Request #5486 · google/clusterfuzz

IvanBM18 · 2026-09-23T18:13:22Z

Bug: b/555370359

Split out of #5480. This PR wires the GCE metadata emulator added in #5480 into the test runner, and covers it.

Note

This is the first of many integration test modules, for this purpose, it uses pytest library instead of unittest, this requires different syntax for asserting, but files & methods naming is basically the same.
This is a experimental integration test module, still a standalone, we require to wire a couple of things with a butler command to allow easy execution of this kind of tests like we do with unit test, this will be done in a subsequent bug: b/540876792
Documentation will be created in b/555424439 once we have our first set of integration test suites & the butler command is ready to run them
- I aim to create documentation & guidelines to be created based off this first tests and overall the discussion we have in the PRs for them, so feel free to push back on even the smallest details, nothing is set in stone!

Changes

Adds gce_metadata_emulator_test.py, skipped right now because its experimental, just in case this ends in the CI process before we intended to.

Tests

GceMetadataEmulatorTest an integration test for how some common workflows behave using the new emulator, runs a trusted and an untrusted emulator side by side and verifies:
GceMetadataEmulatorUntrustedDefaultTest puts a metadata server that returns uworker on the default address and verifies that all resolve to the untrusted identity.

To run it:

PYTHONPATH=src:src/third_party pytest test_suites/fixtures/gce_metadata_server_test.py

We need to later include them in butler.py so that we can:

Run them smoothly
Don't require to do the imports manually

But theres already a bug for that, and the purpose of this is not creating the butler setup but to create an emulator for other tests.

📚 Stacked PRs

Step	PR	Title	Base Branch
0	#5475	`[Better Tests] Adds new test dependencies`	`master`
1	#5479	`[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata`	`better-test/dependencies`
2	#5485	`[Better Tests] Add GCE metadata emulator worker fixtures`	`better-tests/01-compute-metadata-host`
3	#5480	`[Better Tests] Add the WireMock GCE metadata emulator`	`better-tests/01a-metadata-emulator-configs`
4	#5486	`[Better Tests] Bootstrap and cover the GCE metadata emulator` (👉 This PR)	`better-tests/02-metadata-emulator-core`
5	#5481	`[Better Tests] Add fault injection and runtime overrides to metadata emulator`	`better-tests/02a-metadata-emulator-tests`

IvanBM18 · 2026-09-29T01:49:04Z

Im very open to change this file's directory i put it up here because we still haven't defined guidelines or specific dir naming, also this would be a special case its a integration test, but not for a CF feature but for a CF fake/emulator/double so will have to think how we handle this.

This should be close to the JSON data files as well, since it asserts against those.

Done, i moved them based on what i posted on this doc: https://docs.google.com/document/d/1OcvLED_sEJS0Y8RcluL39qqW4v8-liFxiG9eyV20s8c/edit?resourcekey=0-Qca3yUhT4VsnYnUe74T_Iw&tab=t.3y8k4whcf5yf#heading=h.hav9zlk8nbmw

dylanjew · 2026-09-29T21:09:48Z

+class TestGceMetadataEmulator:
+  """Integration tests for the fake GCE metadata server."""
+
+  # TODO(b/555371391) Remove this setup once we have our own butler.py command


Agreed we don't want each test to have to remember to set this up

dylanjew · 2026-09-29T21:11:36Z

This should be close to the JSON data files as well, since it asserts against those.

dylanjew · 2026-09-29T21:13:29Z

+    creds, project_id = google.auth.default()
+
+    assert isinstance(creds, compute_engine.Credentials)
+    assert project_id == 'test-clusterfuzz'


it's a bit confusing where these files come from. Would it be clearer to import the JSON data as as dict and assert against that dict's keys? otherwise I would add a reference to that file

Mmm, what about both!

If needed we can delete this later

nice, these are very clear now!

dylanjew

LGTM

Publishes the emulator's default address from py_unittest before any test imports the Google auth libraries, which resolve that address once and keep it. Without this, an emulator started later in a test is unreachable from already-imported code. Adds integration tests covering what the emulator is for: google.auth and clusterfuzz's own credentials wrapper resolving to the trusted identity, a refresh yielding the configured token, and a trusted and an untrusted emulator serving distinct identities side by side.

…ddress is taken

…ength

Bug: b/555370359 We can redirect `google-auth` metadata requests to a local emulator via the `GCE_METADATA_HOST` environment variable, but ClusterFuzz's own `compute_metadata.py` previously hardcoded `metadata.google.internal` and port `80`. Reading `GCE_METADATA_HOST` and parsing optional ports allows local integration tests to run a metadata emulator on an ephemeral unprivileged port without root privileges or `/etc/hosts` changes. ## Changes - Update `_METADATA_SERVER` in `compute_metadata.py` to read `GCE_METADATA_HOST` from the environment, falling back to `metadata.google.internal`. - Add unit tests in covering `_metadata_host_port()`, `is_gce()` on non-default ports, and `GCE_METADATA_HOST` overrides. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | **1** | **#5479** | **`[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata`** *(👉 This PR)* | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Split out of #5480, which had grown to 821 lines across the fixtures, the emulator that serves them, and the tests that exercise it. This PR is the data half: the JSON configs the GCE metadata emulator seeds itself from. Each config describes one worker identity, so a test can pick whether it runs trusted or untrusted by picking which fixture answers on the default address. The shape of `metadata` mirrors the real metadata server's hierarchy, so a config can be retrieved by reading a live instance: ``` curl -H 'Metadata-Flavor: Google' \ 'http://metadata.google.internal/computeMetadata/v1/?recursive=true' | jq ``` ## Changes - Add `local/emulators/configs/tworker.json`: the trusted worker identity, with its service account email, scopes, a fake access token, and the project and instance metadata a tworker reads. - Add `local/emulators/configs/uworker.json`: the untrusted counterpart, so both can run side by side and be shown to stay distinct. - Add `local/emulators/configs/template.json`: an annotated reference for writing a new config. It is documentation rather than a loadable fixture, so it keeps its `//` comments; nothing parses it. `tworker.json` and `uworker.json` are plain JSON and are the only two loaded. Nothing reads these yet. The emulator that serves them is #5480, next in the stack. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | **2** | **#5485** | **`[Better Tests] Add GCE metadata emulator worker fixtures`** *(👉 This PR)* | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Creates a GCE metadata emulator for our future integration test suite, built on Testcontainers and WireMock. It lets tests resolve credentials and read instance and project metadata without a real GCE host or a local `gcloud` login, and lets them distinguish a trusted from an untrusted environment. `google-auth` and our vendored `oauth2client` evaluate the `GCE_METADATA_*` environment variables once, at import time, and bake the resulting URL in. Importing this module therefore claims two loopback addresses up front; the addresses never move, only what listens on them does. A test picks its in-process identity by picking which fixture answers on the default address: ```python with gce_metadata_emulator.trusted_as_default() as tworker: ... with gce_metadata_emulator.untrusted_as_default() as uworker: ... with gce_metadata_emulator.trusted_untrusted_pair() as (tworker, uworker): ... # uworker sits on the secondary address ``` Nothing is patched to make that work. An emulator on the secondary address is reachable through the client its context manager yields, or by handing its env to a child process: ```python subprocess.run(argv, env={**os.environ, **uworker.env}) ``` This emulator was based off the pre-existing [local emulator go wrap](https://github.com/google/clusterfuzz/blob/master/local/emulators/metadata.go) + what is detailed at the [GCP metadata server docs](https://docs.cloud.google.com/compute/docs/metadata/overview) ## Changes - Add `gce_metadata_emulator.py`, a WireMock Testcontainer fake GCE metadata server. Supports: - Serving fake service account credentials, so a test can tell a trusted from an untrusted environment. - Serving project- and instance-scoped metadata values, including attribute shadowing, seeded from the JSON configs added in #5485. - Rejecting requests that omit `Metadata-Flavor: Google`, the way the real server does. - Failing loudly when two emulators want the same address. The tests covering all of this land in #5486, next in the stack. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | **3** | **#5480** | **`[Better Tests] Add the WireMock GCE metadata emulator`** *(👉 This PR)* | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

…data fake values (#5481) Bug: b/555370359 Following the parent PR, this one introduces changes for our emulator so that we can force it to fail either constantly, or on demand, this opens a ton of possibilities for integration tests, this is technically also a new feature, but to simplify reviews i moved this one to a separate PR. With this changes it now simulates transient GCE metadata server failures (such as HTTP 500s or latency spikes) and mutate instance/project attributes dynamically during a test without restarting the WireMock container. Using WireMock stateful scenarios allows injecting faults that automatically expire after N requests so retry decorators (such as `@retry.wrap` on `compute_metadata.get()`) can be tested deterministically. ## Changes - Adds an interface(for composition instead of inheritance) for so that we can inject fault mapping to any API that uses wiremock as its base starting with the metadata_emulator. - Track runtime instance attribute overrides via `_runtime_instance_attributes` so `set_project_attribute()` does not overwrite attributes shadowed at runtime, and reset them in `clear_faults()`. - Adds tests cases for this. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | **5** | **#5481** | **`[Better Tests] Add fault injection and runtime overrides to metadata emulator`** *(👉 This PR)* | `better-tests/02a-metadata-emulator-tests` |

IvanBM18 changed the title ~~[Better Tests] Bootstrap and cover the GCE metadata emulator~~ [Better Tests] Integration tests for the GCE metadata emulator Sep 24, 2026

IvanBM18 force-pushed the better-tests/02a-metadata-emulator-tests branch 3 times, most recently from 6697734 to cff0823 Compare September 24, 2026 06:19

IvanBM18 mentioned this pull request Sep 24, 2026

[Better Tests] Adds new test dependencies #5475

Merged

IvanBM18 added this pull request to stack #5497 September 25, 2026 01:44

IvanBM18 marked this pull request as ready for review September 25, 2026 02:10

IvanBM18 requested a review from a team as a code owner September 25, 2026 02:10

IvanBM18 requested a review from dylanjew September 25, 2026 02:12

IvanBM18 force-pushed the better-tests/02a-metadata-emulator-tests branch from abccfe0 to a0b53b6 Compare September 28, 2026 16:54

IvanBM18 commented Sep 29, 2026

View reviewed changes

dylanjew reviewed Sep 29, 2026

View reviewed changes

IvanBM18 force-pushed the better-tests/02a-metadata-emulator-tests branch 4 times, most recently from 4635b18 to 8d0905b Compare September 30, 2026 19:23

IvanBM18 changed the title ~~[Better Tests] Integration tests for the GCE metadata emulator~~ [Better Tests] Integration tests for the GCE metadata server fake Sep 30, 2026

IvanBM18 changed the title ~~[Better Tests] Integration tests for the GCE metadata server fake~~ [Better Tests] Integration tests for the GCE metadata fake Sep 30, 2026

IvanBM18 force-pushed the better-tests/02a-metadata-emulator-tests branch from 8d0905b to 45c0d90 Compare September 30, 2026 20:43

IvanBM18 requested a review from dylanjew September 30, 2026 20:48

dylanjew reviewed Sep 30, 2026

View reviewed changes

IvanBM18 requested a review from decoNR September 30, 2026 21:50

IvanBM18 added ready for GOSST review Pull requests ready for GOSST team review Tests PRs focused merely on tests suite changes labels Sep 30, 2026

IvanBM18 force-pushed the better-tests/02a-metadata-emulator-tests branch from 45c0d90 to a551560 Compare October 2, 2026 17:12

decoNR approved these changes Oct 5, 2026

View reviewed changes

Comment thread test_suites/fixtures/gce_metadata_server_test.py

Comment thread test_suites/fixtures/gce_metadata_server_test.py Outdated

IvanBM18 added 8 commits October 6, 2026 16:19

[Better test] Removes warning in a fixture

66602dd

[Better tests] Expect the WireMock container error when an emulator a…

1ec262e

…ddress is taken

[Better-tests] Asserts against a dict instead of hardcoded values

8e8433d

[Better tests] Move test to dir for the new test suite

040618d

[Better tests] Format docstrings in metadata emulator test for line l…

2d8dba5

…ength

[Better tests] Format docstrings in metadata emulator test for line l…

d4a77b3

…ength

[Better tests] Makes pydoc string in 404 test more clear

d92f6b3

IvanBM18 force-pushed the better-tests/02a-metadata-emulator-tests branch from ea6a684 to d92f6b3 Compare October 6, 2026 22:19

IvanBM18 merged commit 76409d8 into master Oct 6, 2026
14 checks passed

IvanBM18 deleted the better-tests/02a-metadata-emulator-tests branch October 6, 2026 22:23

Navigation Menu

Repository navigation

[Better Tests] Integration tests for the GCE metadata fake - #5486

[Better Tests] Integration tests for the GCE metadata fake#5486
IvanBM18 merged 8 commits into
better-tests/02-metadata-emulator-corefrom
better-tests/02a-metadata-emulator-tests

IvanBM18 commented Sep 23, 2026 •

edited

Loading

Uh oh!

IvanBM18 Sep 29, 2026 •

edited

Loading

Uh oh!

dylanjew Sep 29, 2026

Uh oh!

IvanBM18 Sep 30, 2026

Uh oh!

dylanjew Sep 29, 2026

Uh oh!

dylanjew Sep 29, 2026

Uh oh!

dylanjew Sep 29, 2026

Uh oh!

IvanBM18 Sep 30, 2026

Uh oh!

dylanjew Sep 30, 2026

Uh oh!

Uh oh!

dylanjew left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

IvanBM18 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Note

Changes

Tests

📚 Stacked PRs

Uh oh!

IvanBM18 Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dylanjew Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

dylanjew Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

dylanjew Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

dylanjew Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

dylanjew Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dylanjew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

IvanBM18 commented Sep 23, 2026 •

edited

Loading

IvanBM18 Sep 29, 2026 •

edited

Loading