[Better Tests] Add error injection and runtime overrides for GCE metadata fake values by IvanBM18 · Pull Request #5481 · google/clusterfuzz

IvanBM18 · 2026-09-22T21:37:04Z

Bug: b/555370359

Following the parent PR, this one introduces changes for our emulator so that we can force it to fail either constantly, or on demand, this opens a ton of possibilities for integration tests, this is technically also a new feature, but to simplify reviews i moved this one to a separate PR.

With this changes it now simulates transient GCE metadata server failures (such as HTTP 500s or latency spikes) and mutate instance/project attributes dynamically during a test without restarting the WireMock container. Using WireMock stateful scenarios allows injecting faults that automatically expire after N requests so retry decorators (such as @retry.wrap on compute_metadata.get()) can be tested deterministically.

Changes

Adds an interface(for composition instead of inheritance) for so that we can inject fault mapping to any API that uses wiremock as its base starting with the metadata_emulator.
Track runtime instance attribute overrides via _runtime_instance_attributes so set_project_attribute() does not overwrite attributes shadowed at runtime, and reset them in clear_faults().
Adds tests cases for this.

📚 Stacked PRs

Step	PR	Title	Base Branch
0	#5475	`[Better Tests] Adds new test dependencies`	`master`
1	#5479	`[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata`	`better-test/dependencies`
2	#5485	`[Better Tests] Add GCE metadata emulator worker fixtures`	`better-tests/01-compute-metadata-host`
3	#5480	`[Better Tests] Add the WireMock GCE metadata emulator`	`better-tests/01a-metadata-emulator-configs`
4	#5486	`[Better Tests] Bootstrap and cover the GCE metadata emulator`	`better-tests/02-metadata-emulator-core`
5	#5481	`[Better Tests] Add fault injection and runtime overrides to metadata emulator` (👉 This PR)	`better-tests/02a-metadata-emulator-tests`

dylanjew

LGTM

IvanBM18 · 2026-10-02T20:49:50Z

    Config.base_url = self.admin_url
    Mappings.create_mapping(mapping)

-  def _all_mappings(self) -> list[Mapping]:


Removed because the only caller was _remove_existing_faults_for_path(), which is now part of the common logic in WireMockFaultInjector

IvanBM18 · 2026-10-02T20:52:11Z

+}
+
+
+class FaultInjector(abc.ABC):


Created this baseclas in case in the future we need to inject fault to a fake/proxy api, that doesn't follow the wireMock which i see as a possibility, since Wiremock has the restriction that responses are static for the matching given path + payload.

…emulator

…n interface

…#5517) Bug: b/555379018 ## Overview The following Pr stack focuses in the creation of a swarming API fake to be used by the integration tests suite. For this purpose, we perform a small refactor in the swarming package, focused on making some constants public so that they can be reused in our fake api, as well as some general adjustments in the API initialization so that we can give it http paths. ## Changes - Creates `constants.py` which consists of swarming related constants - Now the swarming server url must be given including `https://` or `https://` this way we give the api the flexiblity of using non secure urls such as ones coming from localhost. - There are following changes in the swarming config in the [config repo](https://clusterfuzz-config-472119376969.us-central1.sourcemanager.dev/clusterfuzz-testing/clusterfuzz-config/pulls/387) ## Tests - Added UT for the new validation in the API

'Fault' was an unclear and overly specific term. 'Error' is the more general and widely understood word for what the injector does: make the emulator answer with an HTTP error status (optionally after a delay). - wiremock_faults.py -> wiremock_errors.py - FaultInjector / WireMockFaultInjector -> ErrorInjector / WireMockErrorInjector - inject_fault / clear_faults -> inject_error / clear_errors - fault_injector ctor arg -> error_injector - WireMock mapping metadata keys 'fault' / 'fault_path' -> 'error' / 'error_path'

Bug: b/555370359 We can redirect `google-auth` metadata requests to a local emulator via the `GCE_METADATA_HOST` environment variable, but ClusterFuzz's own `compute_metadata.py` previously hardcoded `metadata.google.internal` and port `80`. Reading `GCE_METADATA_HOST` and parsing optional ports allows local integration tests to run a metadata emulator on an ephemeral unprivileged port without root privileges or `/etc/hosts` changes. ## Changes - Update `_METADATA_SERVER` in `compute_metadata.py` to read `GCE_METADATA_HOST` from the environment, falling back to `metadata.google.internal`. - Add unit tests in covering `_metadata_host_port()`, `is_gce()` on non-default ports, and `GCE_METADATA_HOST` overrides. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | **1** | **#5479** | **`[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata`** *(👉 This PR)* | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Split out of #5480, which had grown to 821 lines across the fixtures, the emulator that serves them, and the tests that exercise it. This PR is the data half: the JSON configs the GCE metadata emulator seeds itself from. Each config describes one worker identity, so a test can pick whether it runs trusted or untrusted by picking which fixture answers on the default address. The shape of `metadata` mirrors the real metadata server's hierarchy, so a config can be retrieved by reading a live instance: ``` curl -H 'Metadata-Flavor: Google' \ 'http://metadata.google.internal/computeMetadata/v1/?recursive=true' | jq ``` ## Changes - Add `local/emulators/configs/tworker.json`: the trusted worker identity, with its service account email, scopes, a fake access token, and the project and instance metadata a tworker reads. - Add `local/emulators/configs/uworker.json`: the untrusted counterpart, so both can run side by side and be shown to stay distinct. - Add `local/emulators/configs/template.json`: an annotated reference for writing a new config. It is documentation rather than a loadable fixture, so it keeps its `//` comments; nothing parses it. `tworker.json` and `uworker.json` are plain JSON and are the only two loaded. Nothing reads these yet. The emulator that serves them is #5480, next in the stack. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | **2** | **#5485** | **`[Better Tests] Add GCE metadata emulator worker fixtures`** *(👉 This PR)* | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Creates a GCE metadata emulator for our future integration test suite, built on Testcontainers and WireMock. It lets tests resolve credentials and read instance and project metadata without a real GCE host or a local `gcloud` login, and lets them distinguish a trusted from an untrusted environment. `google-auth` and our vendored `oauth2client` evaluate the `GCE_METADATA_*` environment variables once, at import time, and bake the resulting URL in. Importing this module therefore claims two loopback addresses up front; the addresses never move, only what listens on them does. A test picks its in-process identity by picking which fixture answers on the default address: ```python with gce_metadata_emulator.trusted_as_default() as tworker: ... with gce_metadata_emulator.untrusted_as_default() as uworker: ... with gce_metadata_emulator.trusted_untrusted_pair() as (tworker, uworker): ... # uworker sits on the secondary address ``` Nothing is patched to make that work. An emulator on the secondary address is reachable through the client its context manager yields, or by handing its env to a child process: ```python subprocess.run(argv, env={**os.environ, **uworker.env}) ``` This emulator was based off the pre-existing [local emulator go wrap](https://github.com/google/clusterfuzz/blob/master/local/emulators/metadata.go) + what is detailed at the [GCP metadata server docs](https://docs.cloud.google.com/compute/docs/metadata/overview) ## Changes - Add `gce_metadata_emulator.py`, a WireMock Testcontainer fake GCE metadata server. Supports: - Serving fake service account credentials, so a test can tell a trusted from an untrusted environment. - Serving project- and instance-scoped metadata values, including attribute shadowing, seeded from the JSON configs added in #5485. - Rejecting requests that omit `Metadata-Flavor: Google`, the way the real server does. - Failing loudly when two emulators want the same address. The tests covering all of this land in #5486, next in the stack. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | **3** | **#5480** | **`[Better Tests] Add the WireMock GCE metadata emulator`** *(👉 This PR)* | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Split out of #5480. This PR wires the GCE metadata emulator added in #5480 into the test runner, and covers it. ## Note - This is the first of many integration test modules, for this purpose, it uses `pytest` library instead of `unittest`, this requires different syntax for asserting, but files & methods naming is basically the same. - This is a experimental integration test module, still a standalone, we require to wire a couple of things with a butler command to allow easy execution of this kind of tests like we do with unit test, this will be done in a subsequent bug: b/540876792 - Documentation will be created in b/555424439 once we have our first set of integration test suites & the butler command is ready to run them - I aim to create documentation & guidelines to be created based off this first tests and overall the discussion we have in the PRs for them, so feel free to push back on even the smallest details, nothing is set in stone! ## Changes - Adds `gce_metadata_emulator_test.py`, skipped right now because its experimental, just in case this ends in the CI process before we intended to. ## Tests - `GceMetadataEmulatorTest` an integration test for how some common workflows behave using the new emulator, runs a trusted and an untrusted emulator side by side and verifies: - `GceMetadataEmulatorUntrustedDefaultTest` puts a metadata server that returns uworker on the default address and verifies that all resolve to the untrusted identity. To run it: ``` PYTHONPATH=src:src/third_party pytest test_suites/fixtures/gce_metadata_server_test.py ``` We need to later include them in `butler.py` so that we can: - Run them smoothly - Don't require to do the imports manually But theres already a bug for that, and the purpose of this is not creating the butler setup but to create an emulator for other tests. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | **4** | **#5486** | **`[Better Tests] Bootstrap and cover the GCE metadata emulator`** *(👉 This PR)* | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

This was referenced Sep 22, 2026

[Better Tests] Support custom host in compute_metadata #5479

Merged

[Better Tests] Adds the GCE metadata fake #5480

Merged

[Better Tests] Adds new test dependencies #5475

Merged

IvanBM18 force-pushed the better-tests/03-metadata-emulator-faults branch 9 times, most recently from 249bcd5 to ad95910 Compare September 23, 2026 17:03

IvanBM18 force-pushed the better-tests/03-metadata-emulator-faults branch from ad95910 to df96aa0 Compare September 23, 2026 17:07

IvanBM18 force-pushed the better-tests/03-metadata-emulator-faults branch from df96aa0 to d60eea1 Compare September 23, 2026 17:09

IvanBM18 commented Sep 23, 2026

View reviewed changes

Comment thread src/clusterfuzz/_internal/tests/core/google_cloud_utils/gce_metadata_emulator_test.py Outdated

IvanBM18 force-pushed the better-tests/03-metadata-emulator-faults branch from d60eea1 to 8e0ed8c Compare September 23, 2026 18:05

IvanBM18 force-pushed the better-tests/03-metadata-emulator-faults branch from 8e0ed8c to f69b6e1 Compare September 23, 2026 18:12

IvanBM18 changed the base branch from better-tests/02-metadata-emulator-core to better-tests/02a-metadata-emulator-tests September 23, 2026 18:13

This was referenced Sep 23, 2026

[Better Tests] Add GCE metadata emulator worker/uworker configs #5485

Merged

[Better Tests] Integration tests for the GCE metadata fake #5486

Merged

IvanBM18 force-pushed the better-tests/03-metadata-emulator-faults branch 2 times, most recently from f89d3c4 to 4600042 Compare September 24, 2026 06:11

IvanBM18 force-pushed the better-tests/03-metadata-emulator-faults branch from 4600042 to 1b3a73f Compare September 24, 2026 06:19

IvanBM18 added this pull request to stack #5497 September 25, 2026 01:44

IvanBM18 force-pushed the better-tests/03-metadata-emulator-faults branch from b0dbe2d to 7872b59 Compare September 30, 2026 20:43

IvanBM18 requested a review from dylanjew September 30, 2026 20:55

dylanjew reviewed Sep 30, 2026

View reviewed changes

IvanBM18 requested a review from decoNR September 30, 2026 21:51

IvanBM18 added ready for GOSST review Pull requests ready for GOSST team review Tests PRs focused merely on tests suite changes labels Sep 30, 2026

IvanBM18 force-pushed the better-tests/03-metadata-emulator-faults branch from d221062 to 2efdf40 Compare October 2, 2026 17:12

IvanBM18 requested a review from dylanjew October 2, 2026 20:42

IvanBM18 removed the ready for GOSST review Pull requests ready for GOSST team review label Oct 2, 2026

IvanBM18 commented Oct 2, 2026

View reviewed changes

IvanBM18 changed the title ~~[Better Tests] Add fault injection and runtime overrides to GCE metadata fake~~ [Better Tests] Add fault injection and runtime overrides for GCE metadata fake values Oct 2, 2026

dylanjew approved these changes Oct 2, 2026

View reviewed changes

IvanBM18 commented Oct 2, 2026

View reviewed changes

IvanBM18 added the ready for GOSST review Pull requests ready for GOSST team review label Oct 2, 2026

IvanBM18 requested a review from a team as a code owner October 6, 2026 19:53

decoNR approved these changes Oct 6, 2026

View reviewed changes

Comment thread test_suites/fixtures/common/wiremock_errors.py

IvanBM18 changed the title ~~[Better Tests] Add fault injection and runtime overrides for GCE metadata fake values~~ [Better Tests] Add error injection and runtime overrides for GCE metadata fake values Oct 6, 2026

IvanBM18 added 5 commits October 6, 2026 16:19

[Better Tests] Add fault injection and runtime overrides to metadata …

bceafb1

…emulator

[Better tests] Makes args clearer by passing them as named args in tests

7ddb134

[Better Tests] Extract shared WireMock fault injector with compositio…

6b6ff02

…n interface

IvanBM18 force-pushed the better-tests/03-metadata-emulator-faults branch from 0841a3b to 3547bc8 Compare October 6, 2026 22:19

IvanBM18 merged commit d5818f2 into master Oct 6, 2026
14 checks passed

IvanBM18 deleted the better-tests/03-metadata-emulator-faults branch October 6, 2026 22:23

Navigation Menu

Repository navigation

[Better Tests] Add error injection and runtime overrides for GCE metadata fake values - #5481

[Better Tests] Add error injection and runtime overrides for GCE metadata fake values#5481
IvanBM18 merged 5 commits into
better-tests/02a-metadata-emulator-testsfrom
better-tests/03-metadata-emulator-faults

IvanBM18 commented Sep 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

dylanjew left a comment

Uh oh!

IvanBM18 Oct 2, 2026

Uh oh!

IvanBM18 Oct 2, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		}


		class FaultInjector(abc.ABC):

Conversation

IvanBM18 commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

📚 Stacked PRs

Uh oh!

Uh oh!

dylanjew left a comment

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

IvanBM18 commented Sep 22, 2026 •

edited

Loading