[Better Tests] Adds new test dependencies by IvanBM18 · Pull Request #5475 · google/clusterfuzz

IvanBM18 · 2026-09-17T20:24:14Z

Bug: b/555367430

This is our first ever PR focused on adding better tests to CF. Starting by adding new dependencies:

pytest: This is the python's modern alternative to unittest allows to simplify test setup, and overall added for reusable mock creation across multiple modules.
pytest-cov: To create test coverage reports over pytest tests.

testcontainers: Allow us to launch containers with real-like external dependencies like datastore., pub sub queue, or in our case, even other CF worker bots, etc.
wiremock: Used alongside testcontainers to create mock/double containers specialized in api testing

I also added a sample test .py script to verify that testcontainers & pytest setup is correct, this script is not executed at CI, since we are not yet ready for this.
EDDIT:
If curious see the commit history of this PR to see the script

For more info read:
https://docs.google.com/document/d/1OcvLED_sEJS0Y8RcluL39qqW4v8-liFxiG9eyV20s8c/edit?usp=sharing&resourcekey=0-Qca3yUhT4VsnYnUe74T_Iw

JuanMBriones

LGTM!

dylanjew

Could you add a md documentation for this environment and running these tests?

dylanjew · 2026-09-23T18:53:29Z

+from testcontainers.core.container import DockerContainer
+from testcontainers.core.wait_strategies import LogMessageWaitStrategy
+
+IMAGE = 'alpine:3.20'


how did you get this value? should we get this from another config file?

As long as i remembered, the alpine image in general has been the most lightweight linux image i can think of , it has basically nothing.

So i googled Alpine image and saw this version as the sable version, though double checking the release history for this, seems like there are newer releases, still WDYT?

dylanjew · 2026-09-23T18:54:14Z

+    import docker  # pylint: disable=import-outside-toplevel
+    docker.from_env().ping()
+    return True
+  except Exception:  # pylint: disable=broad-except


is this when docker isn't set up? Should we log something here rather than failing silently? or is there another error message when this is run without docker?

Yes, and yes, just added a print(on purpose because these file is not on butler, hence i can't import a log library here).

dylanjew · 2026-09-23T18:58:52Z

+tests under src/clusterfuzz/_internal/tests/{core,appengine}, so this file is
+never collected by the `core` or `appengine` targets that CI runs. The
+`test_` prefix (rather than a `_test.py` suffix) is also intentional: it keeps
+pytest collection working while opting out of butler's lint rule requiring an


I agree that we should keep our existing tests but introducing a new prefix syntax is confusing and easily missed when adding new tests.

The issue is that if we use the existing _test.py syntax, the old lint rules would be applied to these new pytest files? Maybe there's a way to enforce this with a test decorator and changing the lint rules?

The linting rule is inforced not trough pylint, but trough a custom script:
https://github.com/google/clusterfuzz/blob/master/src/local/butler/lint.py#L139

IvanBM18 · 2026-09-24T04:04:59Z

Could you add a md documentation for this environment and running these tests?

Im seeing to much confusion regarding this file, this one is merely for demonstration purposes or for a dev to run manually, and is never intended to be part of our new test suite, so maybe its best that i delete it in that case, WDYT? @dylanjew

dylanjew · 2026-09-24T13:49:09Z

Could you add a md documentation for this environment and running these tests?

Im seeing to much confusion regarding this file, this one is merely for demonstration purposes or for a dev to run manually, and is never intended to be part of our new test suite, so maybe its best that i delete it in that case, WDYT? @dylanjew

This one is just a test file, but I'm wondering whether all tests going forward are going to use this new syntax, or if just this test file will be different. I'd like to see how you plan to run new tests and what they'd look like. So yes, maybe that would be more clear in the next PRs.

We can keep the test file if it's helpful for local testing. If we don't expect people to use it to debug their local env we can probably delete it and rely on the follow up tests instead.

IvanBM18 · 2026-09-24T17:08:37Z

This one is just a test file, but I'm wondering whether all tests going forward are going to use this new syntax, or if just this test file will be different. I'd like to see how you plan to run new tests and what they'd look like. So yes, maybe that would be more clear in the next PRs.

We can keep the test file if it's helpful for local testing. If we don't expect people to use it to debug their local env we can probably delete it and rely on the follow up tests instead.

Deleted the file to avoid confusion, plus the new tests should inherently do the same, hence what this tested, will be asserted this inherently.

We can discuss file name syntax, plus other naming conventions in a PR focused on adding the first tests(if curious that one is a draft right now), this one original purpose is just the dependencies, sorry for the confusions this might have caused

dylanjew · 2026-09-25T17:14:15Z

This one is just a test file, but I'm wondering whether all tests going forward are going to use this new syntax, or if just this test file will be different. I'd like to see how you plan to run new tests and what they'd look like. So yes, maybe that would be more clear in the next PRs.
We can keep the test file if it's helpful for local testing. If we don't expect people to use it to debug their local env we can probably delete it and rely on the follow up tests instead.

Deleted the file to avoid confusion, plus the new tests should inherently do the same, hence what this tested, will be asserted this inherently.

We can discuss file name syntax, plus other naming conventions in a PR focused on adding the first tests(if curious that one is a draft right now), this one original purpose is just the dependencies, sorry for the confusions this might have caused

sounds good!

decoNR

I am excited about the "Better Tests" project! We will all benefit from it.

Bug: b/555370359 We can redirect `google-auth` metadata requests to a local emulator via the `GCE_METADATA_HOST` environment variable, but ClusterFuzz's own `compute_metadata.py` previously hardcoded `metadata.google.internal` and port `80`. Reading `GCE_METADATA_HOST` and parsing optional ports allows local integration tests to run a metadata emulator on an ephemeral unprivileged port without root privileges or `/etc/hosts` changes. ## Changes - Update `_METADATA_SERVER` in `compute_metadata.py` to read `GCE_METADATA_HOST` from the environment, falling back to `metadata.google.internal`. - Add unit tests in covering `_metadata_host_port()`, `is_gce()` on non-default ports, and `GCE_METADATA_HOST` overrides. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | **1** | **#5479** | **`[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata`** *(👉 This PR)* | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Split out of #5480, which had grown to 821 lines across the fixtures, the emulator that serves them, and the tests that exercise it. This PR is the data half: the JSON configs the GCE metadata emulator seeds itself from. Each config describes one worker identity, so a test can pick whether it runs trusted or untrusted by picking which fixture answers on the default address. The shape of `metadata` mirrors the real metadata server's hierarchy, so a config can be retrieved by reading a live instance: ``` curl -H 'Metadata-Flavor: Google' \ 'http://metadata.google.internal/computeMetadata/v1/?recursive=true' | jq ``` ## Changes - Add `local/emulators/configs/tworker.json`: the trusted worker identity, with its service account email, scopes, a fake access token, and the project and instance metadata a tworker reads. - Add `local/emulators/configs/uworker.json`: the untrusted counterpart, so both can run side by side and be shown to stay distinct. - Add `local/emulators/configs/template.json`: an annotated reference for writing a new config. It is documentation rather than a loadable fixture, so it keeps its `//` comments; nothing parses it. `tworker.json` and `uworker.json` are plain JSON and are the only two loaded. Nothing reads these yet. The emulator that serves them is #5480, next in the stack. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | **2** | **#5485** | **`[Better Tests] Add GCE metadata emulator worker fixtures`** *(👉 This PR)* | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Creates a GCE metadata emulator for our future integration test suite, built on Testcontainers and WireMock. It lets tests resolve credentials and read instance and project metadata without a real GCE host or a local `gcloud` login, and lets them distinguish a trusted from an untrusted environment. `google-auth` and our vendored `oauth2client` evaluate the `GCE_METADATA_*` environment variables once, at import time, and bake the resulting URL in. Importing this module therefore claims two loopback addresses up front; the addresses never move, only what listens on them does. A test picks its in-process identity by picking which fixture answers on the default address: ```python with gce_metadata_emulator.trusted_as_default() as tworker: ... with gce_metadata_emulator.untrusted_as_default() as uworker: ... with gce_metadata_emulator.trusted_untrusted_pair() as (tworker, uworker): ... # uworker sits on the secondary address ``` Nothing is patched to make that work. An emulator on the secondary address is reachable through the client its context manager yields, or by handing its env to a child process: ```python subprocess.run(argv, env={**os.environ, **uworker.env}) ``` This emulator was based off the pre-existing [local emulator go wrap](https://github.com/google/clusterfuzz/blob/master/local/emulators/metadata.go) + what is detailed at the [GCP metadata server docs](https://docs.cloud.google.com/compute/docs/metadata/overview) ## Changes - Add `gce_metadata_emulator.py`, a WireMock Testcontainer fake GCE metadata server. Supports: - Serving fake service account credentials, so a test can tell a trusted from an untrusted environment. - Serving project- and instance-scoped metadata values, including attribute shadowing, seeded from the JSON configs added in #5485. - Rejecting requests that omit `Metadata-Flavor: Google`, the way the real server does. - Failing loudly when two emulators want the same address. The tests covering all of this land in #5486, next in the stack. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | **3** | **#5480** | **`[Better Tests] Add the WireMock GCE metadata emulator`** *(👉 This PR)* | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

Bug: b/555370359 Split out of #5480. This PR wires the GCE metadata emulator added in #5480 into the test runner, and covers it. ## Note - This is the first of many integration test modules, for this purpose, it uses `pytest` library instead of `unittest`, this requires different syntax for asserting, but files & methods naming is basically the same. - This is a experimental integration test module, still a standalone, we require to wire a couple of things with a butler command to allow easy execution of this kind of tests like we do with unit test, this will be done in a subsequent bug: b/540876792 - Documentation will be created in b/555424439 once we have our first set of integration test suites & the butler command is ready to run them - I aim to create documentation & guidelines to be created based off this first tests and overall the discussion we have in the PRs for them, so feel free to push back on even the smallest details, nothing is set in stone! ## Changes - Adds `gce_metadata_emulator_test.py`, skipped right now because its experimental, just in case this ends in the CI process before we intended to. ## Tests - `GceMetadataEmulatorTest` an integration test for how some common workflows behave using the new emulator, runs a trusted and an untrusted emulator side by side and verifies: - `GceMetadataEmulatorUntrustedDefaultTest` puts a metadata server that returns uworker on the default address and verifies that all resolve to the untrusted identity. To run it: ``` PYTHONPATH=src:src/third_party pytest test_suites/fixtures/gce_metadata_server_test.py ``` We need to later include them in `butler.py` so that we can: - Run them smoothly - Don't require to do the imports manually But theres already a bug for that, and the purpose of this is not creating the butler setup but to create an emulator for other tests. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | **4** | **#5486** | **`[Better Tests] Bootstrap and cover the GCE metadata emulator`** *(👉 This PR)* | `better-tests/02-metadata-emulator-core` | | 5 | #5481 | `[Better Tests] Add fault injection and runtime overrides to metadata emulator` | `better-tests/02a-metadata-emulator-tests` |

…data fake values (#5481) Bug: b/555370359 Following the parent PR, this one introduces changes for our emulator so that we can force it to fail either constantly, or on demand, this opens a ton of possibilities for integration tests, this is technically also a new feature, but to simplify reviews i moved this one to a separate PR. With this changes it now simulates transient GCE metadata server failures (such as HTTP 500s or latency spikes) and mutate instance/project attributes dynamically during a test without restarting the WireMock container. Using WireMock stateful scenarios allows injecting faults that automatically expire after N requests so retry decorators (such as `@retry.wrap` on `compute_metadata.get()`) can be tested deterministically. ## Changes - Adds an interface(for composition instead of inheritance) for so that we can inject fault mapping to any API that uses wiremock as its base starting with the metadata_emulator. - Track runtime instance attribute overrides via `_runtime_instance_attributes` so `set_project_attribute()` does not overwrite attributes shadowed at runtime, and reset them in `clear_faults()`. - Adds tests cases for this. ### 📚 Stacked PRs | Step | PR | Title | Base Branch | | :---: | :--- | :--- | :--- | | 0 | #5475 | `[Better Tests] Adds new test dependencies` | `master` | | 1 | #5479 | `[Better Tests] Support GCE_METADATA_HOST with port in compute_metadata` | `better-test/dependencies` | | 2 | #5485 | `[Better Tests] Add GCE metadata emulator worker fixtures` | `better-tests/01-compute-metadata-host` | | 3 | #5480 | `[Better Tests] Add the WireMock GCE metadata emulator` | `better-tests/01a-metadata-emulator-configs` | | 4 | #5486 | `[Better Tests] Bootstrap and cover the GCE metadata emulator` | `better-tests/02-metadata-emulator-core` | | **5** | **#5481** | **`[Better Tests] Add fault injection and runtime overrides to metadata emulator`** *(👉 This PR)* | `better-tests/02a-metadata-emulator-tests` |

[Better Tests] Adds new test dependencies

d4b482f

IvanBM18 self-assigned this Sep 17, 2026

[Better Tests] Renames file to avoid linting issues

e2a18b7

IvanBM18 requested review from JuanMBriones, Xeicker and dylanjew September 17, 2026 20:37

IvanBM18 marked this pull request as ready for review September 17, 2026 20:37

IvanBM18 requested a review from a team as a code owner September 17, 2026 20:37

[Better tests] Update copyright notice year

a5fc207

JuanMBriones reviewed Sep 17, 2026

View reviewed changes

Comment thread src/clusterfuzz/tests/dev_env_testcontainers_test.py Outdated

Comment thread src/clusterfuzz/tests/dev_env_testcontainers_test.py Outdated

Comment thread src/clusterfuzz/tests/dev_env_testcontainers_test.py Outdated

[Better Tests] Nit: adds styling formatting in script doc

320f4ce

IvanBM18 added the ready for GOSST review Pull requests ready for GOSST team review label Sep 18, 2026

[Better Tests] Add wiremock 2.7.0 dev dependency

5846ed0

IvanBM18 force-pushed the better-test/dependencies branch from c00fea1 to 5846ed0 Compare September 22, 2026 17:25

dylanjew reviewed Sep 23, 2026

View reviewed changes

[Better Tests] Renames test file

51aaf51

IvanBM18 removed the ready for GOSST review Pull requests ready for GOSST team review label Sep 24, 2026

[Better Tests] Renames test file

a4874a5

decoNR reviewed Sep 24, 2026

View reviewed changes

Comment thread .gitignore Outdated

[Better Tests] Deletes demonstration script, plus handles nit comments

33d4e7b

IvanBM18 added this pull request to stack #5497 September 25, 2026 01:44

IvanBM18 added the ready for GOSST review Pull requests ready for GOSST team review label Sep 25, 2026

IvanBM18 added the Tests PRs focused merely on tests suite changes label Sep 25, 2026

decoNR approved these changes Sep 25, 2026

View reviewed changes

IvanBM18 merged commit 7f2449b into master Sep 28, 2026
20 checks passed

IvanBM18 deleted the better-test/dependencies branch September 28, 2026 16:54

Conversation

IvanBM18 commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JuanMBriones left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dylanjew left a comment

Choose a reason for hiding this comment

Uh oh!

dylanjew Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

dylanjew Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dylanjew Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

IvanBM18 commented Sep 24, 2026

Uh oh!

dylanjew commented Sep 24, 2026

Uh oh!

Uh oh!

IvanBM18 commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dylanjew commented Sep 25, 2026

Uh oh!

decoNR left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

IvanBM18 commented Sep 17, 2026 •

edited

Loading

IvanBM18 commented Sep 24, 2026 •

edited

Loading

decoNR left a comment •

edited

Loading