Skip to content

[PoC] Support fault injection in cloud acceptance runs - #7240

Draft
shreyas-goenka wants to merge 3 commits into
mainfrom
shreyas-goenka_data/shreyas/cloud-fault-poc
Draft

shreyas-goenka wants to merge 3 commits into
mainfrom
shreyas-goenka_data/shreyas/cloud-fault-poc

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Cloud acceptance tests can opt into Proxy = true and use existing fault.py commands. The proxy handles fault registration locally, blocks calls for normal faults, and preserves real upstream effects for --after-handler faults.

Reuse testserver.FaultRules and share its JSON registration logic. Keep rules scoped to the incoming synthetic test token, with separate SDK-authenticated upstream credentials. Replace the proxy's retrying API forwarding with HTTP forwarding that preserves statuses, bodies, queries, and response headers. Reject proxy configurations requiring unreachable localhost access from serverless.

Why

fault.py currently registers rules only on the fake server, so fault-based tests cannot run against a real workspace. Extending the existing proxy gives local and cloud tests the same registration and matching behavior.

Tests

  • Race-enabled matcher, fake-server, proxy, and acceptance-config tests.
  • New cloud-capable selftest/fault_proxy, with recording disabled: blocked mutation, committed mutation followed by an injected error, and recovery from an after-handler 503.
  • All five existing proxy selftests locally: basic requests, error forwarding, pipeline CRUD, workspace files, and volume I/O.
  • All 23 DMS acceptance directories locally.
  • Focused Go/Python lint, formatting, whitespace, and dead-code checks.

Cloud E2E on commit 079f9f3998b8 verified TestAccept/selftest/fault_proxy against real workspaces, with exactly one test selected and zero skipped in each Linux job:

Each run proved the pre-forwarding fault prevented a real mutation, the after-handler fault retained a real mutation, and the CLI recovered from an after-handler 503. The test cleans up its workspace directory and starts no compute.

That revision also passed the full CI lint, formatting, generated-file, and Linux DMS test checks. After keeping the matcher in testserver, focused race tests, all six proxy selftests, and Go lint pass locally; CI reruns on the update.

The corporate Go proxy currently blocks the repository's x/net v0.62.0 as recently published. Local validation used a temporary module override to cached v0.61.0; repository dependency files are unchanged.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Commit: f073dd4f8 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=0 unreleased=0 coverage=1 failing=0

test branch main (8b4a341) latest (v1.22.1)
TestAccept/selftest/fault_proxy ✅ ✅ ✅

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: f073dd4

Run: 40730729482

Env ❌​FAIL 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 282 15 5:39
❌​ aws-windows-latest-4core-16gb 2 282 13 4:01
✅​ azure linux-2core-8gb 281 15 5:31
❌​ azure-windows-latest-4core-16gb 2 281 13 5:00
✅​ gcp linux-2core-8gb 282 15 5:52
❌​ gcp-windows-latest-4core-16gb 2 1 281 13 4:39
Test Name aws-windows-latest-4core-16gb azure-windows-latest-4core-16gb gcp-windows-latest-4core-16gb
❌​ TestAccept ❌​F ❌​F ❌​F
❌​ TestAccept/selftest/fault_proxy ❌​F ❌​F ❌​F
🔄​ TestSyncIncrementalFileOverwritesFolder ✅​p ✅​p 🔄​f
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:57 aws linux-2core-8gb TestAccept
3:52 gcp linux-2core-8gb TestAccept
3:44 azure linux-2core-8gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants