Skip to content

Add MockWebServer-based tests for BigqueryClient/BigqueryJavaOutputPlugin - #110

Merged
chikamura merged 10 commits into
feature/ruby-compatibilityfrom
feature/mockwebserver-e2e-tests
Sep 25, 2026
Merged

chikamura merged 10 commits into
feature/ruby-compatibilityfrom
feature/mockwebserver-e2e-tests

Conversation

@chikamura

@chikamura chikamura commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a MockWebServer-based test suite (TestBigqueryClientWithMockServer, TestBigqueryJavaOutputPluginWithMockServer) for BigqueryClient and BigqueryJavaOutputPlugin. To let ./gradlew test verify behavior without talking to the real BigQuery API, adds a way for BigqueryClient to connect to a test server, along with a new utility class for it, BigqueryTestHostSupport.java.

  • Adds a test_host config option. It only takes effect when the TEST_HOST_ENABLED environment variable is also set, which in practice only happens under ./gradlew test.
    • test_host is used for all of BigqueryClient's BigQuery API calls.
    • test_host also overrides load()'s behavior: google-cloud-bigquery:2.14.0 ignores test_host (BigQueryOptions#setHost()) for one of load()'s connections and hardcodes it to https://www.googleapis.com, so this override works around that.
  • Adds TestBigqueryClientWithMockServer and TestBigqueryJavaOutputPluginWithMockServer, covering:
    • Retry/give-up behavior for load, copy, query, merge, and createTableIfNotExist jobs
    • updateTableIfNeed()'s interaction between mode, retain_column_descriptions/retain_column_policy_tags, and column_options[].description
    • An end-to-end run per mode (append, append_direct, replace, delete_in_advance, merge) against a mocked BigQuery API
  • Adds a com.squareup.okhttp3:mockwebserver:4.11.0 dependency and environment "TEST_HOST_ENABLED", "true" to the test task in build.gradle.

Not in scope

Several known Ruby-vs-Java behavior differences are documented as TODO comments in the new tests (not fixed here):

  • column_options[].description only applies to mode: replace in Java, whereas Ruby applies it regardless of mode
  • The plugin's retries config is only consulted by the job-polling loops (load/copy/query); it has no effect on other API calls such as createTableIfNotExist(), getTable(), deleteTable(), createDataset(), or updateTableIfNeed()'s PATCH — those rely on the underlying BigQuery client library's own default retry behavior instead
  • If the load, copy, or merge job itself fails after exhausting retries (e.g. an API error), the run aborts before ever reaching the temp table delete step, leaving the temp table behind
  • A failed temp-table delete blocks the schema update that would otherwise follow it
  • A partition decorator in the table name is passed straight through to createTableIfNotExist() (Ruby strips it)

This PR is test coverage only; those are left as follow-ups.

Test plan

  • ./gradlew test — 104 tests pass
  • ./gradlew spotlessApply

chikamura and others added 7 commits August 28, 2026 14:30
Adds a test-only test_host config option that redirects BigqueryClient
at a local MockWebServer instead of the real BigQuery API and bypasses
credential setup. BigqueryClient#load() also routes load jobs through
jobs.insert in this mode, since TableDataWriteChannel's resumable
upload session hardcodes the real BigQuery host regardless of
BigQueryOptions#getHost() (a google-cloud-bigquery:2.14.0 limitation).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds bounded RetrySettings for the test-host BigQuery client so retry
tests don't wait out the library's real default backoff, refactors
getBigQueryService() to apply setProjectId() once regardless of branch,
and requires the TEST_HOST_ENABLED environment variable (set only by
the `test` Gradle task) in addition to the test_host config, so a
plugin config alone can no longer redirect a real embulk run at an
arbitrary unauthenticated host.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…aOutputPlugin

Introduces a shared MockWebServer test harness (BigqueryMockWebServerTestUtil,
AssertUtil) and builds out coverage for BigqueryClient (retry behavior for
load/copy/query/createTableIfNotExist, updateTableIfNeed's interaction with
mode/retain flags/column_options) and BigqueryJavaOutputPlugin (an
end-to-end run per mode against a mocked BigQuery API). Also consolidates
duplicated request/response fixtures into the shared util as they emerged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
merge_keys was only ever needed to keep testRunMergeModeWithMockWebServer
from triggering the INFORMATION_SCHEMA merge-key lookup; set it there
explicitly instead of baking it into every test's default config.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
is_skip_job_result_check defaulted to true in test_host.yml, so no
MockWebServer test ever exercised getTransactionReport()'s row-count
check or the temp table row-count lookup it performs. Flip it to false
(matching the plugin's own production default) and update the affected
tests: add outputRows to load job responses, enqueue the extra temp
table GET, and shift request indices accordingly.

Also clear BigqueryUtil's static FileWriterHolder map between tests
(test-only: a real embulk run never calls transaction() twice in one
JVM, but JUnit reuses the JVM/threads across test methods, which was
inflating num_input_rows with stale writers from earlier tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Moves the MockWebServer-based tests out of TestBigqueryClient and
TestBigqueryJavaOutputPlugin into new TestBigqueryClientWithMockServer
and TestBigqueryJavaOutputPluginWithMockServer classes, leaving the
original classes containing only their pre-existing, non-MockServer
tests. Drops the now-redundant "WithMockWebServer" suffix from
individual test method names, since the class name already says so,
and updates a README reference to the class that used to own this
behavior.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@chikamura chikamura changed the title [WIP] Add test_host seam for MockWebServer-based BigQuery API tests Add test_host seam for MockWebServer-based BigQuery API tests Sep 4, 2026
@chikamura chikamura changed the title Add test_host seam for MockWebServer-based BigQuery API tests Add MockWebServer-based tests for BigqueryClient/BigqueryJavaOutputPlugin Sep 4, 2026
@chikamura
chikamura requested a review from john-abre September 4, 2026 05:45
@chikamura
chikamura marked this pull request as ready for review September 4, 2026 05:45
… host

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@chikamura
chikamura changed the base branch from master to feature/ruby-compatibility September 8, 2026 02:21
First run against a fresh dataset: the destination table doesn't exist,
so autoCreate()'s tables.delete gets a 404. The client library maps that
to BigQuery#delete() returning false, and the plugin ignores the return
value, so the run proceeds exactly as when the delete succeeds. Pin that
down with a mock-server run that returns 404 on the initial delete.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@john-abre john-abre left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ありがとうございます!&非常に遅くなって申し訳無かったです 🙇

  • 必須じゃないですがテーブルのない時のdelete_in_advance動作確認したければ #121 おまけ用意してみました 🙇

  • テストは「今の状態」の確認で、PRによってRubyっぽくなる時は更新の必要性は承知です 🙇 (具体的にPR1, PR3, PR4, PR5, PR6に合わせる更新が期待)

メモ for 個人理解:

  • BigqueryTestHostSupport creates BigQueryOptions that send BigQuery requests to mock server
  • It also provides a “createLoadJob()” which is used inside of load() to make jobs.insert POST (to mock server) rather than calling Google servers

…-table

Add delete_in_advance test for a missing destination table
@chikamura
chikamura merged commit e7b6d2b into feature/ruby-compatibility Sep 25, 2026
@chikamura
chikamura deleted the feature/mockwebserver-e2e-tests branch September 25, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants