Repository navigation
Add MockWebServer-based tests for BigqueryClient/BigqueryJavaOutputPlugin - #110
Merged
chikamura merged 10 commits intoSep 25, 2026
Merged
Conversation
Adds a test-only test_host config option that redirects BigqueryClient at a local MockWebServer instead of the real BigQuery API and bypasses credential setup. BigqueryClient#load() also routes load jobs through jobs.insert in this mode, since TableDataWriteChannel's resumable upload session hardcodes the real BigQuery host regardless of BigQueryOptions#getHost() (a google-cloud-bigquery:2.14.0 limitation). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds bounded RetrySettings for the test-host BigQuery client so retry tests don't wait out the library's real default backoff, refactors getBigQueryService() to apply setProjectId() once regardless of branch, and requires the TEST_HOST_ENABLED environment variable (set only by the `test` Gradle task) in addition to the test_host config, so a plugin config alone can no longer redirect a real embulk run at an arbitrary unauthenticated host. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…aOutputPlugin Introduces a shared MockWebServer test harness (BigqueryMockWebServerTestUtil, AssertUtil) and builds out coverage for BigqueryClient (retry behavior for load/copy/query/createTableIfNotExist, updateTableIfNeed's interaction with mode/retain flags/column_options) and BigqueryJavaOutputPlugin (an end-to-end run per mode against a mocked BigQuery API). Also consolidates duplicated request/response fixtures into the shared util as they emerged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
merge_keys was only ever needed to keep testRunMergeModeWithMockWebServer from triggering the INFORMATION_SCHEMA merge-key lookup; set it there explicitly instead of baking it into every test's default config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
is_skip_job_result_check defaulted to true in test_host.yml, so no MockWebServer test ever exercised getTransactionReport()'s row-count check or the temp table row-count lookup it performs. Flip it to false (matching the plugin's own production default) and update the affected tests: add outputRows to load job responses, enqueue the extra temp table GET, and shift request indices accordingly. Also clear BigqueryUtil's static FileWriterHolder map between tests (test-only: a real embulk run never calls transaction() twice in one JVM, but JUnit reuses the JVM/threads across test methods, which was inflating num_input_rows with stale writers from earlier tests). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Moves the MockWebServer-based tests out of TestBigqueryClient and TestBigqueryJavaOutputPlugin into new TestBigqueryClientWithMockServer and TestBigqueryJavaOutputPluginWithMockServer classes, leaving the original classes containing only their pre-existing, non-MockServer tests. Drops the now-redundant "WithMockWebServer" suffix from individual test method names, since the class name already says so, and updates a README reference to the class that used to own this behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
chikamura
marked this pull request as ready for review
September 4, 2026 05:45
… host Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
First run against a fresh dataset: the destination table doesn't exist, so autoCreate()'s tables.delete gets a 404. The client library maps that to BigQuery#delete() returning false, and the plugin ignores the return value, so the run proceeds exactly as when the delete succeeds. Pin that down with a mock-server run that returns 404 on the initial delete. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 task done
john-abre
approved these changes
Sep 24, 2026
john-abre
left a comment
There was a problem hiding this comment.
ありがとうございます!&非常に遅くなって申し訳無かったです 🙇
-
必須じゃないですがテーブルのない時のdelete_in_advance動作確認したければ #121 おまけ用意してみました 🙇
-
テストは「今の状態」の確認で、PRによってRubyっぽくなる時は更新の必要性は承知です 🙇 (具体的にPR1, PR3, PR4, PR5, PR6に合わせる更新が期待)
メモ for 個人理解:
- BigqueryTestHostSupport creates BigQueryOptions that send BigQuery requests to mock server
- It also provides a “createLoadJob()” which is used inside of load() to make jobs.insert POST (to mock server) rather than calling Google servers
…-table Add delete_in_advance test for a missing destination table
10 of 21 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a MockWebServer-based test suite (
TestBigqueryClientWithMockServer,TestBigqueryJavaOutputPluginWithMockServer) forBigqueryClientandBigqueryJavaOutputPlugin. To let./gradlew testverify behavior without talking to the real BigQuery API, adds a way forBigqueryClientto connect to a test server, along with a new utility class for it,BigqueryTestHostSupport.java.test_hostconfig option. It only takes effect when theTEST_HOST_ENABLEDenvironment variable is also set, which in practice only happens under./gradlew test.test_hostis used for all ofBigqueryClient's BigQuery API calls.test_hostalso overridesload()'s behavior:google-cloud-bigquery:2.14.0ignorestest_host(BigQueryOptions#setHost()) for one ofload()'s connections and hardcodes it tohttps://www.googleapis.com, so this override works around that.TestBigqueryClientWithMockServerandTestBigqueryJavaOutputPluginWithMockServer, covering:createTableIfNotExistjobsupdateTableIfNeed()'s interaction betweenmode,retain_column_descriptions/retain_column_policy_tags, andcolumn_options[].descriptionappend,append_direct,replace,delete_in_advance,merge) against a mocked BigQuery APIcom.squareup.okhttp3:mockwebserver:4.11.0dependency andenvironment "TEST_HOST_ENABLED", "true"to thetesttask inbuild.gradle.Not in scope
Several known Ruby-vs-Java behavior differences are documented as
TODOcomments in the new tests (not fixed here):column_options[].descriptiononly applies tomode: replacein Java, whereas Ruby applies it regardless of moderetriesconfig is only consulted by the job-polling loops (load/copy/query); it has no effect on other API calls such ascreateTableIfNotExist(),getTable(),deleteTable(),createDataset(), orupdateTableIfNeed()'s PATCH — those rely on the underlying BigQuery client library's own default retry behavior insteadcreateTableIfNotExist()(Ruby strips it)This PR is test coverage only; those are left as follow-ups.
Test plan
./gradlew test— 104 tests pass./gradlew spotlessApply