Skip to content

[Dataproc] Add instance_flexibility_policy with provisioning_model_mix to secondary_worker_config in google_dataproc_workflow_template - #18948

Open
realdev12 wants to merge 3 commits into
GoogleCloudPlatform:mainfrom
realdev12:dataproc-flex-vm-secondary-worker
Open

realdev12 wants to merge 3 commits into
GoogleCloudPlatform:mainfrom
realdev12:dataproc-flex-vm-secondary-worker

Conversation

@realdev12

Copy link
Copy Markdown
Contributor

Purpose

Add support for Dataproc Flexible VMs (instance_flexibility_policy) and provisioning_model_mix to secondary_worker_config in google_dataproc_workflow_template. This enables secondary/preemptible worker groups to combine multiple machine shapes with mixed Standard and Spot/Preemptible VM capacities.

Implementation Scope & Design Decisions

To maintain parity with the existing google_dataproc_cluster resource and keep this PR reviewable and well-scoped:

  1. Provisioning Model Mix: provisioning_model_mix (standard_capacity_base, standard_capacity_percent_above_base) is supported exclusively on secondary_worker_config, adhering to Dataproc API requirements where provisioning model mixing applies only to secondary/preemptible workers.
  2. PR Stacking: Stacked on PR 2 (dataproc-flex-vm-master). Base branch is set to dataproc-flex-vm-master.

Tests & Verification

  • Generated code for both beta and ga providers.
  • Verified compilation in both downstream providers with 0 errors.
  • Extended acceptance test TestAccDataprocWorkflowTemplate_instanceFlexibilityPolicy exercising secondary_worker_config.instance_flexibility_policy along with provisioning_model_mix assertions.
dataproc: added `instance_flexibility_policy` and `provisioning_model_mix` to `secondary_worker_config` in `google_dataproc_workflow_template`

@github-actions

Copy link
Copy Markdown

Googlers: For automatic test runs see go/terraform-auto-test-runs.

@SirGitsalot, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

@modular-magician

modular-magician commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes for commit 9129add:

Diff report

Your PR generated the following diffs in downstream repositories:

Repository Diff Link Changes
google provider View Diff 6 files changed, 4769 insertions(+), 1583 deletions(-)
google-beta provider View Diff 6 files changed, 4792 insertions(+), 1606 deletions(-)

Test report

Analytics

Total Tests Passed Skipped Affected
93 85 6 2
Affected Service Packages
  • dataproc

Learn how VCR tests work


Step 1: Replaying Mode

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit.

Click here to see the affected tests
  • TestAccDataprocWorkflowTemplate_basic
  • TestAccDataprocWorkflowTemplate_instanceFlexibilityPolicy

View the replaying VCR build log


Step 2: Recording Mode

Recording Mode Replaying Rerun Test Name
✅ Log TestAccDataprocWorkflowTemplate_basic
✅ Log TestAccDataprocWorkflowTemplate_instanceFlexibilityPolicy

🟢 All tests passed!

View the recording VCR build log or the debug logs folder for detailed results.

@realdev12, @SirGitsalot VCR tests complete for 9129add!

@github-actions

Copy link
Copy Markdown

@SirGitsalot This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants