Skip to content

Omit deployment metadata from generated jobs and pipelines - #7219

Merged
shreyas-goenka merged 4 commits into
mainfrom
shreyas-goenka_data/shreyas/generate-omit-deployment-metadata
Oct 28, 2026
Merged

Omit deployment metadata from generated jobs and pipelines#7219
shreyas-goenka merged 4 commits into
mainfrom
shreyas-goenka_data/shreyas/generate-omit-deployment-metadata

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Changes

Omit the CLI-managed deployment block from generated job and pipeline YAML.

Why

Regenerating a deployed resource copies deployment_id and version_id from the API into configuration, which validation rejects. This breaks repeated generate job --bind for a history-enabled deployment.

Tests

New local tests.

@github-actions github-actions Bot added the DABs DABs related issues label Oct 28, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 6de33d6

Run: 40729999343

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 17 5:09
✅​ aws-windows-latest-4core-16gb 283 15 3:29
✅​ azure linux-2core-8gb 280 17 5:24
✅​ azure-windows-latest-4core-16gb 282 15 3:31
✅​ gcp linux-2core-8gb 281 17 5:21
🔄​ gcp-windows-latest-4core-16gb 2 281 15 3:40
Test Name gcp-windows-latest-4core-16gb
🔄​ TestFsCpDir 🔄​f
🔄​ TestFsCpDir/local_to_uc-volumes 🔄​f
Top 6 slowest tests (at least 2 minutes):
duration env testname
3:46 gcp linux-2core-8gb TestAccept
3:40 azure linux-2core-8gb TestAccept
3:29 aws linux-2core-8gb TestAccept
3:27 azure-windows-latest-4core-16gb TestAccept
3:23 aws-windows-latest-4core-16gb TestAccept
3:22 gcp-windows-latest-4core-16gb TestAccept

@shreyas-goenka
shreyas-goenka requested a review from denik October 28, 2026 14:04
@shreyas-goenka
shreyas-goenka marked this pull request as ready for review October 28, 2026 14:04
@shreyas-goenka
shreyas-goenka requested review from a team as code owners October 28, 2026 14:04
denik
denik approved these changes Oct 28, 2026
View reviewed changes

@denik denik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not caught by any acceptance test? Should we add one?

@shreyas-goenka

shreyas-goenka commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

It is not caught by any acceptance test? Should we add one?

yes, it was not caught. Sending acceptance tests as a followup.

@shreyas-goenka
shreyas-goenka added this pull request to the merge queue Oct 28, 2026
@shreyas-goenka
shreyas-goenka removed this pull request from the merge queue due to a manual request Oct 28, 2026
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/generate-omit-deployment-metadata branch from d046473 to 8af51ac Compare October 28, 2026 14:32
@shreyas-goenka shreyas-goenka mentioned this pull request Oct 28, 2026
Open
This was referenced Oct 28, 2026
Open
Merged
@shreyas-goenka
Support segment and recursive wildcards in fault patterns (#7238)
6de33d6 
## Changes
Support `*` within a path segment and `**` across path segments in
`fault.py` patterns. Both match zero or more characters; all other
characters are literal. Compile each pattern once when its rule is
registered..

## Why

useful for tests being added in
#7231.
@github-actions

github-actions Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Commit: 6de33d60e | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

3 tests: regression=0 unreleased=1 coverage=2 failing=0

test branch main (8b4a341) latest (v1.22.1)
TestAccept/bundle/dms/operation-upload-fails-once/DMS=true ✅ ✅ ❌
TestAccept/bundle/resources/dashboards/publish-failure-stale-content/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/dashboards/publish-failure-stale-content/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms/operation-upload-fails-once/DMS=true: branch ✅ | main (8b4a341) ✅ | latest (v1.22.1) ❌

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/dms/operation-upload-fails-once/output.txt
        +++ /tmp/TestAcceptbundledmsoperation-upload-fails-onceDMS=true338599140/002/output.txt
        @@ -14,7 +14,7 @@
           "lineage": "[UUID]",
           "serial": 1,
           "features": {
        -    "deployment_history_shadow": {}
        +    "deployment_history": {}
           }
         }
         
        @@ -39,18 +39,9 @@
          "lineage": "[UUID]",
          "serial": 1,
          "features": {
        -  "deployment_history_shadow": {}
        +  "deployment_history": {}
          },
        - "state": {
        -  "resources.schemas.foo": {
        -   "__id__": "main.dms_upload_fails_once_schema_[UNIQUE_NAME]",
        -   "state": {
        -    "catalog_name": "main",
        -    "comment": "v1",
        -    "name": "dms_upload_fails_once_schema_[UNIQUE_NAME]"
        -   }
        -  }
        - }
        + "state": {}
         }
         
         === A recreate records twice, and the first one fails: the second must still carry the sequence id staging left, because a rejected update earned no new one
        @@ -64,113 +54,13 @@
         This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost:
           recreate resources.schemas.foo
         Recreated schemas.foo
        -Warn: Failed to record deployment history: recording operation for resources.schemas.foo: Fault injected by test.
        -Files: 2 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 1 deleted, 0 unchanged
        +Error: recording operation for resources.schemas.foo: Fault injected by test. (500 INJECTED)
         
        -=== The service lists the schema with the failure
        ->>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-resources deployments/[DEPLOYMENT_ID] -o json
        -[
        -  {
        -    "last_action_type": "OPERATION_ACTION_TYPE_RECREATE",
        -    "last_version_id": "2",
        -    "resource_key": "schemas.foo"
        -  }
        -]
        +Endpoint: PATCH [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/operations/schemas.foo?update_mask=state%2Cerror_message%2Cstatus
        +HTTP Status: 500 Internal Server Error
        +API error_code: INJECTED
        +API message: Fault injected by test.
         
        -=== The raw recorded state after the failure
        ->>> print_state.py --no-dms
        -{
        - "state_version": 3,
        - "cli_version": "[CLI_VERSION]",
        - "lineage": "[UUID]",
        - "serial": 2,
        - "features": {
        -  "deployment_history_shadow": {}
        - },
        - "state": {
        -  "resources.schemas.foo": {
        -   "__id__": "dms_other_[UNIQUE_NAME].dms_upload_fails_once_schema_[UNIQUE_NAME]",
        -   "state": {
        -    "catalog_name": "dms_other_[UNIQUE_NAME]",
        -    "comment": "v1",
        -    "name": "dms_upload_fails_once_schema_[UNIQUE_NAME]"
        -   }
        -  }
        - }
        -}
        +Files: 2 uploaded, 0 deleted
         
        ->>> print_requests.py --dms //api/2.0/bundle
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
        -  "q": {
        -    "version_id": "2"
        -  },
        -  "body": {
        -    "cli_version": "[CLI_VERSION]",
        -    "operations": [
        -      {
        -        "action_type": "OPERATION_ACTION_TYPE_RECREATE",
        -        "resource_key": "schemas.foo"
        -      }
        -    ],
        -    "previous_version_id": "1",
        -    "version_type": "VERSION_TYPE_DEPLOY"
        -  }
        -}
        -{
        -  "method": "PATCH",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/operations/schemas.foo",
        -  "q": {
        -    "update_mask": "state,error_message,status"
        -  },
        -  "body": {
        -    "error_message": "",
        -    "sequence_id": 0,
        -    "status": "OPERATION_STATUS_PENDING"
        -  }
        -}
        -{
        -  "method": "PATCH",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/operations/schemas.foo",
        -  "q": {
        -    "update_mask": "state,error_message,resource_id,status"
        -  },
        -  "body": {
        -    "error_message": "",
        -    "resource_id": "dms_other_[UNIQUE_NAME].dms_upload_fails_once_schema_[UNIQUE_NAME]",
        -    "sequence_id": 0,
        -    "state": "{\"state\":{\"catalog_name\":\"dms_other_[UNIQUE_NAME]\",\"comment\":\"v1\",\"name\":\"dms_upload_fails_once_schema_[UNIQUE_NAME]\"}}",
        -    "status": "OPERATION_STATUS_SUCCEEDED"
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete",
        -  "body": {
        -    "completion_reason": "VERSION_COMPLETE_FAILURE"
        -  }
        -}
        -
        -=== A completion failure does not block destroy or remote file cleanup
        ->>> fault.py POST /api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/complete 400 0 1
        -
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.schemas.foo
        -
        -This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
        -  delete resources.schemas.foo
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-upload-fails-once-[UNIQUE_NAME]/default
        -
        -Warn: Failed to complete deployment history version: Fault injected by test.
        -Destroy: 1 deleted
        -
        ->>> print_requests.py --dms //versions/3 --oneline
        -{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/operations/schemas.foo", "q": {"update_mask": "state,error_message,status"}, "body": {"error_message": "", "sequence_id": 0, "status": "OPERATION_STATUS_SUCCEEDED"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
        -
        ->>> musterr env MSYS_NO_PATHCONV=1 [CLI] workspace get-status /Workspace/Users/[USERNAME]/.bundle/dms-upload-fails-once-[UNIQUE_NAME]/default/state/resources.json
        -Error: Path (/Workspace/Users/[USERNAME]/.bundle/dms-upload-fails-once-[UNIQUE_NAME]/default/state/resources.json) doesn't exist.
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.catalog: {
        LOG.catalog:   "catalog_type": "MANAGED_CATALOG",
        LOG.catalog:   "created_at": 1793206756885,
        LOG.catalog:   "created_by": "tester@databricks.com",
        LOG.catalog:   "full_name": "dms_other_r5ul2ov3u5hthgs36kpqjgqvze",
        LOG.catalog:   "metastore_id": "7a07bef4-fcc0-7e79-7a07-bef4fcc07f56",
        LOG.catalog:   "name": "dms_other_r5ul2ov3u5hthgs36kpqjgqvze",
        LOG.catalog:   "owner": "tester@databricks.com",
        LOG.catalog:   "updated_at": 1793206756885,
        LOG.catalog:   "updated_by": "tester@databricks.com"
        LOG.catalog: }

@shreyas-goenka
shreyas-goenka added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit c48c09c Oct 28, 2026
27 checks passed
@shreyas-goenka
shreyas-goenka deleted the shreyas-goenka_data/shreyas/generate-omit-deployment-metadata branch October 28, 2026 17:31
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: c48c09c

Run: 40736981791

Env 🔄​flaky ✅​pass 🙈​skip Time
🔄​ aws linux-2core-8gb 3 1496 1227 183:49
🔄​ aws-windows-latest-4core-16gb 3 1421 1252 162:01
🔄​ azure linux-2core-8gb 3 1343 1279 121:49
✅​ azure-windows-latest-4core-16gb 1271 1304 126:41
✅​ gcp linux-2core-8gb 1333 1284 129:25
✅​ gcp-windows-latest-4core-16gb 1258 1309 138:59
7 interesting tests: 7 flaky
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb
🔄​ TestAccept 🔄​f 🔄​f 🔄​f
🔄​ TestAccept/bundle/invariant/no_drift ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=genie_space.yml.tmpl/READPLAN= ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/resources/clusters/lifecycle-started 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/resources/clusters/lifecycle-started/DMS= 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/resources/clusters/resize-terminated-fallback ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS= ✅​p 🔄​f ✅​p
Top 50 slowest tests (at least 2 minutes):
duration env testname
25:36 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
23:23 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
17:31 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=
16:04 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
14:59 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
12:27 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
12:23 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
12:15 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
12:10 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=true
11:52 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
11:35 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=true
10:58 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
10:40 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
10:38 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
10:33 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:28 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
9:23 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=
9:14 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:12 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
9:10 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
8:46 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
8:27 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:10 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:05 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:59 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:49 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
7:47 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:47 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:38 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:24 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:24 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:12 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:12 aws linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=
7:11 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
7:06 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:05 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
7:05 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
7:05 aws linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
7:02 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:00 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=
6:57 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
6:55 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:53 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
6:50 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:45 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:44 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:42 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:42 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
6:36 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:24 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1

denik added a commit that referenced this pull request Oct 29, 2026
@denik Isaac
Omit deployment metadata from generated jobs and pipelines (main's #7219
f67c069 
 in the struct generators)

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@denik denik denik approved these changes

Assignees

No one assigned

Labels

DABs DABs related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shreyas-goenka @eng-dev-ecosystem-bot @denik