Skip to content

Support bundle deployment bind for DMS-recorded deployments - #7127

Draft
shreyas-goenka wants to merge 1 commit into
mainfrom
isaac/dmsbind-register
Draft

shreyas-goenka wants to merge 1 commit into
mainfrom
isaac/dmsbind-register

Conversation

@shreyas-goenka

Copy link
Copy Markdown
Contributor

Changes

With experimental.deployment_history, bundle deployment bind no longer refuses. It runs the existing bind against a local copy of the resources DMS records (same prompt, same resolved state), then registers the result in a new version with one OPERATION_ACTION_TYPE_BIND operation, completed as success or failure. A first bind creates the deployment. As with a file-based bind, the workspace resource isn't changed; the next deploy applies config differences. Unbind with deployment history stays unsupported.

Why

DMS-recorded targets refused bind entirely. This follows the register-in-its-own-version pattern of #6989 without depending on it; the two can share a helper later. There is no unbind action type in the service yet.

Tests

New bundle/dms/bind (first bind creates the deployment, approval prompt for a config difference, job untouched until deploy, rebind and missing resource rejected) and bundle/dms/record-op-fault (an operation upload failure completes the version with failure; retry succeeds). bind-unbind-unsupported becomes unbind-unsupported. File-based bind goldens are unchanged.

This pull request and its description were written by Isaac.

With experimental.deployment_history, bind no longer refuses. It plans against a
local copy of the resources the service records, using the existing bind (same
prompt and resolved state), then registers the result: a new version with one
OPERATION_ACTION_TYPE_BIND operation carrying the resource's state, completed as
success or failure. A first bind creates the deployment. Like a file-based bind,
the workspace resource is not changed; the next deploy applies config
differences.

Unbind with deployment history stays unsupported: the service has no unbind
action type yet.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

Commit: c2fd55596 | Workflow run

Regression test report

Selected 3 changed tests (limit=20, 0 not selected)

3 tests: regression=2 unreleased=0 coverage=1 failing=0

test branch main (b158966) latest (v1.21.0)
TestAccept/bundle/dms/bind/DMS=true ✅ ❌ ❌
TestAccept/bundle/dms/record-op-fault/DMS=true ✅ ❌ ❌
TestAccept/bundle/dms/unbind-unsupported/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms/bind/DMS=true: branch ✅ | main (b158966) ❌ | latest (v1.21.0) ❌

main (b158966):

server.go:281: No handler for URL: /api/2.0/bundle/deployments/versions?version_id=1
        Body: [66 bytes] {"cli_version":"1.22.0-dev","version_type":"VERSION_TYPE_DESTROY"}
        
        For acceptance tests, add this to test.toml:
        [[Server]]
        Pattern = "POST /api/2.0/bundle/deployments/versions"
        Response.Body = '<response body here>'
        # Response.StatusCode = <response code if not 200>
    acceptance_test.go:1307: Diff:
        --- bundle/dms/bind/output.txt
        +++ /tmp/TestAcceptbundledmsbindDMS=true3322838588/002/output.txt
        @@ -10,111 +10,19 @@
         
         === Bind sc, the first bind: it creates the deployment and registers the schema in version 1
         >>> [CLI] bundle deployment bind sc main.dms_bind_[UNIQUE_NAME]
        -Successfully bound schema with an id 'main.dms_bind_[UNIQUE_NAME]'
        -Run 'bundle deploy' to deploy changes to your workspace
        +Error: bind is not supported for a bundle target that records deployment history
         
        ->>> print_requests.py --dms --method POST //versions --nostamp
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
        -  "q": {
        -    "version_id": "1"
        -  },
        -  "body": {
        -    "cli_version": "[CLI_VERSION]",
        -    "operations": [
        -      {
        -        "action_type": "OPERATION_ACTION_TYPE_BIND",
        -        "resource_key": "schemas.sc"
        -      }
        -    ],
        -    "version_type": "VERSION_TYPE_DEPLOY"
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/1/complete",
        -  "body": {
        -    "completion_reason": "VERSION_COMPLETE_SUCCESS"
        -  }
        -}
         
        -=== foo's config differs from the existing job, so binding it asks for approval
        -=== Bind foo: version 2 registers only foo, and the job is left unchanged
        ->>> [CLI] bundle deployment bind foo [FOO_ID] --auto-approve
        -Successfully bound job with an id '[FOO_ID]'
        -Run 'bundle deploy' to deploy changes to your workspace
        -
        ->>> print_requests.py --dms --method POST //versions --nostamp
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
        -  "q": {
        -    "version_id": "2"
        -  },
        -  "body": {
        -    "cli_version": "[CLI_VERSION]",
        -    "operations": [
        -      {
        -        "action_type": "OPERATION_ACTION_TYPE_BIND",
        -        "resource_key": "jobs.foo"
        -      }
        -    ],
        -    "previous_version_id": "1",
        -    "version_type": "VERSION_TYPE_DEPLOY"
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete",
        -  "body": {
        -    "completion_reason": "VERSION_COMPLETE_SUCCESS"
        -  }
        -}
        -
        ->>> [CLI] jobs get [FOO_ID]
        -{
        -  "name": "bind-target-[UNIQUE_NAME]"
        -}
        -
        -=== An already-bound resource cannot be rebound
        ->>> musterr [CLI] bundle deployment bind foo [FOO_ID] --auto-approve
        -Error: Resource already managed
        -
        -The bundle is already managing resource for resources.jobs.foo and it is bound to the requested ID [FOO_ID].
        -
        -
        -=== The next deploy applies foo's config and permissions, and creates bar
        ->>> [CLI] bundle plan
        -create jobs.bar
        -update jobs.foo
        -create jobs.foo.permissions
        -
        -Plan: 2 to add, 1 to change, 0 to delete, 1 unchanged
        -
        ->>> [CLI] bundle deploy --auto-approve
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-bind-[UNIQUE_NAME]/default/files...
        -Created jobs.bar
        -Created jobs.foo.permissions
        -Updated jobs.foo
        -Files: 4 uploaded, 0 deleted
        -Resources: 2 created, 1 changed, 0 deleted, 1 unchanged
        -
        ->>> [CLI] jobs get [FOO_ID]
        -{
        -  "name": "foo"
        -}
        -
         === Test cleanup: the bound resources are now bundle-managed, so destroy removes them
         >>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.jobs.bar
        -  delete resources.jobs.foo
        -  delete resources.schemas.sc
        +All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-bind-[UNIQUE_NAME]/default
         
        -This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
        -  delete resources.schemas.sc
        +Error: failed to create deployment version: No stub found for pattern: POST /api/2.0/bundle/deployments/versions (501 <nil>)
         
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-bind-[UNIQUE_NAME]/default
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/bundle/deployments/versions?version_id=1
        +HTTP Status: 501 Not Implemented
        +API error_code: <nil>
        +API message: No stub found for pattern: POST /api/2.0/bundle/deployments/versions
         
        -Destroy: 3 deleted
        +
        +Exit code: 1

latest (v1.21.0):

server.go:281: No handler for URL: /api/2.0/bundle/deployments/versions?version_id=1
        Body: [62 bytes] {"cli_version":"1.21.0","version_type":"VERSION_TYPE_DESTROY"}
        
        For acceptance tests, add this to test.toml:
        [[Server]]
        Pattern = "POST /api/2.0/bundle/deployments/versions"
        Response.Body = '<response body here>'
        # Response.StatusCode = <response code if not 200>
    acceptance_test.go:1307: Diff:
        --- bundle/dms/bind/output.txt
        +++ /tmp/TestAcceptbundledmsbindDMS=true3433778567/002/output.txt
        @@ -10,111 +10,19 @@
         
         === Bind sc, the first bind: it creates the deployment and registers the schema in version 1
         >>> [CLI] bundle deployment bind sc main.dms_bind_[UNIQUE_NAME]
        -Successfully bound schema with an id 'main.dms_bind_[UNIQUE_NAME]'
        -Run 'bundle deploy' to deploy changes to your workspace
        +Error: bind is not supported for a bundle target that records deployment history
         
        ->>> print_requests.py --dms --method POST //versions --nostamp
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
        -  "q": {
        -    "version_id": "1"
        -  },
        -  "body": {
        -    "cli_version": "[CLI_VERSION]",
        -    "operations": [
        -      {
        -        "action_type": "OPERATION_ACTION_TYPE_BIND",
        -        "resource_key": "schemas.sc"
        -      }
        -    ],
        -    "version_type": "VERSION_TYPE_DEPLOY"
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/1/complete",
        -  "body": {
        -    "completion_reason": "VERSION_COMPLETE_SUCCESS"
        -  }
        -}
         
        -=== foo's config differs from the existing job, so binding it asks for approval
        -=== Bind foo: version 2 registers only foo, and the job is left unchanged
        ->>> [CLI] bundle deployment bind foo [FOO_ID] --auto-approve
        -Successfully bound job with an id '[FOO_ID]'
        -Run 'bundle deploy' to deploy changes to your workspace
        -
        ->>> print_requests.py --dms --method POST //versions --nostamp
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
        -  "q": {
        -    "version_id": "2"
        -  },
        -  "body": {
        -    "cli_version": "[CLI_VERSION]",
        -    "operations": [
        -      {
        -        "action_type": "OPERATION_ACTION_TYPE_BIND",
        -        "resource_key": "jobs.foo"
        -      }
        -    ],
        -    "previous_version_id": "1",
        -    "version_type": "VERSION_TYPE_DEPLOY"
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete",
        -  "body": {
        -    "completion_reason": "VERSION_COMPLETE_SUCCESS"
        -  }
        -}
        -
        ->>> [CLI] jobs get [FOO_ID]
        -{
        -  "name": "bind-target-[UNIQUE_NAME]"
        -}
        -
        -=== An already-bound resource cannot be rebound
        ->>> musterr [CLI] bundle deployment bind foo [FOO_ID] --auto-approve
        -Error: Resource already managed
        -
        -The bundle is already managing resource for resources.jobs.foo and it is bound to the requested ID [FOO_ID].
        -
        -
        -=== The next deploy applies foo's config and permissions, and creates bar
        ->>> [CLI] bundle plan
        -create jobs.bar
        -update jobs.foo
        -create jobs.foo.permissions
        -
        -Plan: 2 to add, 1 to change, 0 to delete, 1 unchanged
        -
        ->>> [CLI] bundle deploy --auto-approve
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-bind-[UNIQUE_NAME]/default/files...
        -Created jobs.bar
        -Created jobs.foo.permissions
        -Updated jobs.foo
        -Files: 4 uploaded, 0 deleted
        -Resources: 2 created, 1 changed, 0 deleted, 1 unchanged
        -
        ->>> [CLI] jobs get [FOO_ID]
        -{
        -  "name": "foo"
        -}
        -
         === Test cleanup: the bound resources are now bundle-managed, so destroy removes them
         >>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.jobs.bar
        -  delete resources.jobs.foo
        -  delete resources.schemas.sc
        +All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-bind-[UNIQUE_NAME]/default
         
        -This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
        -  delete resources.schemas.sc
        +Error: failed to create deployment version: No stub found for pattern: POST /api/2.0/bundle/deployments/versions (501 <nil>)
         
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-bind-[UNIQUE_NAME]/default
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/bundle/deployments/versions?version_id=1
        +HTTP Status: 501 Not Implemented
        +API error_code: <nil>
        +API message: No stub found for pattern: POST /api/2.0/bundle/deployments/versions
         
        -Destroy: 3 deleted
        +
        +Exit code: 1
TestAccept/bundle/dms/record-op-fault/DMS=true: branch ✅ | main (b158966) ❌ | latest (v1.21.0) ❌

main (b158966):

acceptance_test.go:1307: Diff:
        --- bundle/dms/record-op-fault/output.txt
        +++ /tmp/TestAcceptbundledmsrecord-op-faultDMS=true4216638942/002/output.txt
        @@ -3,69 +3,15 @@
         >>> fault.py PATCH /api/2.0/bundle/* 500 0 1
         
         >>> musterr [CLI] bundle deployment bind foo [JOB_ID] --auto-approve
        -Error: recording operation for resources.jobs.foo: Fault injected by test. (500 INJECTED)
        +Error: bind is not supported for a bundle target that records deployment history
         
        -Endpoint: PATCH [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/1/operations/jobs.foo?update_mask=state%2Cerror_message%2Cresource_id%2Cstatus
        -HTTP Status: 500 Internal Server Error
        -API error_code: INJECTED
        -API message: Fault injected by test.
         
        -
         === The version is still completed - with failure - rather than left holding its lease
         >>> print_requests.py --dms --method POST //versions
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
        -  "q": {
        -    "version_id": "1"
        -  },
        -  "body": {
        -    "cli_version": "[CLI_VERSION]",
        -    "operations": [
        -      {
        -        "action_type": "OPERATION_ACTION_TYPE_BIND",
        -        "resource_key": "jobs.foo"
        -      }
        -    ],
        -    "version_type": "VERSION_TYPE_DEPLOY"
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/1/complete",
        -  "body": {
        -    "completion_reason": "VERSION_COMPLETE_FAILURE"
        -  }
        -}
         
         === Retrying the bind succeeds
         >>> [CLI] bundle deployment bind foo [JOB_ID] --auto-approve
        -Successfully bound job with an id '[JOB_ID]'
        -Run 'bundle deploy' to deploy changes to your workspace
        +Error: bind is not supported for a bundle target that records deployment history
         
        ->>> print_requests.py --dms --method POST //versions
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
        -  "q": {
        -    "version_id": "2"
        -  },
        -  "body": {
        -    "cli_version": "[CLI_VERSION]",
        -    "operations": [
        -      {
        -        "action_type": "OPERATION_ACTION_TYPE_BIND",
        -        "resource_key": "jobs.foo"
        -      }
        -    ],
        -    "previous_version_id": "1",
        -    "version_type": "VERSION_TYPE_DEPLOY"
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete",
        -  "body": {
        -    "completion_reason": "VERSION_COMPLETE_SUCCESS"
        -  }
        -}
        +
        +Exit code: 1
        
FAIL

latest (v1.21.0):

acceptance_test.go:1307: Diff:
        --- bundle/dms/record-op-fault/output.txt
        +++ /tmp/TestAcceptbundledmsrecord-op-faultDMS=true3734687416/002/output.txt
        @@ -3,69 +3,15 @@
         >>> fault.py PATCH /api/2.0/bundle/* 500 0 1
         
         >>> musterr [CLI] bundle deployment bind foo [JOB_ID] --auto-approve
        -Error: recording operation for resources.jobs.foo: Fault injected by test. (500 INJECTED)
        +Error: bind is not supported for a bundle target that records deployment history
         
        -Endpoint: PATCH [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/1/operations/jobs.foo?update_mask=state%2Cerror_message%2Cresource_id%2Cstatus
        -HTTP Status: 500 Internal Server Error
        -API error_code: INJECTED
        -API message: Fault injected by test.
         
        -
         === The version is still completed - with failure - rather than left holding its lease
         >>> print_requests.py --dms --method POST //versions
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
        -  "q": {
        -    "version_id": "1"
        -  },
        -  "body": {
        -    "cli_version": "[CLI_VERSION]",
        -    "operations": [
        -      {
        -        "action_type": "OPERATION_ACTION_TYPE_BIND",
        -        "resource_key": "jobs.foo"
        -      }
        -    ],
        -    "version_type": "VERSION_TYPE_DEPLOY"
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/1/complete",
        -  "body": {
        -    "completion_reason": "VERSION_COMPLETE_FAILURE"
        -  }
        -}
         
         === Retrying the bind succeeds
         >>> [CLI] bundle deployment bind foo [JOB_ID] --auto-approve
        -Successfully bound job with an id '[JOB_ID]'
        -Run 'bundle deploy' to deploy changes to your workspace
        +Error: bind is not supported for a bundle target that records deployment history
         
        ->>> print_requests.py --dms --method POST //versions
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
        -  "q": {
        -    "version_id": "2"
        -  },
        -  "body": {
        -    "cli_version": "[CLI_VERSION]",
        -    "operations": [
        -      {
        -        "action_type": "OPERATION_ACTION_TYPE_BIND",
        -        "resource_key": "jobs.foo"
        -      }
        -    ],
        -    "previous_version_id": "1",
        -    "version_type": "VERSION_TYPE_DEPLOY"
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete",
        -  "body": {
        -    "completion_reason": "VERSION_COMPLETE_SUCCESS"
        -  }
        -}
        +
        +Exit code: 1
        
FAIL

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: c2fd555

Run: 39431117897

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 282 17 4:38
✅​ aws-windows-latest-4core-16gb 284 15 4:15
✅​ azure linux-2core-8gb 281 17 4:28
✅​ azure-windows-latest-4core-16gb 283 15 6:42
✅​ gcp linux-2core-8gb 282 17 4:31
✅​ gcp-windows-latest-4core-16gb 284 15 5:24
Top 3 slowest tests (at least 2 minutes):
duration env testname
6:00 azure-windows-latest-4core-16gb TestAccept
4:51 gcp-windows-latest-4core-16gb TestAccept
3:43 aws-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants