diff --git a/acceptance/bundle/dms_fault/README.md b/acceptance/bundle/dms_fault/README.md new file mode 100644 index 00000000000..76ddea99005 --- /dev/null +++ b/acceptance/bundle/dms_fault/README.md @@ -0,0 +1,54 @@ +Each API case runs the same linear workflow with a job and a pipeline. DMS records +deployment history while files remain the source of truth. All bundle commands +must succeed with the selected API configured to return HTTP 500. + +| Step | Command or action | Assertion | +| --- | --- | --- | +| 1 | Deploy with DMS healthy. | Establish an existing deployment and its DMS deployment ID. | +| 2 | Enable the API fault, rename the bundle and resources with the workspace root pinned, then deploy. | Both resources exist with their updated names. State retains file-backed history recording. The bundle rename exercises `UpdateDeployment`. | +| 3 | Run `bundle summary`. | Output contains both resource IDs. | +| 4 | Run `bundle generate` into the export directory. | The resource YAML exists; pipeline SQL is downloaded with its original content. | +| 5 | Run `bundle deployment unbind`. | The resource ID is removed from bundle state. | +| 6 | Run `bundle generate --bind` into the bundle's resource directory. | Generation and binding succeed for the original resource ID, proving unbind preserved the remote resource. Pipeline SQL is downloaded with its original content. | +| 7 | Run `bundle validate`, then deploy again. | Generated configuration is valid, both resource IDs stay unchanged, and the pipeline references the newly downloaded SQL path. | +| 8 | Disable history and deploy, then enable it and deploy. | Both IDs stay unchanged. The file-backed history marker follows the setting, and disabling history makes no DMS calls. Each toggle changes the job description to exercise an actual resource update. | +| 9 | Run `bundle destroy` with history enabled. | Both remote resources and remote state are gone. | +| 10 | Disable history, deploy fresh resources, then destroy. | The resources exist after deploy and are gone after destroy. Neither command calls DMS. | + +Steps 4–6 run first for the job, then for the pipeline. The fault stays enabled +through step 10 and is cleared in `finally`. For `CreateDeployment`, skip the healthy +setup deployment so the fault applies to the first deployment. + +History is toggled with `experimental.deployment_history: true/false` in YAML. +That explicit setting takes priority over the suite's environment override. + +Each case registers one fault rule. A version wildcard such as +`POST .../versions/*/complete` covers every version the workflow accesses or creates. +Each `*` stays within one path segment, so a version read fault cannot also match +nested operation endpoints. +`GetDeployment` also uses a deployment-ID wildcard so its fault stays active if a +failed lookup triggers a replacement deployment record. + +Each snapshot records the command invocations and output, including warnings and +deployment-version links, then reports whether the injected endpoint was called. +An unused endpoint records `DMS API called: no`. + +Fault cases cover all DMS APIs: + +- `CompleteVersion` +- `CreateDeployment` +- `CreateVersion` +- `DeleteDeployment` +- `GetDeployment` +- `GetOperation` +- `GetResource` +- `GetVersion` +- `Heartbeat` +- `ListDeployments` +- `ListOperations` +- `ListResources` +- `ListVersions` +- `UpdateDeployment` +- `UpdateOperation` + +Missing APIs: none. diff --git a/acceptance/bundle/dms_fault/complete_version/out.test.toml b/acceptance/bundle/dms_fault/complete_version/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/complete_version/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/complete_version/output.txt b/acceptance/bundle/dms_fault/complete_version/output.txt new file mode 100644 index 00000000000..c7d574f4431 --- /dev/null +++ b/acceptance/bundle/dms_fault/complete_version/output.txt @@ -0,0 +1,148 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: complete_version fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + Warn: Failed to complete deployment history version: Fault injected by test. + +>>> bundle summary + Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + Warn: Failed to complete deployment history version: Fault injected by test. + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + Warn: Failed to complete deployment history version: Fault injected by test. + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME] + + Warn: Failed to complete deployment history version: Fault injected by test. + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: yes diff --git a/acceptance/bundle/dms_fault/complete_version/script b/acceptance/bundle/dms_fault/complete_version/script new file mode 100644 index 00000000000..57bef3e61e8 --- /dev/null +++ b/acceptance/bundle/dms_fault/complete_version/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" complete_version diff --git a/acceptance/bundle/dms_fault/create_deployment/out.test.toml b/acceptance/bundle/dms_fault/create_deployment/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/create_deployment/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/create_deployment/output.txt b/acceptance/bundle/dms_fault/create_deployment/output.txt new file mode 100644 index 00000000000..2b1b3f5a082 --- /dev/null +++ b/acceptance/bundle/dms_fault/create_deployment/output.txt @@ -0,0 +1,135 @@ + +=== commands: setup + +=== commands: create_deployment fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files... + Warn: Could not enable deployment history for this deploy: failed to create deployment: Fault injected by test. + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files... + Warn: Could not enable deployment history for this deploy: failed to create deployment: Fault injected by test. + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Files: 2 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files... + Warn: Could not enable deployment history for this deploy: failed to create deployment: Fault injected by test. + Updated jobs.foo + Files: 2 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME] + + Warn: Could not record deployment history for this destroy: failed to create deployment: Fault injected by test. + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: yes diff --git a/acceptance/bundle/dms_fault/create_deployment/script b/acceptance/bundle/dms_fault/create_deployment/script new file mode 100644 index 00000000000..c8f37780f9e --- /dev/null +++ b/acceptance/bundle/dms_fault/create_deployment/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" create_deployment diff --git a/acceptance/bundle/dms_fault/create_version/out.test.toml b/acceptance/bundle/dms_fault/create_version/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/create_version/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/create_version/output.txt b/acceptance/bundle/dms_fault/create_version/output.txt new file mode 100644 index 00000000000..f58fd56b675 --- /dev/null +++ b/acceptance/bundle/dms_fault/create_version/output.txt @@ -0,0 +1,145 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: create_version fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files... + Warn: Could not enable deployment history for this deploy: failed to create deployment version: Fault injected by test. + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-create_version-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-create_version-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-create_version-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-create_version-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files... + Warn: Could not enable deployment history for this deploy: failed to create deployment version: Fault injected by test. + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files... + Warn: Could not enable deployment history for this deploy: failed to create deployment version: Fault injected by test. + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME] + + Warn: Could not record deployment history for this destroy: failed to create deployment version: Fault injected by test. + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: yes diff --git a/acceptance/bundle/dms_fault/create_version/script b/acceptance/bundle/dms_fault/create_version/script new file mode 100644 index 00000000000..3261ce61321 --- /dev/null +++ b/acceptance/bundle/dms_fault/create_version/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" create_version diff --git a/acceptance/bundle/dms_fault/databricks.yml.tmpl b/acceptance/bundle/dms_fault/databricks.yml.tmpl new file mode 100644 index 00000000000..74e22923ff1 --- /dev/null +++ b/acceptance/bundle/dms_fault/databricks.yml.tmpl @@ -0,0 +1,11 @@ +bundle: + name: $RESOURCE_NAME + +workspace: + root_path: $ROOT_PATH + +experimental: + deployment_history: true + +include: + - resources/*.yml diff --git a/acceptance/bundle/dms_fault/delete_deployment/out.test.toml b/acceptance/bundle/dms_fault/delete_deployment/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/delete_deployment/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/delete_deployment/output.txt b/acceptance/bundle/dms_fault/delete_deployment/output.txt new file mode 100644 index 00000000000..8efdbc07a0d --- /dev/null +++ b/acceptance/bundle/dms_fault/delete_deployment/output.txt @@ -0,0 +1,147 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: delete_deployment fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-delete_deployment-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-delete_deployment-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-delete_deployment-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-delete_deployment-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME] + + Warn: retrying after 500 Internal Server Error from DELETE /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: retrying after 500 Internal Server Error from DELETE /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: Could not delete deployment history: Fault injected by test. + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: yes diff --git a/acceptance/bundle/dms_fault/delete_deployment/script b/acceptance/bundle/dms_fault/delete_deployment/script new file mode 100644 index 00000000000..8d34a920f4f --- /dev/null +++ b/acceptance/bundle/dms_fault/delete_deployment/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" delete_deployment diff --git a/acceptance/bundle/dms_fault/get_deployment/out.test.toml b/acceptance/bundle/dms_fault/get_deployment/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/get_deployment/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/get_deployment/output.txt b/acceptance/bundle/dms_fault/get_deployment/output.txt new file mode 100644 index 00000000000..0dda68edc1e --- /dev/null +++ b/acceptance/bundle/dms_fault/get_deployment/output.txt @@ -0,0 +1,159 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: get_deployment fault enabled + +>>> bundle deploy --auto-approve + Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: Could not read deployment history: Fault injected by test. + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: Could not read deployment history: Fault injected by test. + Name: updated-dms-fault-get_deployment-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-get_deployment-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-get_deployment-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-get_deployment-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: Could not read deployment history: Fault injected by test. + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated jobs.foo + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: Could not read deployment history: Fault injected by test. + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Files: 2 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +>>> bundle destroy --auto-approve + Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: Could not read deployment history: Fault injected by test. + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: yes diff --git a/acceptance/bundle/dms_fault/get_deployment/script b/acceptance/bundle/dms_fault/get_deployment/script new file mode 100644 index 00000000000..25f1c6fe032 --- /dev/null +++ b/acceptance/bundle/dms_fault/get_deployment/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" get_deployment diff --git a/acceptance/bundle/dms_fault/get_operation/out.test.toml b/acceptance/bundle/dms_fault/get_operation/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/get_operation/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/get_operation/output.txt b/acceptance/bundle/dms_fault/get_operation/output.txt new file mode 100644 index 00000000000..65498809a65 --- /dev/null +++ b/acceptance/bundle/dms_fault/get_operation/output.txt @@ -0,0 +1,144 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_operation-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: get_operation fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_operation-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-get_operation-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_operation-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-get_operation-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-get_operation-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-get_operation-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_operation-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_operation-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_operation-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_operation-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_operation-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_operation-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_operation-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: no diff --git a/acceptance/bundle/dms_fault/get_operation/script b/acceptance/bundle/dms_fault/get_operation/script new file mode 100644 index 00000000000..ed38a0ce0cd --- /dev/null +++ b/acceptance/bundle/dms_fault/get_operation/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" get_operation diff --git a/acceptance/bundle/dms_fault/get_resource/out.test.toml b/acceptance/bundle/dms_fault/get_resource/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/get_resource/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/get_resource/output.txt b/acceptance/bundle/dms_fault/get_resource/output.txt new file mode 100644 index 00000000000..a562c0f5264 --- /dev/null +++ b/acceptance/bundle/dms_fault/get_resource/output.txt @@ -0,0 +1,144 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_resource-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: get_resource fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_resource-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-get_resource-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_resource-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-get_resource-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-get_resource-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-get_resource-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_resource-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_resource-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_resource-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_resource-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_resource-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_resource-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_resource-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: no diff --git a/acceptance/bundle/dms_fault/get_resource/script b/acceptance/bundle/dms_fault/get_resource/script new file mode 100644 index 00000000000..8f4f5e8b9b5 --- /dev/null +++ b/acceptance/bundle/dms_fault/get_resource/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" get_resource diff --git a/acceptance/bundle/dms_fault/get_version/out.test.toml b/acceptance/bundle/dms_fault/get_version/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/get_version/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/get_version/output.txt b/acceptance/bundle/dms_fault/get_version/output.txt new file mode 100644 index 00000000000..b4473435fe9 --- /dev/null +++ b/acceptance/bundle/dms_fault/get_version/output.txt @@ -0,0 +1,144 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_version-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: get_version fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_version-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-get_version-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_version-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-get_version-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-get_version-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-get_version-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_version-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_version-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_version-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_version-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_version-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_version-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_version-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: no diff --git a/acceptance/bundle/dms_fault/get_version/script b/acceptance/bundle/dms_fault/get_version/script new file mode 100644 index 00000000000..30468ab5e64 --- /dev/null +++ b/acceptance/bundle/dms_fault/get_version/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" get_version diff --git a/acceptance/bundle/dms_fault/heartbeat/out.test.toml b/acceptance/bundle/dms_fault/heartbeat/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/heartbeat/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/heartbeat/output.txt b/acceptance/bundle/dms_fault/heartbeat/output.txt new file mode 100644 index 00000000000..b512e794a75 --- /dev/null +++ b/acceptance/bundle/dms_fault/heartbeat/output.txt @@ -0,0 +1,144 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-heartbeat-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: heartbeat fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-heartbeat-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-heartbeat-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-heartbeat-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-heartbeat-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-heartbeat-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-heartbeat-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-heartbeat-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-heartbeat-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-heartbeat-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-heartbeat-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-heartbeat-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-heartbeat-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-heartbeat-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: no diff --git a/acceptance/bundle/dms_fault/heartbeat/script b/acceptance/bundle/dms_fault/heartbeat/script new file mode 100644 index 00000000000..5fe90e5f2c9 --- /dev/null +++ b/acceptance/bundle/dms_fault/heartbeat/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" heartbeat diff --git a/acceptance/bundle/dms_fault/job.yml.tmpl b/acceptance/bundle/dms_fault/job.yml.tmpl new file mode 100644 index 00000000000..930af269d00 --- /dev/null +++ b/acceptance/bundle/dms_fault/job.yml.tmpl @@ -0,0 +1,5 @@ +resources: + jobs: + foo: + name: $RESOURCE_NAME + description: history-true diff --git a/acceptance/bundle/dms_fault/list_deployments/out.test.toml b/acceptance/bundle/dms_fault/list_deployments/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/list_deployments/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/list_deployments/output.txt b/acceptance/bundle/dms_fault/list_deployments/output.txt new file mode 100644 index 00000000000..0710bf48939 --- /dev/null +++ b/acceptance/bundle/dms_fault/list_deployments/output.txt @@ -0,0 +1,144 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_deployments-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: list_deployments fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_deployments-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-list_deployments-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_deployments-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-list_deployments-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-list_deployments-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-list_deployments-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_deployments-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_deployments-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_deployments-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_deployments-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_deployments-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_deployments-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_deployments-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: no diff --git a/acceptance/bundle/dms_fault/list_deployments/script b/acceptance/bundle/dms_fault/list_deployments/script new file mode 100644 index 00000000000..49edc815cf9 --- /dev/null +++ b/acceptance/bundle/dms_fault/list_deployments/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" list_deployments diff --git a/acceptance/bundle/dms_fault/list_operations/out.test.toml b/acceptance/bundle/dms_fault/list_operations/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/list_operations/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/list_operations/output.txt b/acceptance/bundle/dms_fault/list_operations/output.txt new file mode 100644 index 00000000000..5764b81d690 --- /dev/null +++ b/acceptance/bundle/dms_fault/list_operations/output.txt @@ -0,0 +1,144 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_operations-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: list_operations fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_operations-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-list_operations-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_operations-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-list_operations-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-list_operations-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-list_operations-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_operations-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_operations-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_operations-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_operations-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_operations-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_operations-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_operations-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: no diff --git a/acceptance/bundle/dms_fault/list_operations/script b/acceptance/bundle/dms_fault/list_operations/script new file mode 100644 index 00000000000..3c63f9567c2 --- /dev/null +++ b/acceptance/bundle/dms_fault/list_operations/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" list_operations diff --git a/acceptance/bundle/dms_fault/list_resources/out.test.toml b/acceptance/bundle/dms_fault/list_resources/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/list_resources/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/list_resources/output.txt b/acceptance/bundle/dms_fault/list_resources/output.txt new file mode 100644 index 00000000000..9f45951b03a --- /dev/null +++ b/acceptance/bundle/dms_fault/list_resources/output.txt @@ -0,0 +1,144 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_resources-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: list_resources fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_resources-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-list_resources-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_resources-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-list_resources-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-list_resources-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-list_resources-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_resources-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_resources-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_resources-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_resources-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_resources-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_resources-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_resources-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: no diff --git a/acceptance/bundle/dms_fault/list_resources/script b/acceptance/bundle/dms_fault/list_resources/script new file mode 100644 index 00000000000..1c3c3bffa5d --- /dev/null +++ b/acceptance/bundle/dms_fault/list_resources/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" list_resources diff --git a/acceptance/bundle/dms_fault/list_versions/out.test.toml b/acceptance/bundle/dms_fault/list_versions/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/list_versions/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/list_versions/output.txt b/acceptance/bundle/dms_fault/list_versions/output.txt new file mode 100644 index 00000000000..dabee52cbcd --- /dev/null +++ b/acceptance/bundle/dms_fault/list_versions/output.txt @@ -0,0 +1,144 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_versions-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: list_versions fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_versions-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-list_versions-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_versions-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-list_versions-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-list_versions-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-list_versions-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_versions-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_versions-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_versions-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_versions-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_versions-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_versions-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-list_versions-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: no diff --git a/acceptance/bundle/dms_fault/list_versions/script b/acceptance/bundle/dms_fault/list_versions/script new file mode 100644 index 00000000000..7582b7fb740 --- /dev/null +++ b/acceptance/bundle/dms_fault/list_versions/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" list_versions diff --git a/acceptance/bundle/dms_fault/pipeline.sql.tmpl b/acceptance/bundle/dms_fault/pipeline.sql.tmpl new file mode 100644 index 00000000000..2d54c9c6e56 --- /dev/null +++ b/acceptance/bundle/dms_fault/pipeline.sql.tmpl @@ -0,0 +1 @@ +CREATE OR REFRESH MATERIALIZED VIEW $DATASET_NAME AS SELECT 1 AS value; diff --git a/acceptance/bundle/dms_fault/pipeline.yml.tmpl b/acceptance/bundle/dms_fault/pipeline.yml.tmpl new file mode 100644 index 00000000000..a4752c44730 --- /dev/null +++ b/acceptance/bundle/dms_fault/pipeline.yml.tmpl @@ -0,0 +1,10 @@ +resources: + pipelines: + bar: + name: $RESOURCE_NAME + catalog: main + schema: default + serverless: true + libraries: + - file: + path: ../pipeline.sql diff --git a/acceptance/bundle/dms_fault/run.py b/acceptance/bundle/dms_fault/run.py new file mode 100644 index 00000000000..11e22795ccc --- /dev/null +++ b/acceptance/bundle/dms_fault/run.py @@ -0,0 +1,249 @@ +#!/usr/bin/env python3 +"""Exercise file-backed bundle commands while one DMS API persistently fails.""" + +import json +import os +import re +import shlex +import subprocess +import sys +from pathlib import Path +from string import Template + + +# Keep acceptance-only setup out of imports used by doctest. +def main(): + # Use file paths: Windows does not reliably find Python scripts on PATH. + HELPERS = Path(__file__).resolve().parents[2] / "bin" + sys.path.insert(0, str(HELPERS)) + from add_repl import add_repl + from print_requests import read_json_many + + API = sys.argv[1] + FIXTURES = Path(__file__).parent + CLI = os.environ["CLI"] + REQUESTS = Path(os.environ["OUT_REQUESTS"]) + LOG = Path("LOG.commands").resolve() + PATTERNS = { + "complete_version": "POST /api/2.0/bundle/deployments/{id}/versions/*/complete", + "create_deployment": "POST /api/2.0/bundle/deployments", + "create_version": "POST /api/2.0/bundle/deployments/{id}/versions", + "delete_deployment": "DELETE /api/2.0/bundle/deployments/{id}", + # Keep failing lookups even if the CLI registers a replacement deployment. + "get_deployment": "GET /api/2.0/bundle/deployments/*", + "get_operation": "GET /api/2.0/bundle/deployments/{id}/versions/*/operations/*", + "get_resource": "GET /api/2.0/bundle/deployments/{id}/resources/*", + "get_version": "GET /api/2.0/bundle/deployments/{id}/versions/*", + "heartbeat": "POST /api/2.0/bundle/deployments/{id}/versions/*/heartbeat", + "list_deployments": "GET /api/2.0/bundle/deployments", + "list_operations": "GET /api/2.0/bundle/deployments/{id}/versions/*/operations", + "list_resources": "GET /api/2.0/bundle/deployments/{id}/resources", + "list_versions": "GET /api/2.0/bundle/deployments/{id}/versions", + "update_deployment": "PATCH /api/2.0/bundle/deployments/{id}", + "update_operation": "PATCH /api/2.0/bundle/deployments/{id}/versions/*/operations/*", + } + KNOWN_IDS = set() + + # Run the CLI and check its exit code. + # Log all commands; show bundle commands in the snapshot. + def run(cwd, *args, expected=0): + visible = args[0] == "bundle" + result = subprocess.run( + [CLI, *args], + cwd=cwd, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT if visible else subprocess.PIPE, + ) + output = result.stdout + (result.stderr or "") + with LOG.open("a") as out: + out.write(f"\n{cwd.name}: {args}\nexit={result.returncode}\n{output}") + if visible: + print("\n>>> " + shlex.join(args), flush=True) + for line in output.splitlines(): + print(" " + line if line else "", flush=True) + assert result.returncode == expected, output + return result + + # Enable or clear an HTTP 500 fault for this endpoint. + def fault(pattern, enabled): + subprocess.run( + [sys.executable, str(HELPERS / "fault.py"), pattern, "500", "0", "1000" if enabled else "0"], check=True + ) + + # Count recorded method/path matches to report whether this API was called. + # Here * stays within one path segment; ** can span /. + def hits(pattern): + if not REQUESTS.exists(): + return 0 + regex = re.escape(pattern).replace(r"\*\*", ".*").replace(r"\*", "[^/]*") + return sum( + bool(re.fullmatch(regex, f"{r['method']} {r['path']}")) for r in read_json_many(REQUESTS.read_text()) + ) + + # Read the bundle's local deployment state. + def state(cwd): + return json.loads((cwd / ".databricks/bundle/default/resources.json").read_text()) + + # Check the history marker and file-backed state, then collect and label resource IDs. + def ids(cwd, history=True): + data = state(cwd) + features = data.get("features", {}) + assert ("deployment_history_shadow" in features) == history, data + assert "deployment_history" not in features, data + current = {key: value["__id__"] for key, value in data["state"].items()} + for key, value in current.items(): + if value not in KNOWN_IDS: + add_repl(value, key.rsplit(".", 1)[1].upper() + "_ID") + KNOWN_IDS.add(value) + return current + + # Write the test bundle files with unique resource names and a workspace path. + def setup(phase): + cwd = Path("work", phase).resolve() + (cwd / "resources").mkdir(parents=True) + name = f"dms-fault-{API}-{phase}-{os.environ['UNIQUE_NAME']}" + root = f"/Workspace/Users/{os.environ['CURRENT_USER_NAME']}/.bundle/{name}" + values = {"RESOURCE_NAME": name, "ROOT_PATH": root, "DATASET_NAME": name.replace("-", "_")} + for source, target in [ + ("databricks.yml.tmpl", "databricks.yml"), + ("job.yml.tmpl", "resources/foo.job.yml"), + ("pipeline.yml.tmpl", "resources/bar.pipeline.yml"), + ("pipeline.sql.tmpl", "pipeline.sql"), + ]: + (cwd / target).write_text(Template((FIXTURES / source).read_text()).substitute(values)) + return cwd, root + + # Prepare an existing deployment and enable its API fault. + # Skip the initial deploy for CreateDeployment so its first call fails. + def activate(cwd, root): + print(f"\n=== {cwd.name}: setup", flush=True) + deployment = None + if API != "create_deployment": + run(cwd, "bundle", "deploy", "--auto-approve") + node = json.loads( + run(cwd, "workspace", "get-status", root + "/state/resources.deployment.json", "-o", "json").stdout + ) + deployment = str(node["object_id"]) + pattern = PATTERNS[API].format(id=deployment) + fault(pattern, True) + print(f"\n=== {cwd.name}: {API} fault enabled", flush=True) + return pattern + + # Verify resource names and IDs, plus the job's updated description. + def assert_alive(cwd, current, history=True): + job = json.loads(run(cwd, "jobs", "get", current["resources.jobs.foo"], "-o", "json").stdout) + pipeline = json.loads(run(cwd, "pipelines", "get", current["resources.pipelines.bar"], "-o", "json").stdout) + assert str(job["job_id"]) == current["resources.jobs.foo"] + assert pipeline["pipeline_id"] == current["resources.pipelines.bar"] + assert job["settings"]["name"].startswith("updated-dms-fault-") + assert job["settings"]["description"] == f"history-{str(history).lower()}" + assert pipeline["spec"]["name"].startswith("updated-dms-fault-") + + # Export a resource, optionally adding it to the bundle and binding its original ID. + def generate(cwd, kind, key, identifier, bind=False): + args = ["bundle", "generate", kind, f"--existing-{kind}-id", identifier, "--key", key] + args += [ + "--config-dir", + "resources" if bind else "exported", + "--source-dir", + "src" if bind else "exported-src", + "--force", + ] + if bind: + args.append("--bind") + run(cwd, *args) + + # Toggle YAML history and change the job description to force a real update. + def set_history(cwd, enabled): + value = str(enabled).lower() + previous = str(not enabled).lower() + print(f"\n=== deployment_history: {value}", flush=True) + for filename, prefix in [ + ("databricks.yml", "deployment_history: "), + ("resources/foo.job.yml", "description: history-"), + ]: + subprocess.run( + [ + sys.executable, + str(HELPERS / "update_file.py"), + str(cwd / filename), + prefix + previous, + prefix + value, + ], + check=True, + ) + + # Verify the job, pipeline, and remote state were deleted. + def assert_destroyed(cwd, root, current): + job = run(cwd, "jobs", "get", current["resources.jobs.foo"], expected=1) + assert "does not exist" in job.stderr.lower(), job.stderr + pipeline = run(cwd, "pipelines", "get", current["resources.pipelines.bar"], expected=1) + assert "not found" in pipeline.stderr.lower(), pipeline.stderr + file = run(cwd, "workspace", "get-status", root + "/state/resources.json", expected=1) + assert "doesn't exist" in file.stderr.lower() or "does not exist" in file.stderr.lower(), file.stderr + + cwd, root = setup("commands") + pattern = activate(cwd, root) + try: + # Pinning the root lets a bundle-name change exercise UpdateDeployment in place. + config = cwd / "databricks.yml" + config.write_text(config.read_text().replace("name: dms-fault-", "name: updated-dms-fault-")) + for path in (cwd / "resources").glob("*.yml"): + path.write_text(path.read_text().replace("name: dms-fault-", "name: updated-dms-fault-")) + previous = hits(pattern) + run(cwd, "bundle", "deploy", "--auto-approve") + current = ids(cwd) + assert_alive(cwd, current) + summary = run(cwd, "bundle", "summary").stdout + assert current["resources.jobs.foo"] in summary + assert current["resources.pipelines.bar"] in summary + for kind, key in [("job", "foo"), ("pipeline", "bar")]: + group = "jobs" if kind == "job" else "pipelines" + resource = f"resources.{group}.{key}" + generate(cwd, kind, key, current[resource]) + assert (cwd / "exported" / f"{key}.{kind}.yml").is_file() + if kind == "pipeline": + assert (cwd / "exported-src/pipeline.sql").read_text() == (cwd / "pipeline.sql").read_text() + run(cwd, "bundle", "deployment", "unbind", key) + assert resource not in state(cwd)["state"] + generate(cwd, kind, key, current[resource], bind=True) + assert ids(cwd)[resource] == current[resource] + if kind == "pipeline": + assert (cwd / "src/pipeline.sql").read_text() == (cwd / "pipeline.sql").read_text() + run(cwd, "bundle", "validate") + run(cwd, "bundle", "deploy", "--auto-approve") + assert ids(cwd) == current + pipeline = json.loads(run(cwd, "pipelines", "get", current["resources.pipelines.bar"], "-o", "json").stdout) + assert pipeline["spec"]["libraries"][0]["file"]["path"] == root + "/files/src/pipeline.sql" + + # Toggle existing resources without replacing them. Disabled history must make no DMS calls. + all_dms = "* /api/2.0/bundle/**" + for enabled in [False, True]: + set_history(cwd, enabled) + before = hits(all_dms) + run(cwd, "bundle", "deploy", "--auto-approve") + assert ids(cwd, history=enabled) == current + assert_alive(cwd, current, history=enabled) + if not enabled: + assert hits(all_dms) == before, "History-disabled deploy called DMS" + + run(cwd, "bundle", "destroy", "--auto-approve") + assert_destroyed(cwd, root, current) + + # A fresh deployment and its destroy must also work with history disabled. + set_history(cwd, False) + before = hits(all_dms) + run(cwd, "bundle", "deploy", "--auto-approve") + current = ids(cwd, history=False) + assert_alive(cwd, current, history=False) + run(cwd, "bundle", "destroy", "--auto-approve") + assert_destroyed(cwd, root, current) + assert hits(all_dms) == before, "History-disabled deploy/destroy called DMS" + print("\nDMS API called: " + ("yes" if hits(pattern) > previous else "no"), flush=True) + finally: + fault(pattern, False) + + +if __name__ == "__main__": + main() diff --git a/acceptance/bundle/dms_fault/test.toml b/acceptance/bundle/dms_fault/test.toml new file mode 100644 index 00000000000..0a01bdac577 --- /dev/null +++ b/acceptance/bundle/dms_fault/test.toml @@ -0,0 +1,11 @@ +Cloud = false +RecordRequests = true +Timeout = '120s' +EnvMatrix.DMS = ['true'] +Env.DATABRICKS_BUNDLE_RETRY_INTERVAL_MS = '0' +Ignore = ['work'] +SortLines.deploy_actions = '^ (Created|Updated|Deleted|Recreated|Resized) \S+$' + +[[Repls]] +Old = 'recording operation for resources\.[^:]+:' +New = 'recording operation for [RESOURCE_KEY]:' diff --git a/acceptance/bundle/dms_fault/update_deployment/out.test.toml b/acceptance/bundle/dms_fault/update_deployment/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/update_deployment/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/update_deployment/output.txt b/acceptance/bundle/dms_fault/update_deployment/output.txt new file mode 100644 index 00000000000..09b82f8c549 --- /dev/null +++ b/acceptance/bundle/dms_fault/update_deployment/output.txt @@ -0,0 +1,150 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_deployment-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: update_deployment fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_deployment-commands-[UNIQUE_NAME]/files... + Warn: retrying after 500 Internal Server Error from PATCH /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: retrying after 500 Internal Server Error from PATCH /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: Could not enable deployment history for this deploy: failed to update deployment: Fault injected by test. + Updated jobs.foo + Updated pipelines.bar + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-update_deployment-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_deployment-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-update_deployment-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-update_deployment-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-update_deployment-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_deployment-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_deployment-commands-[UNIQUE_NAME]/files... + Warn: retrying after 500 Internal Server Error from PATCH /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: retrying after 500 Internal Server Error from PATCH /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: Could not enable deployment history for this deploy: failed to update deployment: Fault injected by test. + Updated pipelines.bar + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_deployment-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_deployment-commands-[UNIQUE_NAME]/files... + Warn: retrying after 500 Internal Server Error from PATCH /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: retrying after 500 Internal Server Error from PATCH /api/2.0/bundle/deployments/[DEPLOYMENT_ID] + Warn: Could not enable deployment history for this deploy: failed to update deployment: Fault injected by test. + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_deployment-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_deployment-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_deployment-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: yes diff --git a/acceptance/bundle/dms_fault/update_deployment/script b/acceptance/bundle/dms_fault/update_deployment/script new file mode 100644 index 00000000000..94fdf6e7f40 --- /dev/null +++ b/acceptance/bundle/dms_fault/update_deployment/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" update_deployment diff --git a/acceptance/bundle/dms_fault/update_operation/out.test.toml b/acceptance/bundle/dms_fault/update_operation/out.test.toml new file mode 100644 index 00000000000..88e6fbd847c --- /dev/null +++ b/acceptance/bundle/dms_fault/update_operation/out.test.toml @@ -0,0 +1,2 @@ +Cloud = false +EnvMatrix.DMS = ["true"] diff --git a/acceptance/bundle/dms_fault/update_operation/output.txt b/acceptance/bundle/dms_fault/update_operation/output.txt new file mode 100644 index 00000000000..308345c48c4 --- /dev/null +++ b/acceptance/bundle/dms_fault/update_operation/output.txt @@ -0,0 +1,148 @@ + +=== commands: setup + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_operation-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1 + Created jobs.foo + Created pipelines.bar + Files: 4 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +=== commands: update_operation fault enabled + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_operation-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 + Updated jobs.foo + Updated pipelines.bar + Warn: Failed to record deployment history: recording operation for [RESOURCE_KEY]: Fault injected by test. + Files: 3 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle summary + Name: updated-dms-fault-update_operation-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_operation-commands-[UNIQUE_NAME] + Resources: + Jobs: + foo: + Name: updated-dms-fault-update_operation-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID] + Pipelines: + bar: + Name: updated-dms-fault-update_operation-commands-[UNIQUE_NAME] + URL: [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID] + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force + Job configuration successfully saved to exported/foo.job.yml + Warning: Generated configuration is not included in the bundle + + The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind foo + +>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind + Job configuration successfully saved to resources/foo.job.yml + Successfully bound job with an id '[FOO_ID]' + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force + File successfully saved to exported-src/pipeline.sql + Pipeline configuration successfully saved to exported/bar.pipeline.yml + Warning: Generated configuration is not included in the bundle + + The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml, + so it will not be deployed. Add a matching entry to the 'include' section, for example: + + include: + - exported/*.yml + + +>>> bundle deployment unbind bar + +>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind + File successfully saved to src/pipeline.sql + Pipeline configuration successfully saved to resources/bar.pipeline.yml + Successfully bound pipeline with an id '[BAR_ID]' + +>>> bundle validate + Name: updated-dms-fault-update_operation-commands-[UNIQUE_NAME] + Target: default + Workspace: + User: [USERNAME] + Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_operation-commands-[UNIQUE_NAME] + + Validation OK! + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_operation-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9 + Updated pipelines.bar + Warn: Failed to record deployment history: recording operation for [RESOURCE_KEY]: Fault injected by test. + Files: 6 uploaded, 0 deleted + Resources: 0 created, 1 changed, 0 deleted, 1 unchanged + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_operation-commands-[UNIQUE_NAME]/files... + Updated jobs.foo + Updated pipelines.bar + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +=== deployment_history: true + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_operation-commands-[UNIQUE_NAME]/files... + Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11 + Updated jobs.foo + Updated pipelines.bar + Warn: Failed to record deployment history: recording operation for [RESOURCE_KEY]: Fault injected by test. + Files: 2 uploaded, 0 deleted + Resources: 0 created, 2 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_operation-commands-[UNIQUE_NAME] + + Warn: Failed to record deployment history: recording operation for [RESOURCE_KEY]: Fault injected by test. + Destroy: 2 deleted + +=== deployment_history: false + +>>> bundle deploy --auto-approve + Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_operation-commands-[UNIQUE_NAME]/files... + Created jobs.foo + Created pipelines.bar + Files: 8 uploaded, 0 deleted + Resources: 2 created, 0 changed, 0 deleted, 0 unchanged + +>>> bundle destroy --auto-approve + The following resources will be deleted: + delete resources.jobs.foo + delete resources.pipelines.bar + + This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the + Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: + delete resources.pipelines.bar + + All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-update_operation-commands-[UNIQUE_NAME] + + Destroy: 2 deleted + +DMS API called: yes diff --git a/acceptance/bundle/dms_fault/update_operation/script b/acceptance/bundle/dms_fault/update_operation/script new file mode 100644 index 00000000000..4400cd94944 --- /dev/null +++ b/acceptance/bundle/dms_fault/update_operation/script @@ -0,0 +1 @@ +python3 "$TESTDIR/../run.py" update_operation