diff --git a/knowledge_base/direct_engine_migration/.gitignore b/knowledge_base/direct_engine_migration/.gitignore new file mode 100644 index 0000000..15bcc6d --- /dev/null +++ b/knowledge_base/direct_engine_migration/.gitignore @@ -0,0 +1 @@ +.databricks diff --git a/knowledge_base/direct_engine_migration/README.md b/knowledge_base/direct_engine_migration/README.md new file mode 100644 index 0000000..4a2d278 --- /dev/null +++ b/knowledge_base/direct_engine_migration/README.md @@ -0,0 +1,61 @@ +# Direct engine migration + +This Declarative Automation Bundles example migrates a serverless job from the Terraform engine to the direct engine. + +See the [direct engine documentation](https://docs.databricks.com/dev-tools/bundles/direct). + +## Prerequisites + +* Databricks CLI v1.19.x for the Terraform deployment. Version 1.20.0 removes the Terraform engine. +* Databricks CLI v1.20.0 or above for the migration + +## Usage + +| Engine | Local state | Workspace state | +| --- | --- | --- | +| Terraform | `.databricks/bundle/dev/terraform/terraform.tfstate` | `/state/terraform.tfstate` | +| Direct | `.databricks/bundle/dev/resources.json` | `/state/resources.json` | + +1. Use Databricks CLI v1.19.x to deploy with the Terraform engine. + + ```text + DATABRICKS_BUNDLE_ENGINE=terraform databricks bundle deploy + ``` + +2. Record the job ID with the Terraform engine. + + ```text + DATABRICKS_BUNDLE_ENGINE=terraform databricks bundle summary --output json + ``` + +3. Upgrade to Databricks CLI v1.20.0 or above. Migrate the local state. `databricks bundle deploy` also migrates the state automatically. + + ```text + databricks bundle deployment migrate + ``` + +4. Confirm that the migration plan has zero changes. + + ```text + databricks bundle plan + ``` + +5. Deploy and run the job. Confirm that the job ID is unchanged. + + ```text + databricks bundle deploy + databricks bundle run direct_engine_migration + databricks bundle summary --output json + ``` + +6. Confirm that the deployed bundle still has zero changes. + + ```text + databricks bundle plan + ``` + +7. Remove the job and workspace files. + + ```text + databricks bundle destroy --auto-approve + ``` diff --git a/knowledge_base/direct_engine_migration/databricks.yml b/knowledge_base/direct_engine_migration/databricks.yml new file mode 100644 index 0000000..ecc643b --- /dev/null +++ b/knowledge_base/direct_engine_migration/databricks.yml @@ -0,0 +1,12 @@ +bundle: + name: direct_engine_migration + +include: + - resources/*.yml + +targets: + dev: + default: true + mode: development + workspace: + root_path: /Workspace/bundle-examples/${bundle.name}/dev/${workspace.current_user.short_name} diff --git a/knowledge_base/direct_engine_migration/resources/direct_engine_migration.job.yml b/knowledge_base/direct_engine_migration/resources/direct_engine_migration.job.yml new file mode 100644 index 0000000..d3b5f4b --- /dev/null +++ b/knowledge_base/direct_engine_migration/resources/direct_engine_migration.job.yml @@ -0,0 +1,13 @@ +resources: + jobs: + direct_engine_migration: + name: Direct engine migration + tasks: + - task_key: hello + spark_python_task: + python_file: ../src/main.py + environment_key: default + environments: + - environment_key: default + spec: + environment_version: "6" diff --git a/knowledge_base/direct_engine_migration/src/main.py b/knowledge_base/direct_engine_migration/src/main.py new file mode 100644 index 0000000..a6d2510 --- /dev/null +++ b/knowledge_base/direct_engine_migration/src/main.py @@ -0,0 +1 @@ +print("Hello from the direct engine migration example.")