Change to severless clusters#17
Merged
Merged
Conversation
…emoval Reverse the workshop-only changes so the bundle and notebooks go back to serverless defaults: - Drop the shared_cluster_id lookup variable and every task's existing_cluster_id, so jobs run on the default (serverless) compute again. - Reset catalog_name back to dev: the bundle default, the personal target override, and the catalog/catalog_name widget default in every notebook. - Restore the %pip install + dbutils.library.restartPython setup cells that were removed for the classic ML Runtime clusters. engine: direct is kept intentionally. Regenerate src from solution so the gapped notebooks stay in sync.
staging was deploying with no mode (no dev prefixing, but also none of the production guardrails). Set it to mode: production so it mirrors prod: schedules stay active, no name-prefixing, and the deploy is not tied to a personal user. Production mode needs a fixed, non-user root_path and a run_as identity, so: - Add a cicd_service_principal bundle variable (empty default, so the personal target is unaffected). - staging: mode: production, run_as the service principal, and a fixed root_path matching prod. - CD: pass the SP application id (DATABRICKS_CLIENT_ID) to the bundle via BUNDLE_VAR_cicd_service_principal for the staging deploy.
The ML-outputs schema base name `fraud` sat right next to the medallion schemas (fraud_landing/bronze/silver/gold), which read ambiguously. Rename the ml_workspace base name to `fraud_ml` so it is clearly the ML schema: it resolves to `dev.fraud_ml` on the clean targets and `dev_<user>_fraud_ml` in development mode. Update the interactive per-user fallbacks in the notebooks to match and regenerate the gapped src.
evaluation.py calls fe.score_batch but was missing the _MLFLOW_SPARK_UDF_SERVERLESS_SKIP_DBCONNECT_ARTIFACT flag that batch_inference.py and seed_drift.py got in 7b7cbdd. On serverless the model UDF sandbox reports a non-PEP440 runtime tag (18.x-photon-scala2), so score_batch raised InvalidVersion and the deployment job's Evaluation task failed. Set the flag before fe.score_batch here too.
seed_drift.py calls w.quality_monitors.run_refresh but had no pip install, so on serverless the pre-installed databricks-sdk lacks the quality_monitors attribute and the refresh raised AttributeError. Add the same %pip install --upgrade databricks-sdk + restartPython that monitoring.py uses.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the workshop bundle and notebooks to run without relying on a pre-existing shared cluster (moving toward serverless execution), while also standardizing defaults to the dev catalog and renaming the per-user ML workspace schema from fraud to fraud_ml. It adds notebook-side %pip install setup steps to ensure required libraries are available when running on serverless.
Changes:
- Remove
existing_cluster_idusage and theshared_cluster_idbundle variable to decouple workflows from a specific shared cluster. - Update notebooks to default to
catalog_name=devand to use a renamed per-user schema suffix (dev_<user>_fraud_ml), aligned with the updated schema resource (fraud_ml). - Add serverless-oriented notebook setup (
%pip install+restartPython()), and update CI/CD bundle vars to support production-moderun_asfor staging.
Reviewed changes
Copilot reviewed 35 out of 35 changed files in this pull request and generated 18 comments.
Show a summary per file
| File | Description |
|---|---|
| src/training/notebooks/training.py | Adds serverless setup installs; switches default catalog to dev; updates derived ml_schema suffix to _fraud_ml. |
| src/monitoring/notebooks/seed_drift.py | Adds databricks-sdk install/restart for serverless; switches default catalog to dev; updates derived ml_schema suffix. |
| src/monitoring/notebooks/process_serving_logs.py | Switches default catalog to dev; updates derived ml_schema suffix. |
| src/monitoring/notebooks/monitoring.py | Adds databricks-sdk install/restart for serverless; switches default catalog to dev; updates derived ml_schema suffix. |
| src/feature_engineering/notebooks/build_features.py | Adds Feature Engineering client install/restart for serverless; switches default catalog to dev; updates derived ml_schema suffix. |
| src/exploration/notebooks/eda.py | Switches default catalog to dev. |
| src/deployment/model_deployment/notebooks/query_endpoint.py | Switches default catalog to dev; updates derived ml_schema suffix. |
| src/deployment/model_deployment/notebooks/evaluation.py | Adds mlflow/FE installs + restart for serverless; switches default catalog to dev; updates derived ml_schema suffix; adds serverless workaround env var. |
| src/deployment/model_deployment/notebooks/deployment.py | Adds mlflow/databricks-sdk/FE installs + restart for serverless; switches default catalog to dev; updates derived ml_schema suffix. |
| src/deployment/model_deployment/notebooks/approval.py | Adds mlflow install + restart; switches default catalog to dev; updates derived ml_schema suffix. |
| src/deployment/batch_inference/notebooks/batch_inference.py | Adds mlflow/FE installs + restart for serverless; switches default catalog to dev; updates derived ml_schema suffix. |
| src/data_ingestion/notebooks/load_landing_to_bronze.py | Switches default catalog widget to dev. |
| src/data_ingestion/notebooks/load_bronze_to_silver.py | Switches default catalog widget to dev. |
| solution/training/notebooks/training.py | Mirrors src training serverless setup + fraud_ml schema changes for the solution tree. |
| solution/monitoring/notebooks/seed_drift.py | Mirrors src monitoring serverless setup + fraud_ml schema changes for the solution tree. |
| solution/monitoring/notebooks/process_serving_logs.py | Mirrors src catalog/schema default changes for the solution tree. |
| solution/monitoring/notebooks/monitoring.py | Mirrors src monitoring serverless setup + fraud_ml schema changes for the solution tree. |
| solution/feature_engineering/notebooks/build_features.py | Mirrors src FE serverless setup + fraud_ml schema changes for the solution tree. |
| solution/exploration/notebooks/eda.py | Mirrors src catalog default change for the solution tree. |
| solution/deployment/model_deployment/notebooks/query_endpoint.py | Mirrors src catalog/schema default changes for the solution tree. |
| solution/deployment/model_deployment/notebooks/evaluation.py | Mirrors src evaluation serverless setup + workaround for the solution tree. |
| solution/deployment/model_deployment/notebooks/deployment.py | Mirrors src deployment serverless setup for the solution tree. |
| solution/deployment/model_deployment/notebooks/approval.py | Mirrors src approval serverless setup for the solution tree. |
| solution/deployment/batch_inference/notebooks/batch_inference.py | Mirrors src batch inference serverless setup for the solution tree. |
| solution/data_ingestion/notebooks/load_landing_to_bronze.py | Mirrors src ingestion catalog default change for the solution tree. |
| solution/data_ingestion/notebooks/load_bronze_to_silver.py | Mirrors src ingestion catalog default change for the solution tree. |
| resources/serving-log-processing-workflow.yml | Removes existing_cluster_id from the serving-log processing job task. |
| resources/monitoring-resource.yml | Removes existing_cluster_id from the retraining job’s notebook task. |
| resources/model-training-workflow.yml | Removes existing_cluster_id from the training job task. |
| resources/feature-engineering-workflow.yml | Removes existing_cluster_id from the feature engineering job task. |
| resources/deployment-job-workflow.yml | Removes existing_cluster_id from deployment job tasks (evaluation/approval/deployment). |
| resources/batch-inference-workflow.yml | Removes existing_cluster_id from the batch inference job task. |
| resources/schemas-resource.yml | Renames the ML workspace schema resource from fraud to fraud_ml and updates its documentation. |
| databricks.yml | Removes the shared cluster lookup var; defaults catalog to dev; adds staging run_as via cicd_service_principal; configures staging root_path. |
| .github/workflows/cd.yml | Exposes the CI/CD service principal client ID into bundle vars for staging deployments. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
9
to
12
| tasks: | ||
| - task_key: train | ||
| existing_cluster_id: ${var.shared_cluster_id} | ||
| notebook_task: | ||
| notebook_path: ../${var.code_root}/training/notebooks/training.py |
Comment on lines
10
to
13
| tasks: | ||
| - task_key: build_features | ||
| existing_cluster_id: ${var.shared_cluster_id} | ||
| notebook_task: | ||
| notebook_path: ../${var.code_root}/feature_engineering/notebooks/build_features.py |
Comment on lines
8
to
11
| tasks: | ||
| - task_key: batch_inference | ||
| existing_cluster_id: ${var.shared_cluster_id} | ||
| notebook_task: | ||
| notebook_path: ../${var.code_root}/deployment/batch_inference/notebooks/batch_inference.py |
Comment on lines
12
to
16
| name: ${bundle.target}-mlops-workshop-serving-log-processing | ||
| tasks: | ||
| - task_key: process_serving_logs | ||
| existing_cluster_id: ${var.shared_cluster_id} | ||
| notebook_task: | ||
| notebook_path: ../${var.code_root}/monitoring/notebooks/process_serving_logs.py |
Comment on lines
30
to
34
| tasks: | ||
| - task_key: Evaluation | ||
| existing_cluster_id: ${var.shared_cluster_id} | ||
| max_retries: 0 | ||
| notebook_task: | ||
| notebook_path: ../${var.code_root}/deployment/model_deployment/notebooks/evaluation.py |
|
|
||
| # COMMAND ---------- | ||
|
|
||
| # MAGIC %pip install -q "mlflow>=3.0" --upgrade |
|
|
||
| # COMMAND ---------- | ||
|
|
||
| # MAGIC %pip install -q "mlflow>=3.0" --upgrade |
|
|
||
| # COMMAND ---------- | ||
|
|
||
| # MAGIC %pip install -q "mlflow>=3.0" --upgrade |
Comment on lines
+26
to
+29
| # MAGIC Upgrade MLflow to 3.x for the new registry/logged-model features. Install the | ||
| # MAGIC Feature Engineering client **without `--upgrade`** so it stays at the runtime version | ||
| # MAGIC that created the feature table (mismatched client versions can't resolve the feature | ||
| # MAGIC table). On ML Runtime the client is pre-installed; on serverless this adds it. |
Comment on lines
+26
to
+29
| # MAGIC Upgrade MLflow to 3.x for the new registry/logged-model features. Install the | ||
| # MAGIC Feature Engineering client **without `--upgrade`** so it stays at the runtime version | ||
| # MAGIC that created the feature table (mismatched client versions can't resolve the feature | ||
| # MAGIC table). On ML Runtime the client is pre-installed; on serverless this adds it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.