Skip to content

bundle: add deployment history buffer telemetry - #7232

Draft
shreyas-goenka wants to merge 13 commits into
mainfrom
shreyas-goenka_data/shreyas-goenka/deployment-history-buffer-telemetry
Draft

bundle: add deployment history buffer telemetry#7232
shreyas-goenka wants to merge 13 commits into
mainfrom
shreyas-goenka_data/shreyas-goenka/deployment-history-buffer-telemetry

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Changes

Track deployment-history recording outcomes both per resource operation and per upload attempt.

Each buffered update contains its payload and an explicit pending/succeeded/failed state. The per-operation map points to the latest update. Workers update only the object they uploaded, so older completions cannot overwrite newer state. Intermediate recreation uploads do not finalize an operation. Aggregate operation_recording_success_count, operation_recording_failure_count, and operation_recording_pending_count partition the total staged staged_operation_count. Missing staged operations are implicitly pending, so callers do not need to pass another list of keys.

The existing recording_attempt_success_count and recording_attempt_failure_count still count individual attempts directly at the call sites. Peak queue size and the timeout queue-length measurement are retained. Only aggregate counts are emitted.

Matching ingestion schema: https://github.com/databricks-eng/universe/pull/2913283

Why

A single schema recreation can have one failed upload and one successful upload while its final history is successfully recorded. Report that as one recorded operation while retaining both attempt outcomes.

Tests

  • Race-enabled existing tests for libs/dms, bundle/direct/dstate, bundle/phases, and libs/telemetry/....
  • Existing oversized-state test checks one failed operation and zero pending operations; the coalescing test also checks that newer updates get a separate object and preserve the earlier payload.
  • Telemetry/DMS acceptance cases: the healthy case records all 5 operations; the recreate case records 1 operation with 0 failed/pending operations while retaining 1 successful and 1 failed upload attempt.
  • golangci-lint passes on affected Go packages.

Local checks used a temporary cached golang.org/x/net v0.61.0 override because the internal proxy blocks main's v0.62.0 as recently published. Tracked dependency files are unchanged.

@github-actions github-actions Bot added the DABs DABs related issues label Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: 7af90d6cf | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0

test branch main (e9687c1) latest (v1.22.1)
TestAccept/bundle/dms/multiple-resources/DMS=true ✅ ❌ ❌
TestAccept/bundle/dms/operation-upload-fails-once/DMS=true ✅ ❌ ❌
TestAccept/bundle/dms/multiple-resources/DMS=true: branch ✅ | main (e9687c1) ❌ | latest (v1.22.1) ❌

main (e9687c1):

acceptance_test.go:1341: Diff:
        --- bundle/dms/multiple-resources/output.txt
        +++ /tmp/TestAcceptbundledmsmultiple-resourcesDMS=true3539376518/002/output.txt
        @@ -13,14 +13,14 @@
         === History telemetry survives version completion and counts all five staged operations
         >>> cat [OUTPUT_DIR]/out.requests.txt
         {
        -  "staged_operation_count": 5,
        -  "operation_recording_success_count": 5,
        -  "operation_recording_failure_count": 0,
        -  "operation_recording_pending_count": 0,
        -  "queued_update_count_at_timeout": 0,
        -  "recording_attempt_failure_count": 0,
        -  "recording_attempt_success_count": 5,
        -  "queue_size_in_range": true
        +  "staged_operation_count": null,
        +  "operation_recording_success_count": null,
        +  "operation_recording_failure_count": null,
        +  "operation_recording_pending_count": null,
        +  "queued_update_count_at_timeout": null,
        +  "recording_attempt_failure_count": null,
        +  "recording_attempt_success_count": null,
        +  "queue_size_in_range": false
         }
         
         >>> print_requests.py --dms //versions/1/operations --sort --del-body state --oneline

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/dms/multiple-resources/output.txt
        +++ /tmp/TestAcceptbundledmsmultiple-resourcesDMS=true1699510805/002/output.txt
        @@ -13,14 +13,14 @@
         === History telemetry survives version completion and counts all five staged operations
         >>> cat [OUTPUT_DIR]/out.requests.txt
         {
        -  "staged_operation_count": 5,
        -  "operation_recording_success_count": 5,
        -  "operation_recording_failure_count": 0,
        -  "operation_recording_pending_count": 0,
        -  "queued_update_count_at_timeout": 0,
        -  "recording_attempt_failure_count": 0,
        -  "recording_attempt_success_count": 5,
        -  "queue_size_in_range": true
        +  "staged_operation_count": null,
        +  "operation_recording_success_count": null,
        +  "operation_recording_failure_count": null,
        +  "operation_recording_pending_count": null,
        +  "queued_update_count_at_timeout": null,
        +  "recording_attempt_failure_count": null,
        +  "recording_attempt_success_count": null,
        +  "queue_size_in_range": false
         }
         
         >>> print_requests.py --dms //versions/1/operations --sort --del-body state --oneline
        @@ -38,7 +38,7 @@
           "lineage": "[UUID]",
           "serial": 1,
           "features": {
        -    "deployment_history_shadow": {}
        +    "deployment_history": {}
           }
         }
         
        @@ -83,100 +83,9 @@
          "lineage": "[UUID]",
          "serial": 1,
          "features": {
        -  "deployment_history_shadow": {}
        +  "deployment_history": {}
          },
        - "state": {
        -  "resources.jobs.five": {
        -   "__id__": "[FIVE_ID]",
        -   "state": {
        -    "deployment": {
        -     "deployment_id": "[DEPLOYMENT_ID]",
        -     "kind": "BUNDLE",
        -     "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-multiple-resources-[UNIQUE_NAME]/default/state/metadata.json",
        -     "version_id": "1"
        -    },
        -    "edit_mode": "UI_LOCKED",
        -    "format": "MULTI_TASK",
        -    "max_concurrent_runs": 1,
        -    "name": "five",
        -    "queue": {
        -     "enabled": true
        -    }
        -   }
        -  },
        -  "resources.jobs.four": {
        -   "__id__": "[FOUR_ID]",
        -   "state": {
        -    "deployment": {
        -     "deployment_id": "[DEPLOYMENT_ID]",
        -     "kind": "BUNDLE",
        -     "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-multiple-resources-[UNIQUE_NAME]/default/state/metadata.json",
        -     "version_id": "1"
        -    },
        -    "edit_mode": "UI_LOCKED",
        -    "format": "MULTI_TASK",
        -    "max_concurrent_runs": 1,
        -    "name": "four",
        -    "queue": {
        -     "enabled": true
        -    }
        -   }
        -  },
        -  "resources.jobs.one": {
        -   "__id__": "[ONE_ID]",
        -   "state": {
        -    "deployment": {
        -     "deployment_id": "[DEPLOYMENT_ID]",
        -     "kind": "BUNDLE",
        -     "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-multiple-resources-[UNIQUE_NAME]/default/state/metadata.json",
        -     "version_id": "1"
        -    },
        -    "edit_mode": "UI_LOCKED",
        -    "format": "MULTI_TASK",
        -    "max_concurrent_runs": 1,
        -    "name": "one",
        -    "queue": {
        -     "enabled": true
        -    }
        -   }
        -  },
        -  "resources.jobs.three": {
        -   "__id__": "[THREE_ID]",
        -   "state": {
        -    "deployment": {
        -     "deployment_id": "[DEPLOYMENT_ID]",
        -     "kind": "BUNDLE",
        -     "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-multiple-resources-[UNIQUE_NAME]/default/state/metadata.json",
        -     "version_id": "1"
        -    },
        -    "edit_mode": "UI_LOCKED",
        -    "format": "MULTI_TASK",
        -    "max_concurrent_runs": 1,
        -    "name": "three",
        -    "queue": {
        -     "enabled": true
        -    }
        -   }
        -  },
        -  "resources.jobs.two": {
        -   "__id__": "[TWO_ID]",
        -   "state": {
        -    "deployment": {
        -     "deployment_id": "[DEPLOYMENT_ID]",
        -     "kind": "BUNDLE",
        -     "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-multiple-resources-[UNIQUE_NAME]/default/state/metadata.json",
        -     "version_id": "1"
        -    },
        -    "edit_mode": "UI_LOCKED",
        -    "format": "MULTI_TASK",
        -    "max_concurrent_runs": 1,
        -    "name": "two",
        -    "queue": {
        -     "enabled": true
        -    }
        -   }
        -  }
        - }
        + "state": {}
         }
         
         === Redeploy with no changes: nothing is applied, so no version is created at all - the deployment's version tracks state changes, like serial
        @@ -199,7 +107,7 @@
           "lineage": "[UUID]",
           "serial": 1,
           "features": {
        -    "deployment_history_shadow": {}
        +    "deployment_history": {}
           }
         }
         
        @@ -244,98 +152,7 @@
          "lineage": "[UUID]",
          "serial": 1,
          "features": {
        -  "deployment_history_shadow": {}
        +  "deployment_history": {}
          },
        - "state": {
        -  "resources.jobs.five": {
        -   "__id__": "[FIVE_ID]",
        -   "state": {
        -    "deployment": {
        -     "deployment_id": "[DEPLOYMENT_ID]",
        -     "kind": "BUNDLE",
        -     "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-multiple-resources-[UNIQUE_NAME]/default/state/metadata.json",
        -     "version_id": "1"
        -    },
        -    "edit_mode": "UI_LOCKED",
        -    "format": "MULTI_TASK",
        -    "max_concurrent_runs": 1,
        -    "name": "five",
        -    "queue": {
        -     "enabled": true
        -    }
        -   }
        -  },
        -  "resources.jobs.four": {
        -   "__id__": "[FOUR_ID]",
        -   "state": {
        -    "deployment": {
        -     "deployment_id": "[DEPLOYMENT_ID]",
        -     "kind": "BUNDLE",
        -     "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-multiple-resources-[UNIQUE_NAME]/default/state/metadata.json",
        -     "version_id": "1"
        -    },
        -    "edit_mode": "UI_LOCKED",
        -    "format": "MULTI_TASK",
        -    "max_concurrent_runs": 1,
        -    "name": "four",
        -    "queue": {
        -     "enabled": true
        -    }
        -   }
        -  },
        -  "resources.jobs.one": {
        -   "__id__": "[ONE_ID]",
        -   "state": {
        -    "deployment": {
        -     "deployment_id": "[DEPLOYMENT_ID]",
        -     "kind": "BUNDLE",
        -     "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-multiple-resources-[UNIQUE_NAME]/default/state/metadata.json",
        -     "version_id": "1"
        -    },
... (50 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)
TestAccept/bundle/dms/operation-upload-fails-once/DMS=true: branch ✅ | main (e9687c1) ❌ | latest (v1.22.1) ❌

main (e9687c1):

acceptance_test.go:1341: Diff:
        --- bundle/dms/operation-upload-fails-once/output.txt
        +++ /tmp/TestAcceptbundledmsoperation-upload-fails-onceDMS=true3079759301/002/output.txt
        @@ -71,13 +71,13 @@
         === A failed history upload is counted separately from the queued operations
         >>> cat [OUTPUT_DIR]/out.requests.txt
         {
        -  "staged_operation_count": 1,
        -  "operation_recording_success_count": 1,
        -  "operation_recording_failure_count": 0,
        -  "operation_recording_pending_count": 0,
        -  "recording_attempt_failure_count": 1,
        -  "recording_attempt_success_count": 1,
        -  "queued_update_count_at_timeout": 0
        +  "staged_operation_count": null,
        +  "operation_recording_success_count": null,
        +  "operation_recording_failure_count": null,
        +  "operation_recording_pending_count": null,
        +  "recording_attempt_failure_count": null,
        +  "recording_attempt_success_count": null,
        +  "queued_update_count_at_timeout": null
         }
         
         === The service lists the schema with the failure
        
    acceptance_test.go:1119: 
        LOG.catalog: {
        LOG.catalog:   "catalog_type": "MANAGED_CATALOG",
        LOG.catalog:   "created_at": 1793232095331,
        LOG.catalog:   "created_by": "tester@databricks.com",
        LOG.catalog:   "full_name": "dms_other_ujqmyurtovd7pkglswnacz3ixu",
        LOG.catalog:   "metastore_id": "7a07d600-8dc9-abbc-7a07-d6008dc9acb6",
        LOG.catalog:   "name": "dms_other_ujqmyurtovd7pkglswnacz3ixu",
        LOG.catalog:   "owner": "tester@databricks.com",
        LOG.catalog:   "updated_at": 1793232095331,
        LOG.catalog:   "updated_by": "tester@databricks.com"
        LOG.catalog: }
FAIL

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/dms/operation-upload-fails-once/output.txt
        +++ /tmp/TestAcceptbundledmsoperation-upload-fails-onceDMS=true4064607238/002/output.txt
        @@ -14,7 +14,7 @@
           "lineage": "[UUID]",
           "serial": 1,
           "features": {
        -    "deployment_history_shadow": {}
        +    "deployment_history": {}
           }
         }
         
        @@ -39,18 +39,9 @@
          "lineage": "[UUID]",
          "serial": 1,
          "features": {
        -  "deployment_history_shadow": {}
        +  "deployment_history": {}
          },
        - "state": {
        -  "resources.schemas.foo": {
        -   "__id__": "main.dms_upload_fails_once_schema_[UNIQUE_NAME]",
        -   "state": {
        -    "catalog_name": "main",
        -    "comment": "v1",
        -    "name": "dms_upload_fails_once_schema_[UNIQUE_NAME]"
        -   }
        -  }
        - }
        + "state": {}
         }
         
         === A recreate records twice, and the first one fails: the second must still carry the sequence id staging left, because a rejected update earned no new one
        @@ -64,125 +54,13 @@
         This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost:
           recreate resources.schemas.foo
         Recreated schemas.foo
        -Warn: Failed to record deployment history: recording operation for resources.schemas.foo: Fault injected by test.
        -Files: 2 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 1 deleted, 0 unchanged
        +Error: recording operation for resources.schemas.foo: Fault injected by test. (500 INJECTED)
         
        -=== A failed history upload is counted separately from the queued operations
        ->>> cat [OUTPUT_DIR]/out.requests.txt
        -{
        -  "staged_operation_count": 1,
        -  "operation_recording_success_count": 1,
        -  "operation_recording_failure_count": 0,
        -  "operation_recording_pending_count": 0,
        -  "recording_attempt_failure_count": 1,
        -  "recording_attempt_success_count": 1,
        -  "queued_update_count_at_timeout": 0
        -}
        +Endpoint: PATCH [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/operations/schemas.foo?update_mask=state%2Cerror_message%2Cstatus
        +HTTP Status: 500 Internal Server Error
        +API error_code: INJECTED
        +API message: Fault injected by test.
         
        -=== The service lists the schema with the failure
        ->>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-resources deployments/[DEPLOYMENT_ID] -o json
        -[
        -  {
        -    "last_action_type": "OPERATION_ACTION_TYPE_RECREATE",
        -    "last_version_id": "2",
        -    "resource_key": "schemas.foo"
        -  }
        -]
        +Files: 2 uploaded, 0 deleted
         
        -=== The raw recorded state after the failure
        ->>> print_state.py --no-dms
        -{
        - "state_version": 3,
        - "cli_version": "[CLI_VERSION]",
        - "lineage": "[UUID]",
        - "serial": 2,
        - "features": {
        -  "deployment_history_shadow": {}
        - },
        - "state": {
        -  "resources.schemas.foo": {
        -   "__id__": "dms_other_[UNIQUE_NAME].dms_upload_fails_once_schema_[UNIQUE_NAME]",
        -   "state": {
        -    "catalog_name": "dms_other_[UNIQUE_NAME]",
        -    "comment": "v1",
        -    "name": "dms_upload_fails_once_schema_[UNIQUE_NAME]"
        -   }
        -  }
        - }
        -}
        -
        ->>> print_requests.py --dms //api/2.0/bundle
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
        -  "q": {
        -    "version_id": "2"
        -  },
        -  "body": {
        -    "cli_version": "[CLI_VERSION]",
        -    "operations": [
        -      {
        -        "action_type": "OPERATION_ACTION_TYPE_RECREATE",
        -        "resource_key": "schemas.foo"
        -      }
        -    ],
        -    "previous_version_id": "1",
        -    "version_type": "VERSION_TYPE_DEPLOY"
        -  }
        -}
        -{
        -  "method": "PATCH",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/operations/schemas.foo",
        -  "q": {
        -    "update_mask": "state,error_message,status"
        -  },
        -  "body": {
        -    "error_message": "",
        -    "sequence_id": 0,
        -    "status": "OPERATION_STATUS_PENDING"
        -  }
        -}
        -{
        -  "method": "PATCH",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/operations/schemas.foo",
        -  "q": {
        -    "update_mask": "state,error_message,resource_id,status"
        -  },
        -  "body": {
        -    "error_message": "",
        -    "resource_id": "dms_other_[UNIQUE_NAME].dms_upload_fails_once_schema_[UNIQUE_NAME]",
        -    "sequence_id": 0,
        -    "state": "{\"state\":{\"catalog_name\":\"dms_other_[UNIQUE_NAME]\",\"comment\":\"v1\",\"name\":\"dms_upload_fails_once_schema_[UNIQUE_NAME]\"}}",
        -    "status": "OPERATION_STATUS_SUCCEEDED"
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete",
        -  "body": {
        -    "completion_reason": "VERSION_COMPLETE_FAILURE"
        -  }
        -}
        -
        -=== A completion failure does not block destroy or remote file cleanup
        ->>> fault.py POST /api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/complete 400 0 1
        -
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.schemas.foo
        -
        -This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
        -  delete resources.schemas.foo
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-upload-fails-once-[UNIQUE_NAME]/default
        -
        -Warn: Failed to complete deployment history version: Fault injected by test.
        -Destroy: 1 deleted
        -
        ->>> print_requests.py --dms //versions/3 --oneline
        -{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/operations/schemas.foo", "q": {"update_mask": "state,error_message,status"}, "body": {"error_message": "", "sequence_id": 0, "status": "OPERATION_STATUS_SUCCEEDED"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
        -
        ->>> musterr env MSYS_NO_PATHCONV=1 [CLI] workspace get-status /Workspace/Users/[USERNAME]/.bundle/dms-upload-fails-once-[UNIQUE_NAME]/default/state/resources.json
        -Error: Path (/Workspace/Users/[USERNAME]/.bundle/dms-upload-fails-once-[UNIQUE_NAME]/default/state/resources.json) doesn't exist.
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.catalog: {
        LOG.catalog:   "catalog_type": "MANAGED_CATALOG",
        LOG.catalog:   "created_at": 1793232100346,
        LOG.catalog:   "created_by": "tester@databricks.com",
        LOG.catalog:   "full_name": "dms_other_6cpbkmqy5fa5hek3eilxr2ewqa",
        LOG.catalog:   "metastore_id": "7a07d601-b8bc-ba6f-7a07-d601b8bcbb60",
        LOG.catalog:   "name": "dms_other_6cpbkmqy5fa5hek3eilxr2ewqa",
        LOG.catalog:   "owner": "tester@databricks.com",
        LOG.catalog:   "updated_at": 1793232100346,
        LOG.catalog:   "updated_by": "tester@databricks.com"
        LOG.catalog: }

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 7af90d6

Run: 40790551365

Env ❌​FAIL ✅​pass 🙈​skip Time
❌​ aws linux-2core-8gb 1 282 16 5:21
❌​ aws-windows-latest-4core-16gb 1 284 14 4:02
❌​ azure linux-2core-8gb 1 281 16 5:12
❌​ azure-windows-latest-4core-16gb 1 283 14 3:55
❌​ gcp linux-2core-8gb 1 282 16 5:22
❌​ gcp-windows-latest-4core-16gb 1 284 14 3:48
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
❌​ TestSetupLocalServerlessProvision ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
Top 6 slowest tests (at least 2 minutes):
duration env testname
3:40 azure linux-2core-8gb TestAccept
3:40 gcp linux-2core-8gb TestAccept
3:35 aws linux-2core-8gb TestAccept
3:14 aws-windows-latest-4core-16gb TestAccept
3:08 azure-windows-latest-4core-16gb TestAccept
3:04 gcp-windows-latest-4core-16gb TestAccept

@shreyas-goenka
bundle: inline deployment history drain measurements
a78eec5
@shreyas-goenka
bundle: simplify deployment history backlog telemetry
87ba39a
@shreyas-goenka
bundle: record undrained operations only when drain is interrupted
6032d50
@shreyas-goenka
bundle: directly record queue length on drain timeout
9357085
@shreyas-goenka
bundle: drop operation buffer telemetry unit tests
2b4a818
@shreyas-goenka
bundle: drop deployment state telemetry unit test
e36fb9e
@shreyas-goenka
bundle: track deployment history recording errors
62611f8
@shreyas-goenka
bundle: count history upload successes and errors at the API call
8ee2607
@shreyas-goenka
bundle: include total operation count in failed-upload telemetry output
699f3ec
@shreyas-goenka
bundle: report per-operation recording outcomes alongside upload atte…
fb1c5af 
…mpts
@shreyas-goenka
bundle: use explicit operation, attempt, and queue telemetry names
03655d4
@shreyas-goenka
bundle: attach recording state to each buffered update
7af90d6

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

DABs DABs related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@shreyas-goenka @eng-dev-ecosystem-bot