Skip to content

Persist the DMS version in resources.json serial #7046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shreyas-goenka merged 12 commits into from
Oct 19, 2026
Merged

Persist the DMS version in resources.json serial #7046

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b87d0fd
Persist the DMS version in the resource state serial
shreyas-goenka Oct 14, 2026
beeca57
Clarify why DMS serial is duplicated in resource state
shreyas-goenka Oct 14, 2026
ec206eb
Describe persisted DMS serial as a copy
shreyas-goenka Oct 14, 2026
9c50d7a
Test newer remote DMS state wins with history setting disabled
shreyas-goenka Oct 14, 2026
1b37103
Only assert the authoritative DMS serial after opening state
shreyas-goenka Oct 14, 2026
4733464
Show remote and DMS serials after a failed destroy
shreyas-goenka Oct 14, 2026
f67caca
Use jq to read the deployment node ID in the serial test
shreyas-goenka Oct 14, 2026
d7fed16
Reject resource state serials ahead of deployment history
shreyas-goenka Oct 14, 2026
356b242
Explain recovery from inconsistent deployment history state
shreyas-goenka Oct 15, 2026
4ed7390
Cover inconsistent DMS serial rejection and recovery
shreyas-goenka Oct 15, 2026
c810375
Remove recovery advice from inconsistent deployment state error
shreyas-goenka Oct 16, 2026
d2383c8
Format DMS serial test cases with one field per line
shreyas-goenka Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions acceptance/bin/print_state.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,8 +184,8 @@ def print_recorded_state(filename, target):
# printed state matches a non-recording run without every caller piping through nostamp.
data["state"] = scrub(get_recorded_state(target))

# The service owns the version and the file persists no serial, so take it from the deployment.
# Rebuilt in header order, since the file has no serial key to overwrite in place.
# The file's serial is a copy of the DMS version to ensure outdated resource state is not read.

@denik denik Oct 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should mention that it's serial of post-migration and it's not updated further?

@shreyas-goenka shreyas-goenka Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That'll be a followup PR. Right now it's still just the serial that's being updated every time.

# Rebuild in header order with the authoritative serial from the deployment.
last_version_id = get_last_version_id(target)
serial = int(last_version_id) if last_version_id else 0

Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/dms/declined-deploy/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/dms/depends-on/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/dms/empty-grants/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 2,
"features": {
"deployment_history": {}
},
Expand Down
11 changes: 10 additions & 1 deletion acceptance/bundle/dms/failed-delete/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down Expand Up @@ -175,14 +176,22 @@ API message: Fault injected by test.
}
]

=== The state shows the resource still exists but the operation is marked failed
=== The local state file records failed version 2
>>> print_state.py --no-dms
{
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 2,

@denik denik Oct 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also extract serial from resources.json? To show that it's one before the "serial" as returnd by print_state.py

@shreyas-goenka shreyas-goenka Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already do that. print_state.py --no-dms prints the local state. The remote state is not updated on bundle destroy that's why it's missing

"features": {
"deployment_history": {}
},
"state": {}
}

=== The remote checkpoint stays at serial 1 while DMS advances to version 2
>>> MSYS_NO_PATHCONV=1 [CLI] workspace export /Workspace/Users/[USERNAME]/.bundle/dms-failed-delete-[UNIQUE_NAME]/default/state/resources.json --format RAW
{"serial":1}

>>> print_state.py
{"serial":2}
6 changes: 5 additions & 1 deletion acceptance/bundle/dms/failed-delete/script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,12 @@ trace MSYS_NO_PATHCONV=1 $CLI bundle-deployments list-resources "deployments/${d
title "The failed operation records the error message"
trace MSYS_NO_PATHCONV=1 $CLI bundle-deployments list-operations "deployments/${deployment_id}/versions/2" -o json | jq -S 'map({resource_key, action_type, status, error_message})'

title "The state shows the resource still exists but the operation is marked failed"
title "The local state file records failed version 2"
trace print_state.py --no-dms

title "The remote checkpoint stays at serial 1 while DMS advances to version 2"
trace MSYS_NO_PATHCONV=1 $CLI workspace export "/Workspace/Users/${CURRENT_USER_NAME}/.bundle/dms-failed-delete-${UNIQUE_NAME}/default/state/resources.json" --format RAW | jq -c '{serial}'
trace print_state.py | jq -c '{serial}'

# Drain any accumulated requests (plan, state reads) so the harness has nothing to diff.
rm -f "$OUT_REQUESTS"
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/failed-recreate/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down Expand Up @@ -176,6 +177,7 @@ Files: 2 uploaded, 0 deleted
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 2,
"features": {
"deployment_history": {}
},
Expand Down
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/failed-update-permissions/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down Expand Up @@ -97,6 +98,7 @@ Files: 1 uploaded, 0 deleted
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 2,
"features": {
"deployment_history": {}
},
Expand Down
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/multiple-resources/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 5 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down Expand Up @@ -133,6 +134,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 5 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/no-drift/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down Expand Up @@ -197,6 +198,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 2,
"features": {
"deployment_history": {}
},
Expand Down
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/operation-upload-fails-once/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down Expand Up @@ -79,6 +80,7 @@ Files: 2 uploaded, 0 deleted
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 2,
"features": {
"deployment_history": {}
},
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/dms/provenance/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/dms/record-failure/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ Plan: 1 to add, 0 to change, 0 to delete, 0 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down
10 changes: 10 additions & 0 deletions acceptance/bundle/dms/stale-local-state/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
bundle:
name: dms-stale-local-state-$UNIQUE_NAME

experimental:
deployment_history: true

resources:
jobs:
foo:
name: foo
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/stale-local-state/out.test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = true
EnvMatrix.DMS = [""]
38 changes: 38 additions & 0 deletions acceptance/bundle/dms/stale-local-state/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@

=== Setup (LOG.setup): reconstruct full version 1 without a history marker, then deploy version 2
>>> jq -c {serial, features} .databricks/resources.v1.json
{"serial":1,"features":null}

>>> jq -c {serial, features} .databricks/resources.v2.json
{"serial":2,"features":{"deployment_history":{}}}

=== Local recorded serial 3 ahead of DMS version 2 is rejected
>>> update_file.py .databricks/bundle/default/resources.json "serial": 2 "serial": 3

>>> musterr [CLI] bundle deploy
Error: inconsistent deployment state: resource state serial (3) is ahead of deployment history version (2)


=== 1. Config false and env false: newer remote state replaces the local copy and reads DMS
>>> cp .databricks/resources.v1.json .databricks/bundle/default/resources.json

>>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=false [CLI] bundle summary --force-pull -o json
Warn: Deployment history setting (false) does not match the existing state (true). Using the existing state.
{"job_id":"[FOO_ID]"}

>>> cmp .databricks/resources.v2.json .databricks/bundle/default/resources.json

>>> print_requests.py --dms //resources --get --oneline
{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/resources"}

=== 2. Config removed and env false: restore the same snapshot, then read remote DMS state
>>> cp .databricks/resources.v1.json .databricks/bundle/default/resources.json

>>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=false [CLI] bundle summary --force-pull -o json
Warn: Deployment history setting (false) does not match the existing state (true). Using the existing state.
{"job_id":"[FOO_ID]"}

>>> cmp .databricks/resources.v2.json .databricks/bundle/default/resources.json

>>> print_requests.py --dms //resources --get --oneline
{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/resources"}
38 changes: 38 additions & 0 deletions acceptance/bundle/dms/stale-local-state/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
envsubst < databricks.yml.tmpl > databricks.yml

title "Setup (LOG.setup): reconstruct full version 1 without a history marker, then deploy version 2"
{
$CLI bundle deploy
DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true print_state.py > .databricks/resources.v1.json
DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true replace_ids.py
update_file.py databricks.yml 'name: foo' 'name: updated'
$CLI bundle deploy
cp .databricks/bundle/default/resources.json .databricks/resources.v2.json
deployment_id=$(MSYS_NO_PATHCONV=1 $CLI workspace get-status "/Workspace/Users/${CURRENT_USER_NAME}/.bundle/dms-stale-local-state-${UNIQUE_NAME}/default/state/resources.deployment.json" -o json | jq -r '.object_id')
add_repl "$deployment_id" DEPLOYMENT_ID
} &> LOG.setup
trace jq -c '{serial, features}' .databricks/resources.v1.json
trace jq -c '{serial, features}' .databricks/resources.v2.json

title "Local recorded serial 3 ahead of DMS version 2 is rejected"
trace update_file.py .databricks/bundle/default/resources.json '"serial": 2' '"serial": 3'
trace musterr $CLI bundle deploy

rm -f "$OUT_REQUESTS"

title "1. Config false and env false: newer remote state replaces the local copy and reads DMS"
update_file.py databricks.yml 'deployment_history: true' 'deployment_history: false'
trace cp .databricks/resources.v1.json .databricks/bundle/default/resources.json
trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=false $CLI bundle summary --force-pull -o json | jq -c '{job_id: .resources.jobs.foo.id}'
trace cmp .databricks/resources.v2.json .databricks/bundle/default/resources.json
trace print_requests.py --dms //resources --get --oneline

title "2. Config removed and env false: restore the same snapshot, then read remote DMS state"
update_file.py databricks.yml $'experimental:\n deployment_history: false\n\n' ''
trace cp .databricks/resources.v1.json .databricks/bundle/default/resources.json
trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=false $CLI bundle summary --force-pull -o json | jq -c '{job_id: .resources.jobs.foo.id}'
trace cmp .databricks/resources.v2.json .databricks/bundle/default/resources.json
trace print_requests.py --dms //resources --get --oneline

DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle destroy --auto-approve &> LOG.cleanup
rm -f "$OUT_REQUESTS"
1 change: 1 addition & 0 deletions acceptance/bundle/dms/stale-local-state/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
EnvMatrix.DMS = [""]
3 changes: 2 additions & 1 deletion acceptance/bundle/dms/stale-plan/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down Expand Up @@ -73,7 +74,7 @@ Resources: 0 created, 1 changed, 0 deleted, 0 unchanged

>>> jq {serial} .databricks/bundle/default/resources.json
{
"serial": null
"serial": 2
}

=== Replaying the older plan is rejected: same deployment, outdated version
Expand Down
2 changes: 2 additions & 0 deletions acceptance/bundle/dms/successful-recreate/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
},
Expand Down Expand Up @@ -192,6 +193,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 2,
"features": {
"deployment_history": {}
},
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/state/feature_flags/accept_new/resources.v2.deployment_history.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@
},
"cli_version": "0.0.0-dev",
"lineage": "test-lineage",
"serial": 1,
"serial": 0,
"state": {}
}
2 changes: 1 addition & 1 deletion acceptance/bundle/state/feature_flags/accept_new/resources.v3.deployment_history.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@
},
"cli_version": "0.0.0-dev",
"lineage": "test-lineage",
"serial": 1,
"serial": 0,
"state": {}
}
Loading
Loading