Skip to content

Fixed immutable folder failing with DMS or top level permissions - #7001

Merged
andrewnester merged 4 commits into
mainfrom
fix/immutable-file-path
Oct 12, 2026
Merged

andrewnester merged 4 commits into
mainfrom
fix/immutable-file-path

Conversation

@andrewnester

@andrewnester andrewnester commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Fixed experimental.immutable_folder failing with experimental.deployment_history ("workspace_info.file_path must be an absolute workspace path") and failing bundle validate when top-level permissions are set.

Why

These paths are resolved later during the deployment phase and hence were not correct
Fixes #6960

Tests

Added acceptance tests

@andrewnester
andrewnester requested review from a team as code owners October 9, 2026 10:24
@github-actions github-actions Bot added the DABs DABs related issues label Oct 9, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 015df16

Run: 38235181579

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 276 20 5:42
🔄​ aws-windows-latest-4core-16gb 2 276 18 7:24
✅​ azure linux-2core-8gb 275 20 5:29
✅​ azure-windows-latest-4core-16gb 277 18 7:33
✅​ gcp linux-2core-8gb 276 20 5:25
✅​ gcp-windows-latest-4core-16gb 278 18 7:33
Test Name aws-windows-latest-4core-16gb
🔄​ TestFsCpFileToNonExistentDir/dbfs_to_dbfs 🔄​f
🔄​ TestFsCpFileToNonExistentDir/local_to_dbfs 🔄​f
Top 6 slowest tests (at least 2 minutes):
duration env testname
7:31 azure-windows-latest-4core-16gb TestAccept
7:30 gcp-windows-latest-4core-16gb TestAccept
6:57 aws-windows-latest-4core-16gb TestAccept
4:22 aws linux-2core-8gb TestAccept
4:02 azure linux-2core-8gb TestAccept
3:56 gcp linux-2core-8gb TestAccept

Comment thread bundle/phases/dms.go Outdated

// immutableSnapshotPath returns the workspace path of the immutable folder snapshot. The
// snapshot is not uploaded yet at this point, but its content-addressed path is already known.
func immutableSnapshotPath(b *bundle.Bundle) (string, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems a bit weird to read it from state here? Is this something we can store and read from the *bundle.Bundle object instead?

]

# The snapshot path is content-addressed.
[[Repls]]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the hash is deterministic, should we consider removing this repl? Helps assert the metadata being recorded is indeed the right path.

If this causes issues across operating systems, we can limit this test to just linux. WDYT?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer to run it on all OS but then the hash is not the same across OS. I think having REPL like this is good enough for assertion

@andrewnester
andrewnester added this pull request to the merge queue Oct 12, 2026
Merged via the queue into main with commit ef609ea Oct 12, 2026
26 checks passed
@andrewnester
andrewnester deleted the fix/immutable-file-path branch October 12, 2026 10:52
denik added a commit that referenced this pull request Oct 12, 2026
…ermissions (#7001)

Co-authored-by: Isaac <no-reply@databricks.com>
deco-sdk-tagging Bot added a commit that referenced this pull request Oct 14, 2026
## Release v1.21.0

### CLI

 * Support pipeline IDs in `pipelines history`. ([#6995](#6995))
 * Show a hint for previewing more rows in `pipelines show`. ([#6995](#6995))
 * `databricks environments setup-local` now uses the environment in `UV_PROJECT_ENVIRONMENT` when it is set, as `uv sync` does. ([#7026](#7026))
 * `databricks environments setup-local` now supports uv workspace members: it uses the shared `.venv` at the workspace root and writes the dependency constraints to the root `pyproject.toml`. ([#7018](#7018))

### AI Runtime

 * Limit `databricks air run` submission preflight validation to one five-second attempt before continuing when the service is unavailable. ([#6994](#6994))

### Bundles

 * Bundle file uploads now use the multipart `POST /api/2.0/workspace/import` endpoint instead of the deprecated `POST /api/2.0/workspace-files/import-file`, raising the per-workspace upload rate limit and improving deploy throughput under concurrency. ([#6149](#6149))
 * Diagnostic paths now quote map keys that are not identifiers, e.g. `resources.jobs['${var.env}_job']`. ([#6980](#6980))
 * No longer include `CLAUDE.md` in the bundle templates, now that Claude supports `AGENTS.md`. ([#7000](#7000))
 * Fix `bundle deployment bind` silently binding over a deployment state it could not read, which could take over a resource that was already managed by the bundle. ([#6331](#6331))
 * `databricks bundle deploy`, `databricks bundle sync` and `databricks sync` now retry file, library, state and metadata uploads that fail with a transient server error (5xx or 408), instead of failing. ([#7014](#7014))
 * `databricks bundle deploy`, `databricks bundle sync` and `databricks sync` now retry the workspace path check (`workspace/get-status`) when it fails with a transient server error (5xx or 408) or an HTTP/2 `REFUSED_STREAM`, instead of failing. ([#7044](#7044))
 * Fixed `experimental.immutable_folder` failing validation for targets with `mode: development` ("file_path must start with '~/' or contain the current username"). ([#7027](#7027))
 * Fixed `experimental.immutable_folder` failing with `experimental.deployment_history` ("workspace_info.file_path must be an absolute workspace path") and failing `bundle validate` when top-level `permissions` are set. ([#7001](#7001))
 * Translate relative `.env` file paths in a job's `environment_variables[].spec.files` to their deployed workspace locations. ([#6985](#6985))
 * Resolve job `project_environment` paths relative to the YAML file that defines them. ([#6983](#6983))

### Dependency Updates

 * Bump Go toolchain to 1.26.9. ([#6996](#6996))
 * Update Go SDK to 0.188.0. ([#7038](#7038))
denik added a commit that referenced this pull request Oct 14, 2026
…ermissions (#7001)

Co-authored-by: Isaac <no-reply@databricks.com>
denik added a commit that referenced this pull request Oct 14, 2026
…ermissions (#7001)

Co-authored-by: Isaac <no-reply@databricks.com>
denik added a commit that referenced this pull request Oct 14, 2026
…ermissions (#7001)

Co-authored-by: Isaac <no-reply@databricks.com>
denik added a commit that referenced this pull request Oct 15, 2026
…ermissions (#7001)

Co-authored-by: Isaac <no-reply@databricks.com>
denik added a commit that referenced this pull request Oct 15, 2026
…ermissions (#7001)

Co-authored-by: Isaac <no-reply@databricks.com>
denik added a commit that referenced this pull request Oct 15, 2026
…ermissions (#7001)

Co-authored-by: Isaac <no-reply@databricks.com>
denik added a commit that referenced this pull request Oct 15, 2026
…ermissions (#7001)

Co-authored-by: Isaac <no-reply@databricks.com>
denik added a commit that referenced this pull request Oct 15, 2026
…ermissions (#7001)

Co-authored-by: Isaac <no-reply@databricks.com>
denik added a commit that referenced this pull request Oct 15, 2026
…ermissions (#7001)

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Validation Failing withexperimental.immutable_folder: true Despite Plan and Deploy Succeeding

3 participants