[denik]

Stack, 2 of 4. Based on #7130; next: #7143, then #7098.

Changes

Five mutators now read and edit the typed config directly instead of going through b.Config.Mutate:

• PrependWorkspacePrefix: the workspace.*_path fields.
• ApplyBundlePermissions: the permissions field of each supported resource.
• ExpandPipelineGlobPaths: pipeline libraries.
• ProcessStaticResources: lists resource keys from the typed Resources.
• ProcessRootIncludes: writes the expanded include list after the included files are merged, because merging reloads the typed config from the dynamic tree.

Split out of #7098 (#7098).

Why

This is a step toward removing dyn from the mutator pipeline. Mutators that edit typed config are simpler and don't need to convert between the two forms.

Intentional behavior notes

• (a) Invalid-permission-level warnings now show their source location (in databricks.yml:L:C). Before, ApplyBundlePermissions rebuilt the permissions without locations. One golden changed because of this.
• (b) ApplyBundlePermissions now walks resource names in sorted order. This only changes the order of overlap warnings.
• (c) Expanded include items, and pipeline libraries after glob expansion, now get the location of the original entry at the same index. Before, they had no location.
• (d) In typed config, a library with both notebook and file and an explicit notebook.path: "" looks the same as one with no notebook path.

Tests

• New unit tests for the edge cases: empty and unset permissions/libraries, library order, includes of included files, and resource keys for every resource type. They pass on both the old and the new code.
• The full TestAccept/bundle suite ran locally. The only golden diff is the one in validate/secret_scope_invalid_permission_level.

This pull request and its description were written by Isaac.

[github-actions]

Commit: cd5336354 | Workflow run

Regression test report

Selected 5 changed tests (limit=20, 0 not selected)

10 tests: regression=2 unreleased=0 coverage=8 failing=0

test	branch	main (bef6321)	latest (v1.21.0)
TestAccept/bundle/override/cluster_id_clears_new_cluster_reference/DMS=	✅	✅	✅
TestAccept/bundle/override/cluster_id_clears_new_cluster_reference/DMS=true	✅	✅	✅
TestAccept/bundle/override/preset_clears_forced_schedule_reference/DMS=	✅	✅	✅
TestAccept/bundle/override/preset_clears_forced_schedule_reference/DMS=true	✅	✅	✅
TestAccept/bundle/resource_deps/whole_map_reference_hides_entries/DMS=	✅	✅	✅
TestAccept/bundle/resource_deps/whole_map_reference_hides_entries/DMS=true	✅	✅	✅
TestAccept/bundle/validate/required_struct_reference/DMS=	✅	✅	✅
TestAccept/bundle/validate/required_struct_reference/DMS=true	✅	✅	✅
TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=	✅	❌	❌
TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=true	✅	❌	❌

TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=: branch ✅ | main (bef6321) ❌ | latest (v1.21.0) ❌

main (bef6321):

acceptance_test.go:1307: Diff:
        --- bundle/validate/secret_scope_invalid_permission_level/output.txt
        +++ /tmp/TestAcceptbundlevalidatesecret_scope_invalid_permission_level524789431/002/output.txt
        @@ -2,11 +2,9 @@
         >>> [CLI] bundle validate
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default
        @@ -19,11 +14,9 @@
         >>> [CLI] bundle validate --strict
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default

latest (v1.21.0): same output as main (bef6321)

TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=true: branch ✅ | main (bef6321) ❌ | latest (v1.21.0) ❌

main (bef6321):

acceptance_test.go:1307: Diff:
        --- bundle/validate/secret_scope_invalid_permission_level/output.txt
        +++ /tmp/TestAcceptbundlevalidatesecret_scope_invalid_permission_level2636826397/002/output.txt
        @@ -2,11 +2,9 @@
         >>> [CLI] bundle validate
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default
        @@ -19,11 +14,9 @@
         >>> [CLI] bundle validate --strict
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default
        
FAIL

latest (v1.21.0):

acceptance_test.go:1307: Diff:
        --- bundle/validate/secret_scope_invalid_permission_level/output.txt
        +++ /tmp/TestAcceptbundlevalidatesecret_scope_invalid_permission_level3553828270/002/output.txt
        @@ -2,11 +2,9 @@
         >>> [CLI] bundle validate
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default
        @@ -19,11 +14,9 @@
         >>> [CLI] bundle validate --strict
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default

[eng-dev-ecosystem-bot]

Integration test report

Commit: cd53363

Run: 39632798436

	Env	✅​pass	🙈​skip	Time
✅​	aws linux-2core-8gb	281	20	4:39
✅​	aws-windows-latest-4core-16gb	283	18	4:52
✅​	azure linux-2core-8gb	280	20	4:21
✅​	azure-windows-latest-4core-16gb	282	18	3:33
✅​	gcp linux-2core-8gb	281	20	4:48
✅​	gcp-windows-latest-4core-16gb	283	18	3:27

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
4:48	aws-windows-latest-4core-16gb	TestAccept
3:28	azure-windows-latest-4core-16gb	TestAccept
3:21	gcp-windows-latest-4core-16gb	TestAccept