[chenyuem-db]

Changes

Support source files in pipelines run --refresh and --full-refresh. Local paths resolve through the bundle sync configuration, including notebook naming and exclusions; absolute /Workspace/ paths are also accepted.

databricks pipelines deploy
databricks pipelines run my_pipeline --refresh src/pipeline.py
databricks pipelines run my_pipeline --full-refresh src/pipeline.py

The CLI sends deployed workspace paths in refresh_selection_by_path or full_refresh_selection_by_path. The backend selects the tables whose definition paths match. GetUpdate echoes those selectors, and the CLI summary shows the selected source paths.

Each flag accepts either table names or files. Mixing both within one flag fails before a restart can cancel the running update; using different selector types across the two flags is supported. Existing table selectors remain supported: ./ requires a file, and backticks force table interpretation for ambiguous names. Path resolution follows the effective bundle sync mode, including precedence over deprecated source-linked settings. A bare table name remains valid when a local directory has the same name; ./ explicitly selects a path. Unresolved immutable-folder paths require an absolute /Workspace/ path.

Why

Let users refresh all tables defined in a source file without enumerating table names, as described in the SDP Turbo CLI PRD. Local changes must be deployed first.

This PR covers file-based refresh execution and source-path reporting. The PRD requirement to display the actual selected table-name list is not implemented. This remains an outstanding requirement; displaying the file paths does not fulfill it.

Tests

• Reuse the existing Python pipeline fixture with two table definitions. Acceptance tests assert the serialized request and completion summary for both file refresh modes, preserve table-based refresh coverage, and verify invalid input is rejected before restart.
• The shared test API retains selectors per update and returns them from GetUpdate, matching the backend contract. It does not parse source code or simulate graph-to-table resolution.
• Existing unit tests cover multiple files, notebook path conversion, sync exclusions, invalid-notebook errors, effective deployment modes and deprecated-setting overrides, table/directory name collisions, supported cross-mode selections, and incompatible selectors.

Verified the deployment-mode and table/directory regressions fail before the fix and pass afterward. Current local validation passed go test ./cmd/pipelines ./bundle/run ./libs/testserver, pipeline-run and bundle-run acceptance tests without update mode, and ./task fmt, ./task checks, and ./task lint.

Local validation retains SDK v0.191.0 and temporarily uses golang.org/x/net v0.61.0 because the internal proxy blocked v0.62.0. The temporary dependency change is removed before committing; this PR retains main's v0.62.0 and has no dependency diff. CI on the new commit must validate the official dependency set. No manual live-workspace refresh was performed.

Implementation and PR description prepared with Codex.

[eng-dev-ecosystem-bot]

Integration test report

Commit: 9fd46d5

Run: 40780618531

	Env	❌​FAIL	✅​pass	🙈​skip	Time
❌​	aws linux-2core-8gb	1	280	16	5:10
❌​	aws-windows-latest-4core-16gb	1	282	14	5:11
❌​	azure linux-2core-8gb	1	279	16	5:01
❌​	azure-windows-latest-4core-16gb	1	281	14	5:33
❌​	gcp linux-2core-8gb	1	280	16	5:07
❌​	gcp-windows-latest-4core-16gb	1	282	14	5:51

	Test Name	aws linux-2core-8gb	aws-windows-latest-4core-16gb	azure linux-2core-8gb	azure-windows-latest-4core-16gb	gcp linux-2core-8gb	gcp-windows-latest-4core-16gb
❌​	TestSetupLocalServerlessProvision	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
5:23	gcp-windows-latest-4core-16gb	TestAccept
5:04	azure-windows-latest-4core-16gb	TestAccept
4:35	aws-windows-latest-4core-16gb	TestAccept
3:47	aws linux-2core-8gb	TestAccept
3:41	azure linux-2core-8gb	TestAccept
3:38	gcp linux-2core-8gb	TestAccept

[github-actions]

Commit: 9fd46d510 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0

test	branch	main (e9687c1)	latest (v1.22.1)
TestAccept/pipelines/run/refresh-flags	✅	❌	❌

TestAccept/pipelines/run/refresh-flags: branch ✅ | main (e9687c1) ❌ | latest (v1.22.1) ❌

main (e9687c1):

acceptance_test.go:1341: Diff:
        --- pipelines/run/refresh-flags/output.txt
        +++ /tmp/TestAcceptpipelinesrunrefresh-flags2199150452/002/output.txt
        @@ -127,15 +127,15 @@
         Update for pipeline completed successfully.
         
         Pipeline configurations for this update:
        -• Refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files/pipeline_file.py]
        +• Refreshed [pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
           "path": "/api/2.0/pipelines/[UUID]/updates",
           "body": {
        -    "refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files/pipeline_file.py"
        +    "refresh_selection": [
        +      "pipeline_file.py"
             ]
           }
         }
        @@ -153,15 +147,15 @@
         Update for pipeline completed successfully.
         
         Pipeline configurations for this update:
        -• Full refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files/pipeline_file.py]
        +• Full refreshed [pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
           "path": "/api/2.0/pipelines/[UUID]/updates",
           "body": {
        -    "full_refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files/pipeline_file.py"
        +    "full_refresh_selection": [
        +      "pipeline_file.py"
             ]
           }
         }
        @@ -172,37 +160,15 @@
         
         If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
         
        -Error: invalid --refresh: source file "./missing.sql" not found; use a table name or an existing source file
        +Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
         
        ->>> musterr [CLI] pipelines run --refresh pipeline_file.py --full-refresh-all --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Update ID: [UUID]
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Update for pipeline completed successfully.
         
        -Error: cannot use --refresh or --full-refresh together with --full-refresh-all, these flags are mutually exclusive
        +Pipeline configurations for this update:
        +• Refreshed [./missing.sql]
         
        ->>> musterr [CLI] pipelines run --full-refresh ./missing.sql --restart --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Unexpected success
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: invalid --full-refresh: source file "./missing.sql" not found; use a table name or an existing source file
        -
        ->>> test ! -e [OUTPUT_DIR]/out.requests.txt
        -
        -=== Rejecting mixed selectors before restarting the pipeline
        ->>> musterr [CLI] pipelines run --refresh table1,pipeline_file.py --restart --no-wait
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: cannot mix table names and source files in --refresh; use only table names or only source files
        -
        ->>> musterr [CLI] pipelines run --full-refresh table2,pipeline_file.py --restart --no-wait
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: cannot mix table names and source files in --full-refresh; use only table names or only source files
        -
        ->>> test ! -e [OUTPUT_DIR]/out.requests.txt
        +Exit code: 1
        
FAIL

latest (v1.22.1): same output as main (e9687c1)