Repository navigation
Add the databricks pipelines test command - #7297
tharunbhuk-db wants to merge 16 commits into
Conversation
Submits a test-only pipeline update with optional test selections and pytest arguments. Relative selections resolve to deployed workspace paths through the bundle sync root, or against the pipeline's root_path when the pipeline is identified with --pipeline-id. Only -k and -m are accepted in --pytest-arg. Co-authored-by: Isaac <no-reply@databricks.com>
Cover each rejected pytest option with a unit test, and add an acceptance case for `--pytest-arg VALUE`, which keeps commas and spaces. Co-authored-by: Isaac <no-reply@databricks.com>
The service names a test case relative to pytest's rootdir: the deepest folder that holds the pipeline's root_path and every test source, or, without a root_path, the folder of every test source. A test ID copied from the results now runs that test again in any project layout. A relative selection that exists locally still maps to its deployed path. Any other relative selection resolves against that folder, which the command reads from the deployed pipeline. With --pipeline-id, all relative selections resolve against it. Co-authored-by: Isaac <no-reply@databricks.com>
…nd preview A local test path mapped to workspace.file_path in every sync mode. A source-linked deployment runs the source files, so a local path now maps under the sync root, as TranslatePaths maps resource paths. An immutable deployment runs a snapshot whose path the command does not know, so a local path is an error there that points to test IDs and workspace paths, which still work. The folder of test IDs is read once, and only when a selection is a relative path. Paths in errors use forward slashes. The Short text starts with *Preview*, like other preview commands. Tests: the option list covers each rejected form once, and a folder test source without root_path has its parent as the folder of test IDs, as the service computes it. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
Commit: Regression test reportSelected 6 changed tests (limit=20, 0 not selected) 6 tests: regression=6 unreleased=0 coverage=0 failing=0
TestAccept/pipelines/databricks-cli-help: branch ✅ | main (44f2eda) ❌ | latest (v1.23.0) ❌main (44f2eda): latest (v1.23.0): same output as main (44f2eda) TestAccept/pipelines/test/arguments: branch ✅ | main (44f2eda) ❌ | latest (v1.23.0) ❌main (44f2eda): latest (v1.23.0): TestAccept/pipelines/test/immutable: branch ✅ | main (44f2eda) ❌ | latest (v1.23.0) ❌main (44f2eda): latest (v1.23.0): same output as main (44f2eda) TestAccept/pipelines/test/pipeline-id: branch ✅ | main (44f2eda) ❌ | latest (v1.23.0) ❌main (44f2eda): latest (v1.23.0): same output as main (44f2eda) TestAccept/pipelines/test/selection: branch ✅ | main (44f2eda) ❌ | latest (v1.23.0) ❌main (44f2eda): latest (v1.23.0): same output as main (44f2eda) TestAccept/pipelines/test/submit-error: branch ✅ | main (44f2eda) ❌ | latest (v1.23.0) ❌main (44f2eda): latest (v1.23.0): |
Integration test reportCommit: 66e7f7e
Top 17 slowest tests (at least 2 minutes):
|
Error messages and edge cases stay in unit tests; wiring, request bodies and command output stay in acceptance tests. Examples avoid /Workspace/Shared.
…hile it is experimental
…orkspace paths Setting MSYS_NO_PATHCONV for the whole folder also stopped Git Bash from converting the paths of the Python helpers, so print_requests.py failed to start on Windows. Co-authored-by: Isaac <no-reply@databricks.com>
| return nil, fmt.Errorf("test selection %q has no path", selection) | ||
| } | ||
| if strings.HasPrefix(p, "/") { | ||
| p = path.Clean(p) |
There was a problem hiding this comment.
path.Clean quietly turns /Workspace/x/../y into a different path than the user typed, and path.Join in rootPathSelection does the same for relative selections. Could we pass selections through as typed instead? The service already rejects .. and wildcards with its own error, so dropping the normalization is less code and keeps the error about what the user actually wrote. The test at test_test.go:53 would then expect the input unchanged.
| err = apiClient.Do( | ||
| ctx, | ||
| http.MethodPost, | ||
| fmt.Sprintf("/api/2.0/pipelines/%s/updates", pipelineID), |
There was a problem hiding this comment.
--pipeline-id goes into the URL path unescaped. Could we validate the ID format, or url.PathEscape it?
| // resolveTestTargetByID identifies the pipeline by --pipeline-id; all positional arguments are selections. | ||
| func resolveTestTargetByID(cmd *cobra.Command, args []string, pipelineID string) (testTarget, error) { | ||
| if cmd.Flag("target").Value.String() == "" { | ||
| cmd.SetContext(root.SkipLoadBundle(cmd.Context())) |
There was a problem hiding this comment.
Question: with --pipeline-id inside a project, the bundle isn't loaded, so the bundle's host and profile are ignored, and the command can talk to a different workspace than pipelines run would. Is that intended?
| cmd := &cobra.Command{ | ||
| Use: "test [PIPELINE_KEY] [TEST_SELECTION...] [flags] [--pytest-arg=ARG ...]", | ||
| Short: "*Preview* Run the tests of a pipeline", | ||
| Long: `Run the tests of a pipeline in a test-only update. |
There was a problem hiding this comment.
Public text:
- Short says "Preview" while Long and the gate say "experimental". Could we pick one label and name the env var in the help?
- Other strings:
- "outside a project" → "Run this command from a bundle directory, or pass --pipeline-id."
- "synchronized bundle files" → "%s is not in the files synced from %s; pass its /Workspace path."
- The immutable error → "Relative test paths need synced local files, but workspace.sync_mode is immutable; pass the /Workspace path."
| option, value := arg[:2], strings.TrimPrefix(arg[2:], "=") | ||
| return fmt.Errorf("pytest argument %q must be split: pass --pytest-arg=%s --pytest-arg=%s", arg, option, shellquote.BashArg(value)) | ||
| case !strings.HasPrefix(arg, "-"): | ||
| return fmt.Errorf("unsupported pytest argument %q: pass test paths and test IDs as TEST_SELECTION arguments", arg) |
There was a problem hiding this comment.
Nit: --pytest-arg="" lands here and reports unsupported pytest argument "": pass test paths..., which is misleading for an empty value. Since this flag's rules are the CLI's own, an explicit empty-value message would help. Also nit: the Use string puts selections before flags, while sibling commands use run [flags] [KEY].
| ctx := cmd.Context() | ||
|
|
||
| if _, err := resources.Lookup(b, key, isPipeline); err != nil { | ||
| return testTarget{}, fmt.Errorf("%w; the first argument must be a pipeline KEY when test selections are given", err) |
There was a problem hiding this comment.
In a one-pipeline project, pipelines test tests/test_orders.py takes the path as the KEY. The rule is fine, but the error could name the pipeline and show the working command, so an agent can self-correct in one step: "tests/test_orders.py" is not a pipeline in this bundle (pipelines: my_pipeline). Usage: databricks pipelines test my_pipeline tests/test_orders.py.
| and -m are supported, each followed by its value in the next --pytest-arg, for | ||
| example: | ||
|
|
||
| databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'`, |
There was a problem hiding this comment.
Agents and CI scripts learn the command from --help, which has one example. Could the Long text cover the common flows and the outcome contract?
- Examples: run all tests of a pipeline; run one file; re-run one failing test by ID; CI with
--junit-xml; resume after--timeout. - A short exit-code paragraph: when it exits 0, and that failures, incomplete runs and errors exit 1.
- Both env vars needed while the feature is experimental.
🥞 Stacked PR
Use this link to review incremental changes.
Changes
databricks pipelines test [PIPELINE_KEY] [TEST_SELECTION...], which runs a pipeline's tests in a test-only update. The help marks it*Preview*. Without a selection, all of the pipeline's tests run.DATABRICKS_CLI_PIPELINES_TEST=true; otherwise it exits before any API call, with a message that names the variable. The service also gates test runs with its own flag.tests/test_orders.py::test_backfill. The API accepts only absolute workspace paths, so the CLI converts relative paths: in a bundle, a relative path is a local path from the working directory, must exist, and maps to its deployed path; with--pipeline-id, it is relative to the pipeline'sroot_path.--pytest-argpasses-kand-mto pytest, each followed by its value in the next--pytest-arg, as the API requires. The value cannot be empty or start with-. Other pytest options are rejected with guidance.--pipeline-idnames the pipeline.Why
Pipeline tests run only from the editor today. CI jobs and agents need a command that runs them.
Tests
acceptance/pipelines/test/{arguments,selection,pipeline-id,immutable,submit-error}, including the command with the variable unset.This pull request and its description were written by Isaac.