Skip to content

air: show workspace-supported accelerators in compute help - #6993

Merged
caroline-db merged 3 commits into
mainfrom
air/list-compute-options
Oct 28, 2026
Merged

air: show workspace-supported accelerators in compute help#6993
caroline-db merged 3 commits into
mainfrom
air/list-compute-options

Conversation

@caroline-db

@caroline-db caroline-db commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Changes

  • Fetch workspace compute options from the AI Training endpoint for databricks air run -h config.compute.
  • Display the options returned by the backend, including per-node accelerator counts and preview stages. The CLI does not apply its own accelerator filter.
  • Show an explicit message when the backend returns no options.
  • Bound the lookup to three seconds and fall back to the built-in list when the request is unavailable or fails. Unexpected failures also produce a warning.

Why

Static help can advertise accelerators that cannot be used in the selected workspace. The AI Training backend determines which options are supported by both the workspace and the Jobs AI Runtime task API, keeping that compatibility logic out of the CLI.

The fallback preserves useful help while the endpoint is unavailable or still being rolled out.

Tests

  • ./task test
  • ./task lint
  • ./task checks
  • Verified successful, empty, unavailable, and failed endpoint responses.
  • Verified backend filtering end to end.

@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 8, 2026
@caroline-db
caroline-db force-pushed the air/list-compute-options branch 2 times, most recently from 4464f3f to 6b2cfe5 Compare October 8, 2026 18:37
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 8aacd6e

Run: 40587000474

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 16 5:21
✅​ aws-windows-latest-4core-16gb 283 14 3:24
✅​ azure linux-2core-8gb 280 16 5:15
✅​ azure-windows-latest-4core-16gb 282 14 3:23
✅​ gcp linux-2core-8gb 281 16 5:31
✅​ gcp-windows-latest-4core-16gb 283 14 4:50
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:46 gcp-windows-latest-4core-16gb TestAccept
3:57 gcp linux-2core-8gb TestAccept
3:47 aws linux-2core-8gb TestAccept
3:47 azure linux-2core-8gb TestAccept
3:18 aws-windows-latest-4core-16gb TestAccept
3:17 azure-windows-latest-4core-16gb TestAccept

pardis-beikzadeh-db
pardis-beikzadeh-db reviewed Oct 8, 2026
View reviewed changes
Comment thread cmd/air/runconfig.go Outdated
pardis-beikzadeh-db
pardis-beikzadeh-db approved these changes Oct 9, 2026
View reviewed changes
Comment thread cmd/air/run.go
Comment thread cmd/air/run.go
@caroline-db
caroline-db force-pushed the air/list-compute-options branch 2 times, most recently from 4af9e9a to efefea9 Compare October 21, 2026 18:08
@github-actions

github-actions Bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: 8aacd6e51 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0

test branch main (814bdfe) latest (v1.22.1)
TestAccept/air/config-help ✅ ❌ ❌
TestAccept/air/config-help: branch ✅ | main (814bdfe) ❌ | latest (v1.22.1) ❌

main (814bdfe):

acceptance_test.go:1341: Diff:
        --- air/config-help/output.txt
        +++ /tmp/TestAcceptairconfig-help3126629019/002/output.txt
        @@ -74,10 +74,6 @@
         config.compute
           Which accelerators to run on and how many.
         
        -  Supported accelerator types:
        -    GPU_1xH100 - 1 accelerator per node [Private Preview]
        -    GPU_8xH100 - 8 accelerators per node
        -
           Fields:
             num_accelerators  Total number of GPUs to allocate.
             accelerator_type  Which accelerator to run on, e.g. GPU_1xA10.
        
FAIL

latest (v1.22.1): same output as main (814bdfe)

@caroline-db
caroline-db marked this pull request as ready for review October 21, 2026 19:21
@caroline-db
caroline-db requested review from a team as code owners October 21, 2026 19:21
@caroline-db
caroline-db requested a review from rugpanov October 21, 2026 19:21
@caroline-db
caroline-db force-pushed the air/list-compute-options branch from efefea9 to 8aacd6e Compare October 27, 2026 21:27
@caroline-db
caroline-db added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit 0d6ee1c Oct 28, 2026
28 checks passed
@caroline-db
caroline-db deleted the air/list-compute-options branch October 28, 2026 01:14
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 0d6ee1c

Run: 40614054406

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 1467 1217 120:09
✅​ azure linux-2core-8gb 1316 1269 101:12
✅​ azure-windows-latest-4core-16gb 1241 1294 120:35
🔄​ gcp linux-2core-8gb 2 1301 1274 109:11
✅​ gcp-windows-latest-4core-16gb 1228 1299 122:28
Test Name gcp linux-2core-8gb
🔄​ TestFsCpFileToDirFileNotOverwritten 🔄​f
🔄​ TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_uc-volumes 🔄​f
Top 50 slowest tests (at least 2 minutes):
duration env testname
9:41 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:36 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
8:49 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
8:06 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:57 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:40 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:33 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:33 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:27 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:26 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:26 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:24 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:20 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:08 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:01 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:50 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:48 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:46 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
6:46 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:40 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:35 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:30 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:20 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:20 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
6:15 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:07 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
5:46 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
5:34 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
5:33 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:26 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:22 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
5:21 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:15 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:08 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
5:02 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
4:44 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
4:29 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
4:15 azure linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
4:10 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
4:10 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
4:07 aws linux-2core-8gb TestAccept
3:55 gcp linux-2core-8gb TestAccept
3:53 gcp-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
3:52 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
3:51 azure linux-2core-8gb TestAccept
3:44 gcp-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=
3:39 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
3:35 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
3:34 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
3:32 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maggiewang-db maggiewang-db maggiewang-db approved these changes
@rugpanov rugpanov Awaiting requested review from rugpanov rugpanov is a code owner automatically assigned from databricks/eng-deco-cli
+1 more reviewer
@pardis-beikzadeh-db pardis-beikzadeh-db pardis-beikzadeh-db approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@caroline-db @eng-dev-ecosystem-bot @maggiewang-db @pardis-beikzadeh-db