Skip to content

air: don't announce retries before a run fails - #7270

Merged
maggiewang-db merged 2 commits into
mainfrom
maggiewang/air-drop-retry-notice
Oct 30, 2026
Merged

maggiewang-db merged 2 commits into
mainfrom
maggiewang/air-drop-retry-notice

Conversation

@maggiewang-db

@maggiewang-db maggiewang-db commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Remove the "Failed attempts will be retried up to N times." line that databricks air run --watch prints between "Monitoring run and streaming logs..." and the logs divider. A retry is still reported when it actually happens (Retrying (1 of 3)...), so no information is lost.

Why

max_retries defaults to 3, so every watched run printed this line before any logs streamed. On a healthy run it reads as if an attempt had already failed, which users reported as a papercut.

Tests

  • TestRunWatchStreamsLogs now asserts that a successful watched run never mentions retries. The existing logstream_test.go coverage of Retrying (1 of 3)... on a real retry is unchanged.
  • All cmd/air unit tests and the TestAccept/air/ acceptance tests pass locally; golangci-lint on cmd/air reports no issues.

This pull request and its description were written by Isaac.

maggiewang-db and others added 2 commits October 30, 2026 18:35
`air run --watch` printed "Failed attempts will be retried up to 3 times."
on every run, because max_retries defaults to 3. On a healthy run this reads
as if something already failed. The log stream already prints
"Retrying (n of m)..." when a retry actually happens.

Tracking: CNXT-2481.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 30, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: eebc79e

Run: 41109167411

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 15 5:29
✅​ aws-windows-latest-4core-16gb 283 13 3:42
✅​ azure linux-2core-8gb 280 15 5:15
✅​ azure-windows-latest-4core-16gb 282 13 3:34
✅​ gcp linux-2core-8gb 281 15 5:30
✅​ gcp-windows-latest-4core-16gb 283 13 3:29
Top 6 slowest tests (at least 2 minutes):
duration env testname
3:59 aws linux-2core-8gb TestAccept
3:45 azure linux-2core-8gb TestAccept
3:43 gcp linux-2core-8gb TestAccept
3:35 aws-windows-latest-4core-16gb TestAccept
3:28 azure-windows-latest-4core-16gb TestAccept
3:23 gcp-windows-latest-4core-16gb TestAccept

@maggiewang-db
maggiewang-db marked this pull request as ready for review October 30, 2026 19:27
@maggiewang-db
maggiewang-db requested review from a team as code owners October 30, 2026 19:27
@maggiewang-db
maggiewang-db added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit eb72905 Oct 30, 2026
28 checks passed
@maggiewang-db
maggiewang-db deleted the maggiewang/air-drop-retry-notice branch October 30, 2026 19:49
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: eb72905

Run: 41118487922

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 1504 1226 134:42
🔄​ aws-windows-latest-4core-16gb 3 1426 1251 147:31
🔄​ azure linux-2core-8gb 3 1348 1278 103:57
🔄​ azure-windows-latest-4core-16gb 4 1272 1303 127:12
✅​ gcp linux-2core-8gb 1338 1283 110:02
✅​ gcp-windows-latest-4core-16gb 1263 1308 134:03
9 interesting tests: 9 flaky
Test Name aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb
🔄​ TestAccept 🔄​f 🔄​f ✅​p
🔄​ TestAccept/bundle/invariant/continue_293 ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/invariant/continue_293/DMS=/INPUT_CONFIG=app.yml.tmpl ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=job_table_update_trigger.yml.tmpl/READPLAN=1 ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN= ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN=1 ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=volume_uppercase_name.yml.tmpl/READPLAN=1 ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/resources/postgres_roles/recreate 🔄​f 🙈​s 🙈​s
🔄​ TestAccept/bundle/resources/postgres_roles/recreate/DMS=true 🔄​f
Top 50 slowest tests (at least 2 minutes):
duration env testname
10:46 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
10:35 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:54 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
9:30 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:29 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
9:23 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:11 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:33 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:32 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:31 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:30 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:29 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:28 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:27 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:26 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:24 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:38 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:37 gcp-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
7:33 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
7:27 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
7:25 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:19 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
7:17 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:17 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
7:16 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:10 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:07 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:05 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:54 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:51 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
6:48 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:48 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:39 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:36 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
6:31 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:31 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
6:29 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:24 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:11 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:10 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
6:09 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
6:00 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:51 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:31 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:28 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
5:20 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:14 gcp-windows-latest-4core-16gb TestAccept
5:13 azure linux-2core-8gb TestAccept/bundle/resources/job_runs/failed_run/DMS=
5:10 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
5:08 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=

deco-sdk-tagging Bot added a commit that referenced this pull request Nov 4, 2026
## Release v1.24.0

### Notable Changes

 * A variable used as the value of a secret is now sensitive: referencing it from any other field, such as a job parameter or a secret's `comment`, is an error because the value would be used in plain text. Mark other secret-holding variables with `sensitive: true` to get the same protection. ([#7288](#7288))

### CLI

 * Fix `databricks auth login` and `databricks configure` overwriting the parent profile's host and token when saving a profile with a dotted name such as `team.prod` next to an existing `[team]`. ([#7266](#7266))
 * `databricks auth login` and `databricks auth token` now reject the name of an existing profile after "Create a new profile" instead of logging in to that profile. ([#7196](#7196))
 * `databricks auth login` and `databricks auth token` now reject a host that is not a URL with a hostname, such as `/`, `?o=123` or `https://`, instead of falling back to `DATABRICKS_HOST` or failing later. An empty host after "Enter a host URL manually" in the login profile picker no longer starts discovery login. ([#7195](#7195))
 * Prompts that validate their input, such as the `databricks configure` host prompt, no longer open with a red cross before anything is typed. ([#7314](#7314))
 * `databricks ssh connect` now documents `--environment-version` for choosing the serverless environment version, starts a separate default-named server for each version, and rejects the flag together with `--cluster`. Default connection names also include the environment version in their hash, so the default name when `--environment-version` is not specified changes with this release. Configuration associated with the old connection name (secret scope, SSH keys and known hosts in your local `.databricks` folder) will no longer be used and can be safely removed. ([#7310](#7310))

### AI Runtime

 * Show workspace-supported accelerator types in `databricks air run -h config.compute.accelerator_type`. ([#7275](#7275))
 * `air list` now reports `No active runs found.` for an empty active-only result. ([#7276](#7276))
 * Stop printing a retry notice on every `air run --watch`; retries are still reported when they happen. ([#7270](#7270))

### Bundles

 * Fix `bundle plan` reporting an update to an app's `config.env` values on every run, which also redeployed the app on every `bundle deploy`. The Apps API returns environment variable names only, so `value` and `value_from` are no longer compared against the remote deployment. ([#7326](#7326))
 * Add the `data_quality_monitors` resource for data quality monitors on Unity Catalog schemas and tables (direct engine only). ([#7094](#7094))
 * `bundle deploy` now asks for confirmation before recreating or deleting a Feature View, and `bundle deploy` and `bundle destroy` warn that FeatureSpecs using a Feature View are not checked. ([#7164](#7164))
 * Accept grant `privileges` values that the SDK enum does not list yet: `bundle validate` no longer warns (or errors under `--strict`) on them, and the PyDABs `Privilege` enum accepts unknown strings instead of raising. The JSON schema and PyDABs still advertise the known values for completion. ([#7315](#7315))
 * Add the `materialization_sets` resource, which materializes Feature Store Feature Views to offline and online stores. ([#7164](#7164))
 * Detect notebooks with the same rule as the workspace: a `.py`, `.sql`, `.scala` or `.r` file is a notebook when its first line contains `Databricks notebook source`, for example after a byte order mark. Previously such a file was tracked as a plain file, so removing it locally left the remote notebook behind. ([#7290](#7290))
 * Add `sensitive: true` for bundle variables. A variable used as the value of a secret is sensitive by default. `bundle validate -o json` redacts only sensitive variables instead of every string variable, and a sensitive variable can only be referenced from a sensitive resource field (such as a secret's `value`) or from another sensitive variable. ([#7288](#7288))

### Dependency Updates

 * Bump dependencies with known vulnerabilities. ([#7305](#7305))
 * Bump dependencies with known vulnerabilities. ([#7327](#7327))
 * Update the Go SDK to [v0.191.1](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.191.1). ([#7277](#7277))
 * Update the Go SDK to [v0.193.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.193.0). ([#7303](#7303))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants