Skip to content

Fix AWS Spark Python task acceptance test access mode - #7187

Open
shreyas-goenka wants to merge 3 commits into
mainfrom
shreyas-goenka_data/shreyas-goenka/fix-aws-spark-python-task
Open

shreyas-goenka wants to merge 3 commits into
mainfrom
shreyas-goenka_data/shreyas-goenka/fix-aws-spark-python-task

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

The AWS test account enabled admin protection for no-isolation shared clusters on October 22 at 14:43 UTC. The CI principal is a workspace admin, and the test left cluster access mode unspecified. Workspace file uploads and API reads succeed, but this protection prevents the driver from reading the Python file. The original test passed at 12:38 UTC and failed by 19:09 UTC that day; an account audit event confirms the setting enablement between those runs.

Set USER_ISOLATION (standard access mode) in the existing shared cluster template. This restores the task's workspace-file access while retaining the security protection. The final diff is one added line.

The restriction is documented: https://docs.databricks.com/aws/en/admin/account-settings/no-isolation-shared

Validation:

  • A controlled comparison on the same AWS legacy cluster/file confirms admin failure (PermissionError) and non-admin success; both principals can export through the Workspace API.
  • Both affected fixtures pass locally and on AWS under both DMS variants. The Spark test also passes without a .databricks ignore.
  • Temporary clusters, jobs, files, and test permissions were cleaned up. YAML formatting, shell syntax, and git diff whitespace checks pass.
  • Windows, Azure, and GCP were not rerun.

@github-actions github-actions Bot added the DABs DABs related issues label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 87e25b108 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=0 unreleased=0 coverage=2 failing=0

test branch main (accb3df) latest (v1.22.1)
TestAccept/bundle/resources/clusters/deploy/simple/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/clusters/deploy/simple/DMS=true ✅ ✅ ✅

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 87e25b1

Run: 40494297295

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 284 15 7:20
✅​ aws-windows-latest-4core-16gb 286 13 12:19
✅​ azure linux-2core-8gb 283 15 6:51
🔄​ azure-windows-latest-4core-16gb 2 283 13 8:55
✅​ gcp linux-2core-8gb 284 15 7:26
✅​ gcp-windows-latest-4core-16gb 286 13 9:32
Test Name azure-windows-latest-4core-16gb
🔄​ TestFsRmNonEmptyDirectory 🔄​f
🔄​ TestFsRmNonEmptyDirectory/uc-volumes 🔄​f
Top 15 slowest tests (at least 2 minutes):
duration env testname
7:15 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=true
7:00 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
6:44 gcp-windows-latest-4core-16gb TestAccept
6:32 azure-windows-latest-4core-16gb TestAccept
4:58 aws-windows-latest-4core-16gb TestAccept
3:52 gcp linux-2core-8gb TestAccept
3:50 aws linux-2core-8gb TestAccept
3:46 azure linux-2core-8gb TestAccept
2:42 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
2:40 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=true
2:28 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
2:26 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=true
2:18 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=true
2:05 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
2:02 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=true

@shreyas-goenka
shreyas-goenka marked this pull request as ready for review October 27, 2026 14:39
@shreyas-goenka
shreyas-goenka requested review from a team as code owners October 27, 2026 14:40

@denik denik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we change the config? it seems like a backend issue. I have ES ticket with jobs team. We can use known failures for now.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants