Skip to content

Http REFUSED_STREAM when uploading bundle files/artifacts leaves bundle in a broken state #6472

New issue
New issue
Closed
Closed
Http REFUSED_STREAM when uploading bundle files/artifacts leaves bundle in a broken state#6472
Assignees
andrewnester
Labels
BugSomething isn't workingDABsDABs related issues

Description

@jmahlik
jmahlik
opened on Sep 1, 2026

Describe the issue

When uploading large amounts of wheels, the cli errors sporadically with http2 REFUSED_STREAM errors. When this happens the artifacts have been uploaded but the bundle files have not been updated to reference them. The old artifacts are deleted, so the bundle is left in a broken state.

It's difficult to reproduce since it's a transient http2 error. About one in 200-500 deploys hits this issue. It always appears on a GET request to /api/2.0/workspace/get-status. It may be an sdk issue with the retries on databricks.WorkspaceClient.

It seems like the deployment should be rolled back if it fails mid-deploy. Or the requests retried more gracefully. Maybe a configuration option for number of retries could be implemented?

Here's the edited output from a failed deployment.

Building project...
Building python_deps...

...

Lots of artifact uploads
Uploading ...-py3-none-any.whl...
Uploading ...-py3-none-any.whl...
Uploading ...-py3-none-any.whl...
Uploading ...-py3-none-any.whl...
Uploading ...-py3-none-any.whl...
Uploading ...-py3-none-any.whl...

...

Uploading bundle files to /Workspace/<CENSORED>.bundle/files...
Error: Get "https://<CENSORED>.cloud.databricks.com/api/2.0/workspace/get-status?path=<CENSORED>.bundle%2Ffiles": stream error: stream ID 287; REFUSED_STREAM; received from peer

Configuration

Put a lot of wheels in the directory. An example is a requirements file with 100+ dependencies. With the build looking something like this.

  artifacts:
  project:
    type: whl
    build: |
      uv build . --wheel --out-dir dist
    path: .
    files:
      - source: ./dist/*.whl
  python_deps:
    type: whl
    path: .
    # TODO: uncomment if deps need updated, takes forever to plan
    build: |
      python -m pip download \
        -r requirements.txt \
        --no-deps \
        --only-binary :all: \
        --platform manylinux_2_28_x86_64 \
        --platform manylinux_2_27_x86_64 \
        --platform manylinux_2_17_x86_64 \
        --platform manylinux2014_x86_64 \
        --python-version 3.12 \
        --implementation cp \
        --abi none \
        --abi abi3 \
        --abi "cp312" \
        -d dist_deps
    files:
      - source: ./dist_deps/*

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

  1. Run databricks bundle deploy ...
  2. Get a stream error
  3. The bundle is now out of sync with the artifacts

Expected Behavior

Roll back the deploy or retry the request.

Actual Behavior

Prod is broken, need to redeploy to fix.

OS and CLI version

Linux, tested on 0.298.0 and latest 1.14.1

Is this a regression?

No

Debug Logs

This happens in CI, haven't been able to reproduce it with debug logs.

Activity

  1. jmahlik
    added
    DABsDABs related issues
    on Sep 1, 2026

  2. andrewnester commented on Sep 2, 2026

    @andrewnester
    andrewnester
    on Sep 2, 2026
    Contributor

    We have a new feature called immutable folder which allows bundle to deploy to unique location and change the resources referencing the artifacts atomically, this would prevent this issue. Please give it a try.

    #5254

  3. andrewnester
    added
    BugSomething isn't working
    on Sep 2, 2026
  4. andrewnester
    self-assigned this
    on Sep 2, 2026

  5. jmahlik commented on Sep 2, 2026

    @jmahlik
    jmahlik
    on Sep 2, 2026
    Author

    We have a new feature called immutable folder which allows bundle to deploy to unique location and change the resources referencing the artifacts atomically, this would prevent this issue. Please give it a try.

    #5254

    It does sound like that would solve the problem. I just gave it a shot. Unfortunately, the bundle is too large in this case. Looks like the size limit is 100MB but the bundle is about 250MB.

    databricks bundle deploy -t user

Building project...
Building python_deps...
Uploading immutable bundle snapshot...
Error: snapshot upload: Server received a request which exceeds maximum allowed content length. RequestSize(bytes): 264767245, Limit(bytes): 104857600 (400 BAD_REQUEST)

Endpoint: POST https://CENSORED/api/2.0/repos/snapshots
HTTP Status: 400 Bad Request
API error_code: BAD_REQUEST
API message: Server received a request which exceeds maximum allowed content length. RequestSize(bytes): 264767245, Limit(bytes): 104857600

  6. pietern commented on Oct 9, 2026

    @pietern
    pietern
    on Oct 9, 2026
    Contributor

    @jmahlik A couple of questions:

    1. Roughly how much time does it take to complete all the artifact uploads?
    2. Do you have a HTTP/2 proxy sitting between this runtime environment and Databricks?

    Our SDK (via Go's stdlib) already retries on REFUSED_STREAM but in this case it seems like the server continues to return the same error, exhausting Go's own retries, and then failing. We're looking into adding another retry one level up where we can force the underlying connection to be recreated. Answers to the above questions would be helpful in dialing in on the root cause. Thanks!

  7. jmahlik commented on Oct 12, 2026

    @jmahlik
    jmahlik
    on Oct 12, 2026
    Author
    1. Roughly how much time does it take to complete all the artifact uploads?

    About 30 seconds from artifact upload until uploads complete.

    1. Do you have a HTTP/2 proxy sitting between this runtime environment and Databricks?

    No for the cases where we see this failing. I've tested behind a proxy and works the same. Hard to reproduce the issue because it's sprious.

  8. janniklasrose commented on Oct 14, 2026

    @janniklasrose
    janniklasrose
    on Oct 14, 2026
    Member

    Hi @jmahlik, an update on our side:

    This week's release will include #6149 which changes the underlying file import API and hopefully makes file uploads more stable.

    #7014 is also in flight and adds retries on bundle deploy.

    #7044 should fix the /get-status errors

  9. janniklasrose
    added 3 commits that reference this issue on Oct 14, 2026

    Retry workspace get-status on transient failures during sync

    7e6acb1

    Retry workspace get-status on transient failures during sync

    150235f

    retry: scope get-status transport retry to REFUSED_STREAM

    a17e854
  10. dhakad22klx
    added a commit that references this issue on Oct 18, 2026

    Retry workspace get-status on transient failures during sync (databri…

    a689393

  11. andrewnester commented on Oct 19, 2026

    @andrewnester
    andrewnester
    on Oct 19, 2026
    Contributor

    The fix was released in the latest CLI version, please give it a try and if the issue persists feel free to reopen the issue

  12. andrewnester
    closed this as completedon Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BugSomething isn't workingDABsDABs related issues

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions