Repository navigation
fix(appkit): fall back to inline results when a Cloud Fetch download is refused - #722
Closed
pffigueiredo wants to merge 3 commits into
Closed
pffigueiredo wants to merge 3 commits into
pffigueiredo wants to merge 3 commits into
Conversation
Charts on the Arrow path fail with HTTP 500 when workspace storage refuses the app server's Cloud Fetch download, and hang until the route's 120 s first-byte timer fires when storage never answers (LA-2169). Add tests for the behavior the fix needs, all failing against v0.91.0: - Processor: link headers are sent, a 400 is not retried, a 403 re-mints the link once, a wait for headers is bounded and counts as a denial, every download failure carries EXTERNAL_LINK_DOWNLOAD_FAILED. - Delivery: a failed first chunk is replaced by an inline JSON_ARRAY run rebuilt as Arrow, and a result too large for inline is never stranded. - Route: over real HTTP against a loopback storage host, charts render, a definite refusal is remembered for ten minutes, and errors say why. - Rebuild: Arrow rebuilt from positional rows matches the native schema and values for every scalar type, including nulls. Signed-off-by: Pedro Figueiredo <pedro.figueiredo@databricks.com>
…is refused A chart on the Arrow path asks for ARROW_STREAM, which a standard SQL warehouse serves only as EXTERNAL_LINKS. When workspace storage refused the app server's download, the route made three attempts and returned HTTP 500 with "Query execution failed". When storage never answered, the route's 120 s first-byte timer returned a 503 that blamed the warehouse (LA-2169). Processor - Send the link's httpHeaders with every GET, and the new link's headers after a re-mint. - Wait at most responseTimeoutMs (10 s) for response headers only. A timeout is not retried. Body reads keep the per-read idle timeout, so a healthy chunk may stream for longer. - A 403 re-mints the link once and retries at once. Any other 4xx except 408 and 429 stops. Network errors and 5xx keep the 1 s and 2 s backoff. - Every download failure, before or after the headers, carries errorCode EXTERNAL_LINK_DOWNLOAD_FAILED and the last HTTP status. Delivery - deliverArrowBytesWithFallback pulls the first chunk before the route sends headers. On a failure before the first byte it runs the query as INLINE + JSON_ARRAY and rebuilds the Arrow IPC stream from the positional rows and manifest columns, typed by parseDatabricksType. - Only a 400 or 403 after the re-mint is remembered as "links-denied", for ten minutes. A timeout, network error, or body failure falls back once and is forgotten. - A result the inline run cannot hold (manifest byte count over 25 MiB, or a multi-chunk inline response) is never served truncated. After an inconclusive failure Cloud Fetch gets one patient pass. After a refusal the error is EXTERNAL_LINKS_DENIED and says why. - Any other inline failure is rethrown unchanged. A rebuild failure is ARROW_REBUILD_FAILED. Connector - executeStatement accepts rawRows to keep dataArray and carry the manifest columns on the result. EXTERNAL_LINKS results now carry totalByteCount. The inline fallback run is cached like the inline Arrow attempt, under its own key. @databricks/appkit-ui is unchanged. Signed-off-by: Pedro Figueiredo <pedro.figueiredo@databricks.com>
Builder pods run with internet off and no SEG, so a download straight to cloud storage hangs instead of refusing. Remember that as links-denied once the inline fallback has served the result, so later charts skip the wait. Signed-off-by: Pedro Figueiredo <pedro.figueiredo@databricks.com>
Contributor
Author
|
Moved to #723, an in-repo branch, so CI can run: fork PRs don't get the OIDC token the JFrog npm setup needs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A chart on the Arrow path fails with HTTP 500 "Query execution failed" when workspace storage refuses the app server's Cloud Fetch download. This happens, for example, when the storage account's firewall doesn't allow the serverless network the app runs on (Jira LA-2169). AppKit makes three attempts and has no fallback. On pods with internet off, a blocked host can also hang until the route's 120 s timer returns a 503 that blames the warehouse. This PR makes the chart render in both cases. Healthy workspaces get the same bytes and the same executions as before.
What changed
httpHeaders. It waits at most 10 s for response headers and keeps the per-read idle timeout for the body. On a 403 it re-mints the link once with no wait. On any other 4xx except 408 and 429 it stops. Every failure carriesEXTERNAL_LINK_DOWNLOAD_FAILED.deliverArrowBytesWithFallbackwrapsdeliverArrowBytes. If the download fails before the first byte, the wrapper runs the query asINLINE+JSON_ARRAYand rebuilds the same Arrow IPC stream from the raw rows andparseDatabricksType. The route and@databricks/appkit-uidon't change."links-denied"for the warehouse for 10 minutes, so later queries skip the doomed download. A missing answer is recorded only after the inline result has been served.Scope
EXTERNAL_LINKS_DENIEDand a message that names the storage download.UTCinstead of the warehouse session time zone. The instants are identical, and@databricks/appkit-uidoesn't read the label.Tradeoffs
Blast radius
The change is limited to the
ARROW_STREAMroute of the analytics plugin and the chunk downloader. The JSON path, metric views, and the browser package don't change. Every new branch runs only after a download has failed. Before this PR, that path returned a 500.Verification
vitest run packages/appkit: 4301 passed, 1 skipped.pnpm --filter @databricks/appkit typecheckandpnpm checkare clean.template/and viewed in headless Chrome. A preload answered the storage downloads with 403, 400, or no response. With 0.89.0, the twoformat="arrow_stream"charts showed "Error loading chart: Query execution failed", and a hung download returned 503 after 120 s. With this branch, every chart rendered: 2.8 s on a 403 and 12.9 s on a hang, with 7 ms on the next load in both cases. Healthy storage still used Cloud Fetch. Across 17 columns of real data, the rebuild matched native Arrow in 14. The other 3 are the documented limits above.