PyArrow: Keep the storage account out of ADLS paths in parse_location - #3884
Open
krishnakaanchan-png wants to merge 1 commit into
Open
PyArrow: Keep the storage account out of ADLS paths in parse_location#3884krishnakaanchan-png wants to merge 1 commit into
parse_location#3884krishnakaanchan-png wants to merge 1 commit into
Conversation
For Azure the netloc is <container>@<account>.<host>, so building the path as netloc + path put the account inside the path and PyArrow then read the whole first segment as the container name. Return only the container instead, which matches what PyArrow's own from_uri produces for the same location.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
parse_locationbuilds the path asf"{uri.netloc}{uri.path}". That is correct for S3, where the netloc is the bucket. For Azure the netloc iscontainer@account.dfs.core.windows.net, so the account ends up inside the path and PyArrow reads that whole first segment as the container name. EveryPyArrowFileIOcall on a canonical ABFS location is therefore pointed at a container that cannot exist, since container names allow only lowercase letters, numbers and hyphens.PyArrow itself handles these locations correctly.
FileSystem.from_uri("abfss://myfs@myacct.dfs.core.windows.net/wh/d.parquet")returnsmyfs/wh/d.parquet, so the problem is only on our side. After this changeparse_locationreturns the same path that PyArrow does.The full trace and the Azure documentation references are in #2698.
This is the path half only. The other half is that
_initialize_azure_fstakes no netloc, so the account can come only fromadls.account-nameand never from the location itself. I will send that separately, it needs a call on precedence when the property and the location disagree.Related to #2698. Not closing it here, since the account derivation is still pending.
Are these changes tested?
Yes. Two new parametrised tests over
abfs,abfss,wasbandwasbs, one for the account qualified form and one for the container only form.I also added an S3 case to
test_parse_location. That function has to keep serving both shapes and there was nothing pinning the S3 behaviour that the new branch has to preserve.make lintis clean andtests/io/test_pyarrow.pypasses.Are there any user-facing changes?
Yes. Locations of the form
abfs[s]://<container>@<account>.dfs.core.windows.net/<path>now resolve to the correct container underPyArrowFileIO. Earlier they resolved to a container named after the entire netloc. S3, HDFS and the container only Azure form are unchanged.