Resolve object-store URLs in MultiFileDataSource via the cloud registry - #9558
Open
balicat wants to merge 1 commit into
Open
Resolve object-store URLs in MultiFileDataSource via the cloud registry#9558balicat wants to merge 1 commit into
balicat wants to merge 1 commit into
Conversation
Contributor
|
Should we do this at a lower level (can object store not do this?) |
Author
|
Agreed — that's the better home for it. Reworking to resolve URLs in core instead, in the |
MultiFileDataSource::with_glob documents S3/GCS support through an explicit FileSystemRef, but a bare URL with fs: None was treated as a local path - cwd-joined and mangled - so object-store URLs only worked for callers that carry their own store wiring (the Python and Java bindings each do). Anything on the C FFI hit a local-filesystem error. Resolve them in core instead, so every consumer gets the same behavior: with_glob leaves object-store URLs untouched, and build() routes a (url, None) source through the shared vortex-cloud Registry - the same env-based resolution as the Python binding's resolve_store - wrapping the store in Compat and an ObjectStoreFileSystem. Parse failures, file URLs, and single-character schemes (Windows drive paths) keep resolving locally, and the behavior is gated behind a new vortex-file feature object_store_registry, threaded through the vortex facade and enabled by vortex-ffi. Tests at both levels assert that an unconfigured az:// URL fails in the registry's store builder rather than as an unmatched local glob. Signed-off-by: David Linton <e.david.linton@gmail.com>
balicat
force-pushed
the
ffi-object-store-urls
branch
from
August 24, 2026 16:03
f8f244e to
9ff11a0
Compare
Author
|
Reworked as discussed — resolution now happens in core: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MultiFileDataSource::with_globdocuments S3/GCS support through anexplicit
FileSystemRef, but a bare URL withfs: Noneis treated as alocal path — cwd-joined and mangled — so object-store URLs only work for
callers that carry their own store wiring (the Python and Java bindings
each do). Anything on the C FFI — C#, C, Swift bindings — hits a
local-filesystem error on URL paths.
Reworked per review (originally this lived in the FFI): URLs are now
resolved in core, through the existing
vortex-cloudRegistry, so everyconsumer gets the same behavior and the FFI change reduces to enabling a
feature.
Changes
with_globleaves object-store URLs untouched (the cwd-join forrelative paths would corrupt them —
s3://…counts as a relative pathon Windows).
build()routes each(url, None)source through a sharedvortex_cloud::Registry— the same env-based resolution as the Pythonbinding's
resolve_store— wrapping the store inCompat(for theHTTP client's tokio requirement) and an
ObjectStoreFileSystem.file://URLs, and single-character schemes (Windowsdrive paths like
C:/data.vortex) keep resolving locally, exactly asbefore.
vortex-filefeatureobject_store_registry(bringing in
vortex-cloud/registry), threaded through the existingfacade feature of the same name, and enabled by
vortex-ffi— so theFFI's own diff is one feature flag plus a test.
Tests at both levels (
vortex-fileandvortex-ffi) assert that anunconfigured
az://URL fails inside the registry's store builder ratherthan as an unmatched local glob — a deterministic, network-free probe
that the registry path was taken.
Validated end-to-end from a .NET binding over the C FFI against
Cloudflare R2 on the public internet: a single-series predicate scan of
an 18 MB / 4.5 M-row file completes in ~2 s cold via both
https://andsigned
s3://— one HEAD, one 64 KB footer read, and two parallel rangereads, byte-identical to the ranges the same plan issues against a local
file; the local-file path is regression-identical to a stock build.