Skip to content

Resolve object-store URLs in MultiFileDataSource via the cloud registry - #9558

Open
balicat wants to merge 1 commit into
vortex-data:developfrom
balicat:ffi-object-store-urls
Open

Resolve object-store URLs in MultiFileDataSource via the cloud registry#9558
balicat wants to merge 1 commit into
vortex-data:developfrom
balicat:ffi-object-store-urls

Conversation

@balicat

@balicat balicat commented Aug 23, 2026

Copy link
Copy Markdown

Summary

MultiFileDataSource::with_glob documents S3/GCS support through an
explicit FileSystemRef, but a bare URL with fs: None is treated as a
local path — cwd-joined and mangled — so object-store URLs only work for
callers that carry their own store wiring (the Python and Java bindings
each do). Anything on the C FFI — C#, C, Swift bindings — hits a
local-filesystem error on URL paths.

Reworked per review (originally this lived in the FFI): URLs are now
resolved in core, through the existing vortex-cloud Registry, so every
consumer gets the same behavior and the FFI change reduces to enabling a
feature.

Changes

  • with_glob leaves object-store URLs untouched (the cwd-join for
    relative paths would corrupt them — s3://… counts as a relative path
    on Windows).
  • build() routes each (url, None) source through a shared
    vortex_cloud::Registry — the same env-based resolution as the Python
    binding's resolve_store — wrapping the store in Compat (for the
    HTTP client's tokio requirement) and an ObjectStoreFileSystem.
  • Parse failures, file:// URLs, and single-character schemes (Windows
    drive paths like C:/data.vortex) keep resolving locally, exactly as
    before.
  • Gated behind a new vortex-file feature object_store_registry
    (bringing in vortex-cloud/registry), threaded through the existing
    facade feature of the same name, and enabled by vortex-ffi — so the
    FFI's own diff is one feature flag plus a test.
  • With the feature off, behavior is byte-for-byte unchanged.

Tests at both levels (vortex-file and vortex-ffi) assert that an
unconfigured az:// URL fails inside the registry's store builder rather
than as an unmatched local glob — a deterministic, network-free probe
that the registry path was taken.

Validated end-to-end from a .NET binding over the C FFI against
Cloudflare R2 on the public internet: a single-series predicate scan of
an 18 MB / 4.5 M-row file completes in ~2 s cold via both https:// and
signed s3:// — one HEAD, one 64 KB footer read, and two parallel range
reads, byte-identical to the ranges the same plan issues against a local
file; the local-file path is regression-identical to a stock build.

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Should we do this at a lower level (can object store not do this?)

@balicat

balicat commented Aug 24, 2026

Copy link
Copy Markdown
Author

Agreed — that's the better home for it. Reworking to resolve URLs in core instead, in the fs: None path of MultiFileDataSource next to where the local filesystem is auto-created, so every consumer gets it and the FFI change all but disappears. Will update this PR.

MultiFileDataSource::with_glob documents S3/GCS support through an
explicit FileSystemRef, but a bare URL with fs: None was treated as a
local path - cwd-joined and mangled - so object-store URLs only worked
for callers that carry their own store wiring (the Python and Java
bindings each do). Anything on the C FFI hit a local-filesystem error.

Resolve them in core instead, so every consumer gets the same behavior:
with_glob leaves object-store URLs untouched, and build() routes a
(url, None) source through the shared vortex-cloud Registry - the same
env-based resolution as the Python binding's resolve_store - wrapping
the store in Compat and an ObjectStoreFileSystem. Parse failures, file
URLs, and single-character schemes (Windows drive paths) keep resolving
locally, and the behavior is gated behind a new vortex-file feature
object_store_registry, threaded through the vortex facade and enabled
by vortex-ffi.

Tests at both levels assert that an unconfigured az:// URL fails in the
registry's store builder rather than as an unmatched local glob.

Signed-off-by: David Linton <e.david.linton@gmail.com>
@balicat
balicat force-pushed the ffi-object-store-urls branch from f8f244e to 9ff11a0 Compare August 24, 2026 16:03
@balicat balicat changed the title FFI: route object-store URLs through env-configured filesystems Resolve object-store URLs in MultiFileDataSource via the cloud registry Aug 24, 2026
@balicat

balicat commented Aug 24, 2026

Copy link
Copy Markdown
Author

Reworked as discussed — resolution now happens in core: with_glob keeps URLs intact and build() routes (url, None) sources through the existing vortex-cloud::Registry, gated behind a vortex-file/object_store_registry feature threaded through the facade. The FFI diff is down to a feature flag plus a test. Title and description updated to match.

@joseph-isaacs
joseph-isaacs requested a review from myrrc August 24, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants