Move the UUID extension dtype into encodings/uuid - #9588
Conversation
`vortex.uuid` was split across `vortex-array` (the extension dtype and its vtable) and `vortex-arrow` (the Arrow canonical `arrow.uuid` import/export plugins), and both crates registered it unconditionally in their default sessions. Pull both halves into a new `vortex-uuid` crate under `encodings/`, following the same shape as `vortex-parquet-variant`: the dtype, its Arrow plugins, and an `initialize` that registers all of them with a session. The default `VortexSession` calls `vortex_uuid::initialize`, so sessions built through the `vortex` facade are unchanged. Sessions built from `vortex_array::array_session()` alone no longer resolve `vortex.uuid` and must initialize the crate themselves. Supporting changes: - `vortex_arrow::nulls` and `vortex_arrow::has_valid_extension_type` are now public, since the UUID Arrow plugins live outside the crate. - `ArrowSession` derives `Default` now that it pre-registers no plugins. - The UUID-based tests in `vortex-arrow` (which used UUID as their exemplar Arrow extension type) move to `vortex-uuid` alongside the plugins. - Downstream users import from `vortex::encodings::uuid` / `vortex_uuid`. Signed-off-by: Joseph Isaacs <joe.isaacs@live.co.uk>
Assert that `VortexSession::default()` resolves `vortex.uuid` from the dtype registry and round-trips an Arrow `arrow.uuid` column, so the registration that moved out of `DTypeSession::default` and `vortex_arrow::initialize` cannot be silently dropped from the facade. Signed-off-by: Joseph Isaacs <joe.isaacs@live.co.uk>
Merging this PR will degrade performance by 12.61%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | canonicalize_sparse_list[(512, 7, 4)] |
571.7 µs | 662.3 µs | -13.67% |
| ❌ | Simulation | canonicalize_sparse_list[(1024, 17, 8)] |
508.7 µs | 584.2 µs | -12.93% |
| ❌ | Simulation | compress_fsst[(500, 64, 8)] |
530.5 µs | 597.5 µs | -11.21% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/lucid-feynman-58mu0n (f1e6c20) with develop (e4b3421)
Footnotes
-
54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.986x ➖, 1↑ 1↓)
No file size changes detected. |
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (0.990x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.016x ➖, 1↑ 3↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.008x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.029x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.982x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 2↓)
duckdb / vortex-compact / ns (0.991x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.018x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.061x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.983x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.001x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
File Size Changes (200 files changed, +0.0% overall, 99↑ 101↓)
Totals:
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.013x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 0↓)
File Size Changes (10 files changed, -63.8% overall, 0↑ 10↓)
Totals:
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.006x ➖, 2↑ 2↓)
duckdb / vortex-compact / ns (1.013x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.997x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 2↑ 3↓)
duckdb / vortex-compact / ns (0.998x ➖, 2↑ 2↓)
duckdb / parquet / ns (1.007x ➖, 3↑ 8↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.009x ➖, 1↑ 2↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.964x ➖, 1↑ 2↓)
datafusion / vortex-compact / ns (1.168x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.035x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.973x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.034x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.537x ❌, 0↑ 4↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.997x ➖, 1↑ 2↓)
duckdb / parquet / ns (0.992x ➖, 1↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.032x ➖, 0↑ 2↓)
datafusion / vortex-compact / ns (1.042x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.032x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (1.071x ➖, 1↑ 4↓)
duckdb / vortex-compact / ns (1.035x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.029x ➖, 0↑ 0↓)
|
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (1.044x ➖, 0↑ 1↓)
random-access / parquet / ns (0.995x ➖, 0↑ 0↓)
random-access / lance / ns (0.784x ✅, 1↑ 0↓)
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.044x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.037x ➖, 0↑ 3↓)
duckdb / vortex-compact / ns (1.039x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.031x ➖, 0↑ 0↓)
|
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.977x ➖, 1↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.981x ➖, 1↑ 0↓)
vortex / parquet / ns (1.005x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
|
vortex.uuidwas split acrossvortex-array(the extension dtype and its vtable) andvortex-arrow(the Arrow canonicalarrow.uuidimport/export plugins), and both crates registered it unconditionally in their default sessions. Pull both halves into a newvortex-uuidcrate underencodings/, following the same shape asvortex-parquet-variant: the dtype, its Arrow plugins, and aninitializethat registers all of them with a session.The default
VortexSessioncallsvortex_uuid::initialize, so sessions built through thevortexfacade are unchanged. Sessions built fromvortex_array::array_session()alone no longer resolvevortex.uuidand must initialize the crate themselves.Supporting changes:
vortex_arrow::nullsandvortex_arrow::has_valid_extension_typeare now public, since the UUID Arrow plugins live outside the crate.ArrowSessionderivesDefaultnow that it pre-registers no plugins.vortex-arrow(which used UUID as their exemplar Arrow extension type) move tovortex-uuidalongside the plugins.vortex::encodings::uuid/vortex_uuid.break
Only a import change.