Repository navigation
feat(adbc): add Oracle, Db2, GizmoSQL, and Quack driver support - #839
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #839 +/- ##
==========================================
- Coverage 81.40% 80.56% -0.85%
==========================================
Files 527 527
Lines 85274 86216 +942
Branches 15789 16044 +255
==========================================
+ Hits 69421 69456 +35
- Misses 11731 12558 +827
- Partials 4122 4202 +80
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
cofin
changed the title
…ate baselines (#838) ## Summary Prepares the `v0.67.0` release, upgrades project dependencies in `uv.lock`, updates Sphinx documentation configuration, optimizes statement compilation, caching, and driver hot-path execution across sync and async adapters, unifies DuckDB row materialization under the native Python type contract, and hardens the adapter and subsystem performance benchmark gates with refreshed baselines. ## Key Changes ### Core Compilation & Statement Caching (`sqlspec/core/`) - **Zero-Copy Cached AST Sharing (`statement.py`, `compiler.py`)**: - Track shared SQLGlot AST expressions on `ProcessedState` (`expression_is_shared=True`) and expose `ProcessedState.ensure_owned_expression()` and `SQL._peek_expression()`. - Defer AST deep-copying to public `SQL.expression` and `SQL.statement_expression` accessors and mutating compile transforms (`_apply_parameter_casts`), eliminating per-call `.copy()` overhead on compile-cache hits, parameter rebinds, and parse-cache lookups. - **Single-Pass Parameter Processing & Precomputed Coercion (`parameters/_processor.py`, `statement.py`, `utils/dispatch.py`)**: - Precompute and memoize type-coercion fallback items (`type_coercion_fallbacks`) on `ParameterProcessor` and cache fallback-tuple lookups in `type_coercion_dispatcher`. - Short-circuit sequence and mapping coercion when no container elements require conversion (`_NO_MATCH` sentinel). - Share a single rebind `ParameterProcessor` instance across cached `SQL` rebinds and construct cached direct `SQL` instances with a single `ProcessedState` allocation (`SQL._create_cached_direct`). - **Lazy Fast-Path Result Statements (`result/_base.py`)**: - Add `SQLResult.create_cached_select` so cached `SELECT` executions defer `SQL` statement construction until `result.statement` is accessed. ### Driver Execution Pipeline (`sqlspec/driver/`, `sqlspec/config.py`) - **Expanded Fast-Path Eligibility (`_common.py`, `_sync.py`, `_async.py`)**: - Add `_resolve_fast_params(parameters, kwargs)` so parameterless (`execute(sql)`) and scalar-positional (`execute(sql, a, b)`) calls route through `_cached_execution` alongside tuple, list, and dict parameters. - Guard `_cache_statement` against caching statements whose `parsed_expression` is `None` or whose `operation_type` is `"COMMAND"` or `"UNKNOWN"`. - Precompute `CachedQuery.needs_style_remap` at cache-store time and skip `StatementFilter` scans when raw parameters are already normalized. - **Per-Config Shared Statement Caches (`_query_cache.py`, `_common.py`, `config.py`)**: - Introduce thread-safe (`threading.RLock`) `QueryCache` and `StatementCacheHolder` owned by the database config and shared across all sessions created with the config's default `StatementConfig`. - **Fast-Path Observability & Inline Exception Isolation (`_common.py`, `_sync.py`, `_async.py`)**: - Keep the statement-cache fast path enabled when spans, statement observers, or lifecycle hooks are active via `_execute_with_observability_sync` and `_execute_with_observability_async`. - Check `sys.exc_info()[1] is None` inline before entering exception handlers on fast paths, invoking generator-isolated `_run_with_sync_exception_handler` / `_run_with_async_exception_handler` only when an ambient exception is active. - Add DB-API cursor fast-path `SELECT` execution (`_execute_cache_hit_with_cursor`) for cursor-based adapters (`cursor is not self.connection`). - **Single-Pass `execute_many` Preparation (`_common.py`)**: - Reuse batch parameters already coerced during `statement.compile()` via `_many_parameters_already_prepared` and copy parameter rows in `_check_declared_parameters` only when declared defaults are applied (`_row_needs_declared_default`). ### Database Adapters (`sqlspec/adapters/`, `sqlspec/extensions/adk/`) - **DuckDB Native Type Contract & Bulk Insert Caching (`duckdb/core.py`, `duckdb/driver.py`)**: - Unify slow-path (`dispatch_execute`) and cache-hit (`_execute_cache_hit_with_cursor`) row materialization in `_materialize_select_rows` so all `execute()`, `select()`, and `fetch*` calls return DuckDB-native `fetchall()` Python types (`HUGEINT`, `UHUGEINT`, `BIT`, `TIMETZ`, `TIMESTAMP_NS`, `INTERVAL`, `MAP`, `BIGNUM`/`VARINT`, fixed-size arrays, and nested `UUID` always materialize via `fetchall()`; compatible schemas use Arrow when observed row counts reach `ARROW_MIN_ROWS = 64`). - Cache parsed `(target_table, column_sql)` fragments in `StatementCacheHolder.duckdb_bulk_sql` for Arrow bulk `execute_many` inserts and replace `contextlib.suppress` in `resolve_rowcount`. - **Aiosqlite Pool & Fast-Path Cursor Execution (`aiosqlite/pool.py`, `aiosqlite/driver.py`, `sqlite/driver.py`)**: - Add `_try_acquire_idle_connection()` to bypass `asyncio.wait_for` when an idle worker connection is immediately available and skip `end_transaction()` on release when `raw_conn.in_transaction` is `False`. - **PostgreSQL, CockroachDB & ADK Store Typing (`asyncpg`, `psqlpy`, `psycopg`, `cockroach_asyncpg`, `cockroach_psycopg`, `extensions/adk/`)**: - Explicitly type `_POSTGRES_METADATA_DOMAINS` and `_COCKROACH_METADATA_DOMAINS` as `tuple[str, ...]` and define concrete `_drop_tables_sql` / `_drop_memory_table_sql` base methods on `_ADKStoreCommon` and `_ADKMemoryStoreCommon`. ### Release, Benchmarks & Documentation (`Makefile`, `tools/`, `docs/`, `pyproject.toml`, `uv.lock`) - **Version, Lockfile & Docs (`pyproject.toml`, `uv.lock`, `docs/changelog.rst`, `docs/conf.py`)**: - Bump version to `0.67.0`, upgrade locked dependencies, update Sphinx configuration for MyST v5.1+, and update `docs/changelog.rst`. - **Benchmark Harness, Gate & Baselines (`Makefile`, `tools/scripts/`, `tools/perf_baselines/`)**: - Enforce workload metadata parity (`rows`, `pool_size`, `mypyc_compiled`) and add `--mode ratio` (`abs-fallback`) and `--print-run-args` to `bench_compare.py`. - Update `make bench-gate-adapters` to derive run arguments from each baseline JSON and evaluate all requested drivers before exiting. - Add `scalar_param_select`, `no_param_select`, `complex_query`, `session_per_request`, and `observability_select` scenarios across `sqlite`, `aiosqlite`, `duckdb`, `asyncpg`, `psycopg_sync`, `psycopg_async`, `psqlpy`, and `oracle`, plus 9 interleaved stub-driver micro-benchmarks in `bench_subsystems.py`. - Re-capture baselines in `tools/perf_baselines/` (`sqlite.json`, `aiosqlite.json`, `duckdb.json`, `subsystems.json`, `oracle.json`, `asyncpg.json`, `psycopg_sync.json`, `psycopg_async.json`, `psqlpy.json`, `postgres.json`, `adbc.json`) and update `tools/perf_baselines/README.md`. ## Verification ```bash make lint make bench-gate-adapters DRIVERS="sqlite aiosqlite duckdb" uv run --no-sync pytest tests/unit -q uv run --no-sync pytest tests/integration/adapters/sqlite tests/integration/adapters/duckdb -q uv run --no-sync python tools/scripts/mypyc_inventory.py --check uv run --no-sync python tools/scripts/mypyc_smoke.py ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expand SQLSpec's Apache Arrow Database Connectivity (
sqlspec.adapters.adbc) adapter, extension stores (litestar,events, andadk), and documentation to support the pure-Go GizmoData ADBC drivers (adbc-driver-oracle,adbc-driver-db2,adbc-driver-gizmosql,adbc-driver-quack). Also restructure the ADBC adapter documentation (docs/reference/adapters/adbc.rst) and related adapter/extension pages for clarity and quick navigation.Key Changes
ADBC Core Adapter (
sqlspec/adapters/adbc/)core.py,config.py):oracle,oracledb,oracle://), IBM Db2 (db2,ibmdb2,ibm_db,db2://,DATABASE=...;HOSTNAME=...;), native GizmoSQL (gizmosql,gizmo,gizmosql://,gizmo://), and DuckDB Quack (quack,quack://,quack+tls://), as well as internal dialect/driver-manager routing formssql/tsql.adbc_driver_gizmosql.dbapi.connectwhenadbc-driver-gizmosqlis installed and fall back toadbc_driver_flightsql.dbapi.connectotherwise; supportauth_type="external"(adbc.gizmosql.auth_type) for OAuth browser SSO.username/passwordintodb_kwargsforadbc-driver-oracleandtokenintodb_kwargs["adbc.quack.token"]foradbc-driver-quack.core.py):ParameterStyle.POSITIONAL_COLON:1, :2fororacle;ParameterStyle.QMARK?fordb2andmssql/tsql) withpreserve_parameter_format=Falsewhile supporting named input styles (NAMED_COLON,POSITIONAL_COLON,QMARK, andNAMED_AT).oracle,db2,mssql, andduckdb(including Quack) fromadbc_get_info()vendor/driver metadata.core.py):ORA-xxxxxcodes, IBM Db2SQLSTATE/SQLxxxxNcodes (including reason code68lock timeout vs.911/913deadlock), and SQL ServerMsg/ TDS vendor codes (including547check vs. foreign-key constraint disambiguation) to canonicalSQLSpecErrorsubclasses.driver.py,data_dictionary.py):SAVEPOINT <name> ON ROLLBACK RETAIN CURSORS/RELEASE SAVEPOINT <name>fordb2;SAVEPOINT <name>with no-op release fororacle;SAVE TRANSACTION <name>/ROLLBACK TRANSACTION <name>with no-op release formssql/tsql) and identifier validation viavalidate_savepoint_name.enable_lowercase_column_names=Trueby default fororacleanddb2dialects across row and Arrow execution paths while preserving quoted mixed-case identifiers.JSONon Oracle 21c+ andCLOBon 19c and earlier) inAdbcDataDictionary.Extension Stores (
sqlspec/adapters/adbc/{litestar,events,adk}/)litestar/store.py):oracle,db2, andmssql/tsqldialect support toADBCStorewith dialect-specific column types (BLOB/VARBINARY(MAX)), UTC timestamp expressions, idempotent table/index creation and drop guards, and atomicMERGE/ update-then-insert session persistence without untypedNoneArrow binds.events/store.py):AdbcEventQueueStorewithoracle,db2, andmssqlDDL, column types, and idempotent schema creation/drop wrappers.adk/store.py):AdbcADKStoreandAdbcADKMemoryStorefororacle,db2, andmssqlwith_oracle_store_sql(:1, :2placeholder rewriting), dialect-specific table/index guards (PL/SQL-955/-1408/-942blocks for Oracle,SYSCAT.TABLES/SYSCAT.INDEXESprobes for Db2, andOBJECT_ID/sys.indexeschecks for SQL Server),OFFSET ? ROWS FETCH NEXT ? ROWS ONLYpagination, andPOSSTRmemory search on Db2.Documentation (
docs/)docs/reference/adapters/adbc.rst,docs/usage/drivers_and_querying.rst,docs/reference/adapters/db2.rst,docs/extensions/adk/,docs/reference/extensions/events.rst,docs/usage/frameworks/litestar/session_stores.rst):docs/reference/adapters/adbc.rstinto a clear quick-reference driver matrix (install,driver_namealiases, URI schemes, parameter styles, dialect), per-backend connection recipes, and concise operational sections for pip-installable ADBC drivers.Verification