Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -346,10 +346,22 @@ bench-gate: ## Run performance regressio
DRIVERS ?= duckdb aiosqlite
bench-gate-adapters: ## Compare selected adapter benchmarks with committed baselines
@mkdir -p /tmp/sqlspec-bench
@for driver in ${DRIVERS}; do \
uv run python tools/scripts/bench.py --driver "$${driver}" --extended --json-output "/tmp/sqlspec-bench/$${driver}.json"; \
uv run python tools/scripts/bench_compare.py "tools/perf_baselines/$${driver}.json" "/tmp/sqlspec-bench/$${driver}.json" --threshold "$${BENCHMARK_THRESHOLD:-30}"; \
done
@failed_drivers=""; \
for driver in ${DRIVERS}; do \
baseline="tools/perf_baselines/$${driver}.json"; \
run_args=$$(uv run --no-sync python tools/scripts/bench_compare.py "$${baseline}" --print-run-args); \
if ! uv run --no-sync python tools/scripts/bench.py --driver "$${driver}" --extended $${run_args} --json-output "/tmp/sqlspec-bench/$${driver}.json"; then \
failed_drivers="$${failed_drivers} $${driver}"; \
continue; \
fi; \
if ! uv run --no-sync python tools/scripts/bench_compare.py "$${baseline}" "/tmp/sqlspec-bench/$${driver}.json" --mode "$${BENCHMARK_MODE:-ratio}" --threshold "$${BENCHMARK_THRESHOLD:-30}"; then \
failed_drivers="$${failed_drivers} $${driver}"; \
fi; \
done; \
if [ -n "$${failed_drivers}" ]; then \
echo "Adapter benchmark gate failed for:$${failed_drivers}"; \
exit 1; \
fi

.PHONY: bench-subsystems
bench-subsystems: ## Run subsystem micro-benchmarks
Expand Down
107 changes: 107 additions & 0 deletions docs/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,113 @@ important operational fixes.
Recent Updates
==============

v0.67.0 - Enterprise ADBC drivers, Spanner graph and hybrid search, hot-path execution, and Python 3.11+
--------------------------------------------------------------------------------------------------------

**Added:**

* ``sqlspec.adapters.adbc`` adds pure-Go GizmoData ADBC driver support for
Oracle (``adbc-driver-oracle``), IBM Db2 (``adbc-driver-db2``), native
GizmoSQL (``adbc-driver-gizmosql`` with automatic preference over
``adbc-driver-flightsql``, ``gizmosql://`` URIs, and
``gizmosql_auth_type="external"`` browser OAuth/SSO), and DuckDB Quack
(``adbc-driver-quack`` with ``quack://`` / ``quack+tls://`` URIs and
``adbc.quack.token`` bearer authentication), including dialect resolution,
parameter binding (``:1`` positional colon for Oracle, ``?`` qmark for Db2),
savepoints, SQLSTATE/vendor exception mapping, and default
``enable_lowercase_column_names=True`` normalization for Oracle and Db2.
(`#839 <https://github.com/litestar-org/sqlspec/pull/839>`_)
* ``ADBCStore`` (Litestar sessions), ``AdbcEventQueueStore`` (event channels),
``AdbcADKStore``, and ``AdbcADKMemoryStore`` (Google ADK sessions and memory)
add Oracle (``oracle``) and IBM Db2 (``db2``) backend support with idempotent
DDL guards (PL/SQL ``SQLCODE`` blocks for Oracle and ``SYSCAT`` catalog probes
for Db2) and dialect-specific timestamps, types, and search predicates.
(`#839 <https://github.com/litestar-org/sqlspec/pull/839>`_)
* Spanner and Spangres SQLGlot dialects support ``CREATE [OR REPLACE] PROPERTY
GRAPH [IF NOT EXISTS]`` and ``DROP PROPERTY GRAPH [IF EXISTS]`` DDL (with
``NODE TABLES``, ``EDGE TABLES``, ``SOURCE KEY``, ``DESTINATION KEY``,
``LABEL``, and ``PROPERTIES`` clauses), ``GRAPH_TABLE`` queries with ``MATCH``
path patterns and ``COLUMNS`` projections via ``graph_table()``, ``CREATE
VECTOR INDEX [IF NOT EXISTS] ... STORING (...)``, ``DROP VECTOR INDEX [IF
EXISTS]``, ``DROP SEARCH INDEX [IF EXISTS]``, ``HIDDEN`` generated column
constraints (``TOKENLIST AS (TOKENIZE_FULLTEXT(...)) HIDDEN``), and
``approx_cosine_distance(..., options=...)``, ``snippet()``, and
``score_ngrams()`` expression helpers.
(`#834 <https://github.com/litestar-org/sqlspec/pull/834>`_)
* ``SpannerSyncADKMemoryStore`` and ``SpannerAsyncADKMemoryStore`` add vector
embedding columns (``ARRAY<FLOAT32>``) with ``CREATE VECTOR INDEX`` DDL,
optional ``CREATE PROPERTY GRAPH`` DDL overlays, batched ``UNNEST``
deduplication on ``insert_memory_entries()``, and 4-mode ``search_entries()``
supporting Hybrid Reciprocal Rank Fusion (RRF) combining
``APPROX_COSINE_DISTANCE`` and ``SEARCH``/``SCORE``, vector similarity search,
full-text ``SCORE`` ranking, and case-insensitive substring fallback.
(`#834 <https://github.com/litestar-org/sqlspec/pull/834>`_)
* Added ``SpannerSyncADKArtifactStore`` and ``SpannerAsyncADKArtifactStore``
implementations of ``BaseSyncADKArtifactStore`` and
``BaseAsyncADKArtifactStore`` with composite primary keys
``(app_name, user_id, session_id, filename, version)``, user-scoped
``session_id`` normalization, and optional ``ROW DELETION POLICY`` TTL
cleanup.
(`#834 <https://github.com/litestar-org/sqlspec/pull/834>`_)
* ``SpannerSyncEventQueueStore`` and ``SpannerAsyncEventQueueStore`` add
snapshot-read candidate polling (``provide_session(transaction=False)``),
automatic ``Aborted`` retry wrapping via ``run_in_transaction()``, native
``ROW DELETION POLICY`` / ``TTL INTERVAL`` retention (``cleanup_on_ack=False``),
optional ``shard_id`` hash-shard prefixing (``shard_count > 1`` on
``EventsConfig``), covering queue indexes (``STORING`` on GoogleSQL,
``INCLUDE`` on ``spangres``), full ``spangres`` DDL generation, and
``database.update_ddl()`` execution for ``create_table()`` and
``drop_table()``.
(`#834 <https://github.com/litestar-org/sqlspec/pull/834>`_)
* Added Python 3.15 support across package metadata, CI test matrices, and
``cibuildwheel`` MyPyC wheel builds.
(`#835 <https://github.com/litestar-org/sqlspec/pull/835>`_)

**Changed:**

* Upgraded the minimum supported Python version to 3.11+ (``requires-python =
">=3.11, <4.0"``), removed the ``tomli`` fallback in favor of stdlib
``tomllib``, and modernized ``enum.StrEnum``, ``datetime.UTC``, builtin
``TimeoutError``, and stdlib ``typing`` usage across the codebase.
(`#835 <https://github.com/litestar-org/sqlspec/pull/835>`_)
* Standardized driver session lifecycles, statement compilation caching, SQLGlot
builder immutability, migration version tracking, ``driver_features``
type-system coverage, PEP 562 lazy package exports, and MyPyC compilation
across all 20 database adapters (``core``, ``type_converter``, ``pool``, and
``_typing`` modules).
(`#837 <https://github.com/litestar-org/sqlspec/pull/837>`_)
* Unified DuckDB row materialization across ``execute()``, ``select()``, and
``fetch*`` methods to consistently return DuckDB-native ``fetchall()`` Python
types on both first and cached executions, while ``select_to_arrow()``
continues to expose raw Arrow tables.
(`#838 <https://github.com/litestar-org/sqlspec/pull/838>`_)
* Reduced per-statement and per-session execution overhead with per-config shared
statement caches (``StatementCacheHolder``), zero-copy cached SQLGlot AST
sharing with copy-on-mutation via ``ensure_owned_expression()``, fast-path
execution for parameterless and scalar-positional calls, fast-path execution
when statement observers are active, lazy ``SQLResult.statement``
materialization on cached ``SELECT`` hits, single-pass ``execute_many()``
parameter preparation, and idle-queue/clean-transaction fast paths in
``AiosqliteConnectionPool``.
(`#838 <https://github.com/litestar-org/sqlspec/pull/838>`_)

**Fixed:**

* Hardened parameter conversion, Arrow batch streaming (``adbc_ingest`` stream
forwarding and ``mssql_python`` batch streaming), transaction and savepoint
rollback recovery, ``SlotPooledStatement`` cloning on ``SQLResult.statement``,
``sqlglot >= 30.22.0`` single-table ``MappingSchema`` column qualification,
dialect transform isolation (``Spanner``, ``Spangres``, ``DB2``, ``PGVector``,
and ``ParadeDB``), self-referential table dependency ordering and composite
foreign key introspection, Windows ``file:///D:/...`` storage URI
normalization, and exception mapping across core, storage, migrations,
extensions, and database adapters.
(`#837 <https://github.com/litestar-org/sqlspec/pull/837>`_)
* Prevented unparsed or ``COMMAND``/``UNKNOWN`` fallback compilations from
populating the driver statement cache, and cached parsed target-table metadata
on DuckDB Arrow bulk ``execute_many()`` inserts.
(`#838 <https://github.com/litestar-org/sqlspec/pull/838>`_)

v0.66.0 - Native AsyncIO Spanner adapter, extension stores, and sync/async parity
---------------------------------------------------------------------------------

Expand Down
4 changes: 2 additions & 2 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,22 +65,22 @@
"auto_pytabs.sphinx_ext",
"sphinx_copybutton",
"sphinx_click",
"click_extra.sphinx",
"sphinx_design",
"sphinx_paramlinks",
"sphinxcontrib.mermaid",
"numpydoc",
"sphinx_iconify",
"tools.sphinx_ext.playground",
]
myst_enable_extensions = ["alert", "colon_fence"]
intersphinx_mapping = {
"python": ("https://docs.python.org/3", None),
"litestar": ("https://docs.litestar.dev/latest/", None),
"click": ("https://click.palletsprojects.com/en/stable/", None),
"anyio": ("https://anyio.readthedocs.io/en/stable/", None),
"multidict": ("https://multidict.aio-libs.org/en/stable/", None),
"cryptography": ("https://cryptography.io/en/latest/", None),
"pydantic": ("https://docs.pydantic.dev/latest/", None),
"pydantic": ("https://pydantic.dev/docs/validation/latest/", None),
"sanic": ("https://sanic.readthedocs.io/en/latest/", None),
"flask": ("https://flask.palletsprojects.com/en/stable/", None),
"fastapi": ("https://fastapi.tiangolo.com/", None),
Expand Down
7 changes: 4 additions & 3 deletions docs/extensions/adk/adapters.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,14 @@ Use async adapters for best performance with ADK runners:
- **MySQL/MariaDB**: ``aiomysql``, ``asyncmy``
- **SQLite**: ``aiosqlite`` (development and single-process)
- **Oracle**: ``oracledb``
- **IBM Db2**: ``db2`` (native ``ibm_db`` sync and async modes)
- **DuckDB**: ``duckdb`` (analytics; reduced-scope for ADK)
- **ADBC**: ``adbc`` (Arrow-native portability; reduced-scope for ADK)
- **ADBC**: ``adbc`` (Arrow-native portability across PostgreSQL, SQLite, DuckDB, BigQuery, Snowflake, Oracle, and IBM Db2)
- **Spanner**: ``spanner`` (Google Cloud, globally distributed; supports both async and sync modes)
- **SQL Server**: ``mssql_python`` (native Arrow and BulkCopy), ``pymssql`` (FreeTDS), ``arrow_odbc`` (Arrow over Microsoft ODBC Driver 18)

Sync adapters (``psycopg`` sync mode, ``spanner`` sync mode, ``sqlite``, ``mysqlconnector``,
``pymysql``, ``mssql_python``, ``pymssql``, ``arrow_odbc``) work but require wrapping with ``anyio`` for async
Sync adapters (``psycopg`` sync mode, ``spanner`` sync mode, ``db2`` sync mode, ``sqlite``, ``mysqlconnector``,
``pymysql``, ``mssql_python``, ``pymssql``, ``arrow_odbc``, ``adbc``) work with ADK and use ``anyio`` worker-thread dispatch for async
ADK runners.

Each Adapter Provides
Expand Down
34 changes: 20 additions & 14 deletions docs/extensions/adk/backends.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,11 @@ The table below classifies every backend by its ADK support level.
- Full
- Full
- Oracle Database driver.
* - db2
- Supported
- Full
- Basic
- IBM Db2 (LUW); ``POSSTR`` memory search.
* - duckdb
- Reduced-scope
- Full
Expand All @@ -92,7 +97,7 @@ The table below classifies every backend by its ADK support level.
- Reduced-scope
- Full
- Basic
- Portability layer; native adapters provide optimized search.
- Arrow Database Connectivity across PostgreSQL, SQLite, DuckDB, BigQuery, Snowflake, Oracle, and IBM Db2.
* - mssql_python
- Supported
- Full
Expand Down Expand Up @@ -172,9 +177,8 @@ PostgreSQL backends (asyncpg, psycopg, psqlpy) provide the fullest feature set:
- Full-text search via ``tsvector`` for memory entries.
- ``UPSERT`` and ``RETURNING`` clauses for atomic operations.
- ``append_event_and_update_state()`` executes as a single transaction.
- ``fillfactor``, ``autovacuum_vacuum_scale_factor``, and
``autovacuum_analyze_scale_factor`` tune the generated tables. The existing
``fillfactor=80`` default is preserved.
- ``fillfactor`` (default ``80``), ``autovacuum_vacuum_scale_factor``, and
``autovacuum_analyze_scale_factor`` tune the generated tables.

**Recommended for production deployments.**

Expand Down Expand Up @@ -290,25 +294,27 @@ DuckDB provides session and event storage but has limitations:
ADBC
----

ADBC (Arrow Database Connectivity) provides a driver-agnostic interface:
ADBC (Arrow Database Connectivity) provides a portable Arrow-native store
(:class:`~sqlspec.adapters.adbc.adk.AdbcADKStore` and
:class:`~sqlspec.adapters.adbc.adk.AdbcADKMemoryStore`):

- Works with any ADBC-compatible driver (PostgreSQL, SQLite, DuckDB, etc.).
- Dialect-aware DDL, parameter binding, and pagination for PostgreSQL, SQLite,
DuckDB, BigQuery, Snowflake, Oracle (``:1, :2`` placeholders and PL/SQL
idempotent DDL), and IBM Db2 (``SYSCAT`` catalog guards and ``POSSTR`` memory
search).
- Arrow-native data transfer for high-throughput event ingestion.
- Backend capabilities depend on the underlying database driver.
- Memory search uses the portable baseline path; choose a native adapter for
backend-specific FTS, retention, and storage tuning.
- Portable substring memory search across all supported ADBC dialects; choose a
dedicated native adapter when you need backend-specific full-text search
indexes or retention policies.

SQL Server (mssql-python and pymssql)
-------------------------------------

``mssql_python`` and ``pymssql`` provide SQL Server-backed ADK storage:

- Session and event storage use SQL Server tables with ``DATETIME2(6)`` and
``NVARCHAR(MAX)`` JSON payload columns. Microsoft's
`type mapping reference <https://learn.microsoft.com/en-us/sql/connect/python/mssql-python/data-type-mappings>`_
lists native ``JSON`` as unsupported, even when the server supports the type.
The existing ``native_json=True`` override remains available for deployments
that have independently verified driver support.
``NVARCHAR(MAX)`` JSON payload columns. Pass ``native_json=True`` when your
SQL Server deployment and driver support native ``JSON`` columns.
- ``append_event_and_update_state()`` commits the session update, event row, and
scoped state in one transaction.
- Memory search is ``LIKE``-based across memory entries.
Expand Down
Loading
Loading