Merge branch 'main' into fix-decimal-precision-scale-901

Preserve the reviewed decimal compatibility entry under Unreleased while keeping the upstream 4.7.0 release notes unchanged. The decimal implementation and regression blobs are unchanged.

Author: leninworld

CHANGELOG.md

@@ -2,7 +2,13 @@
 
 # Unreleased
 - Correct explicit `DecimalParameter` precision and scale ordering and honor a declared zero scale. Callers that followed the previously reversed parameter descriptions must use `precision` for total digits and `scale` for fractional digits. Explicit zero precision is now sent as declared instead of silently using automatic sizing.
-- Add the `escape_catalog_name` connection parameter (default `False`) to opt in to escaping unescaped `_` in metadata catalog names on Thrift and kernel connections. Existing escapes and `%` wildcards are preserved.
+
+# 4.7.0 (2026-10-20)
+- Upgrade Databricks SQL Kernel to 1.2.0.
+- Add the kernel-only `enable_geospatial_support` option (default `False`) to return GEOMETRY / GEOGRAPHY values as SRID/WKB objects instead of WKT/EWKT strings.
+- Add opt-in `escape_catalog_name` support (default `False`) for metadata queries on Thrift and kernel connections.
+- Share feature-flag values across connections to the same workspace, with typed reads and coordinated refreshes.
+- Allow Apache Thrift 0.25.x and OAuthlib 4.x, including security fixes while preserving support for large Thrift results.
 
 # 4.6.0 (2026-09-24)
 - Upgrade Databricks SQL Kernel to 1.1.0; the kernel dependency is now stable and no longer experimental.

CONNECTION_PARAMETERS.md

@@ -153,6 +153,7 @@ to change without notice.
 | `max_download_threads`                | `int`  |   ✅   |   ❌   | `10`          | Worker threads for cloud-fetch downloads. Not forwarded to the kernel.                                        |
 | `enable_query_result_lz4_compression` | `bool` |   ✅   |   ❌   | `True`        | LZ4-compress result payloads. Not forwarded; the kernel handles compression internally.                       |
 | `_disable_pandas`                     | `bool` |   ✅   |   ✅   | `False`       | Skip the pandas-based Arrow→row deserialization and materialize rows directly with PyArrow. This is a **Python-side** result-conversion toggle, not a wire option: the kernel returns results as Arrow (`RecordBatch`es) and the connector runs the *same* `_convert_arrow_table` for both backends, so the flag is honored on the kernel path too. Affects only row fetches (`fetchone`/`fetchmany`/`fetchall`); the `fetch*_arrow` methods return the Arrow table unchanged regardless of this flag. |
+| `enable_geospatial_support`           | `bool` |   ❌   |   ✅   | `False` | Return GEOMETRY / GEOGRAPHY as `{"srid": int, "wkb": bytes}` values when `True`, or as WKT / EWKT strings when `False`. This is a local result conversion and is never forwarded to SEA. |
 | `_use_arrow_native_complex_types`     | `bool` |   ✅   |   ✅   | `True`        | Return `ARRAY`/`MAP`/`STRUCT` as native Arrow types instead of JSON strings. Forwarded to the kernel.         |
 | `_use_arrow_native_decimals`          | `bool` |   ✅   |   ❌   | `True`        | Thrift wire encoding for `DECIMAL`: `True` → native Arrow `decimal128`, `False` → Arrow string. **No value-level effect**, though: the connector unconditionally re-casts the column back to `decimal128` (`convert_decimals_in_arrow_table`, `thrift_backend.py`), so both `fetchall()` and `fetchall_arrow()` yield `Decimal` / `decimal128(p,s)` either way (verified live). Not forwarded to the kernel, which always returns native Arrow decimals. |
 | `_use_arrow_native_timestamps`        | `bool` |   ✅   |   ❌   | `True`        | Thrift wire encoding for `TIMESTAMP`: `True` → native Arrow timestamp (→ Python `datetime`), `False` → Arrow string (→ Python **`str`**). **Unlike decimals there is no re-cast**, so `False` genuinely surfaces strings — and `cursor.description` still reports the type code as `'timestamp'`, a mismatch to watch for (verified live). Note the connector always also sends the `spark.thriftserver.arrowBasedRowSet.timestampAsString=false` conf, but the `timestampAsArrow=False` flag wins. Not forwarded to the kernel, which always returns native Arrow timestamps. |

KERNEL_REV

@@ -1 +1 @@
-ab1e6a707e4a15a3a7a8c78d8fdd12c752c51102
+39ba84f5ab55c2b0bc874fd58a6f9215ee27cb23

poetry.lock

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "databricks-sql-connector"
-version = "4.6.0"
+version = "4.7.0"
 description = "Databricks SQL Connector for Python"
 authors = ["Databricks <databricks-sql-connector-maintainers@databricks.com>"]
 license = "Apache-2.0"
@@ -67,10 +67,10 @@ requests-kerberos = {version = "^0.15.0", optional = true}
 # is ``abi3`` with ``Requires-Python: >=3.9`` (built ``abi3-py39``); the
 # dependency remains gated to the connector's Python >= 3.10 support floor.
 #
-# Floor is 1.1.0 (``^1.1.0`` == ``>=1.1.0,<2.0.0``), allowing compatible
+# Floor is 1.2.0 (``^1.2.0`` == ``>=1.2.0,<2.0.0``), allowing compatible
 # 1.x releases while keeping the next major upgrade deliberate. The kernel's
 # pyarrow constraint remains >=23.0.1,<24.
-databricks-sql-kernel = {version = "^1.1.0", optional = true, python = ">=3.10"}
+databricks-sql-kernel = {version = "^1.2.0", optional = true, python = ">=3.10"}
 
 
 [tool.poetry.extras]

pyproject.toml

@@ -71,7 +71,7 @@ def __repr__(self):
 DATE = DBAPITypeObject("date")
 ROWID = DBAPITypeObject()
 
-__version__ = "4.6.0"
+__version__ = "4.7.0"
 USER_AGENT_NAME = "PyDatabricksSqlConnector"
 
 # These two functions are pyhive legacy

src/databricks/sql/__init__.py

@@ -170,6 +170,26 @@ def _kernel_session_accepts_kwarg(name: str) -> bool:
     return name in params
 
 
+def _kernel_geospatial_kwargs(value: bool) -> Dict[str, bool]:
+    """Build the explicit native geospatial-support kwarg.
+
+    The value is always passed explicitly so the driver and kernel contracts
+    cannot drift. Older kernel wheels do not declare
+    ``enable_geospatial_support`` and must fail clearly rather than silently
+    return a different public value shape.
+    """
+    if not isinstance(value, bool):
+        raise ValueError(
+            "enable_geospatial_support must be a bool; " f"got {type(value).__name__}"
+        )
+    if not _kernel_session_accepts_kwarg("enable_geospatial_support"):
+        raise NotSupportedError(
+            "enable_geospatial_support requires a newer databricks-sql-kernel "
+            "wheel that exposes geospatial result representation support."
+        )
+    return {"enable_geospatial_support": value}
+
+
 def _kernel_telemetry_kwargs(options: Dict[str, Any]) -> Dict[str, Any]:
     """Build phase-7 telemetry/system kwargs for ``databricks_sql_kernel.Session``.
 
@@ -263,6 +283,12 @@ def __init__(
         # The kernel binding owns type and range validation.
         self._request_timeout_secs = kwargs.get("request_timeout_secs")
         self._max_connections = kwargs.get("max_connections")
+        # Opt-in native GEOMETRY / GEOGRAPHY support. True requests
+        # the canonical Arrow struct and surfaces as ``{"srid": int, "wkb":
+        # bytes}`` through pyarrow; False requests WKT / EWKT strings. This is
+        # intentionally separate from ``session_configuration``: it is never
+        # forwarded to SEA.
+        self._enable_geospatial_support = kwargs.get("enable_geospatial_support", False)
         # Kernel telemetry phase 7 adds binding/runtime identity and
         # telemetry config kwargs directly to ``databricks_sql_kernel.Session``.
         self._telemetry_options = kwargs.get("telemetry_options") or {}
@@ -381,6 +407,9 @@ def open_session(
             # kernel's ``retry_*`` kwargs. Empty when at defaults.
             retry_kwargs = _kernel_retry_kwargs(self._retry_options)
             telemetry_kwargs = _kernel_telemetry_kwargs(self._telemetry_options)
+            geospatial_kwargs = _kernel_geospatial_kwargs(
+                self._enable_geospatial_support
+            )
             max_connections_kwargs: Dict[str, Any] = {}
             if _kernel_session_accepts_kwarg("max_connections"):
                 max_connections_kwargs["max_connections"] = self._max_connections
@@ -428,6 +457,7 @@ def open_session(
                 **tls_kwargs,
                 **retry_kwargs,
                 **telemetry_kwargs,
+                **geospatial_kwargs,
                 **max_connections_kwargs,
                 **http_headers_kwargs,
             )
@@ -1015,11 +1045,9 @@ def get_tables(
         if self._kernel_session is None:
             raise InterfaceError("get_tables requires an open session.")
         try:
-            # ``table_types`` is filtered kernel-side (the kernel applies
-            # it to the reshaped result, case-insensitively as of the
-            # batch-3 kernel change), so we forward it and let the kernel
-            # do the work — no connector-side drain + refilter. Passing it
-            # through preserves streaming for large schemas.
+            # The kernel filters ``table_types`` with case-sensitive
+            # matching for Thrift parity. Forward the values unchanged
+            # to preserve those semantics and streaming for large schemas.
             stream = self._kernel_session.metadata().list_tables(
                 catalog=escape_metadata_catalog_name(
                     _catalog_or_none(catalog_name), self._escape_catalog_name

src/databricks/sql/backend/kernel/client.py

@@ -3,6 +3,7 @@
 import errno
 import logging
 import math
+import re
 import time
 import threading
 from typing import Dict, List, Optional, Union, Any, TYPE_CHECKING
@@ -807,38 +808,50 @@ def _col_to_description(col, field=None, host_url=None):
         else:
             precision, scale = None, None
 
-        # Extract variant type from field if available
+        # Extract logical types that Thrift surfaces as strings from Arrow metadata.
         if field is not None:
             try:
-                # Check for variant type in metadata
                 if field.metadata and b"Spark:DataType:SqlName" in field.metadata:
                     sql_type = field.metadata.get(b"Spark:DataType:SqlName")
-                    if sql_type == b"VARIANT":
-                        cleaned_type = "variant"
+                    logical_type = re.fullmatch(
+                        rb"(variant|geometry|geography)(?:\([^()]*\))?",
+                        sql_type.strip(),
+                        re.IGNORECASE,
+                    )
+                    if logical_type:
+                        cleaned_type = logical_type.group(1).decode("ascii").lower()
             except Exception as e:
-                logger.debug(f"Could not extract variant type from field: {e}")
+                logger.debug(f"Could not extract logical type from field: {e}")
 
         return col.columnName, cleaned_type, None, None, precision, scale, None
 
     @staticmethod
     def _hive_schema_to_description(t_table_schema, schema_bytes=None, host_url=None):
+        arrow_fields = []
         field_dict = {}
         if pyarrow and schema_bytes:
             try:
                 arrow_schema = pyarrow.ipc.read_schema(pyarrow.py_buffer(schema_bytes))
-                # Build a dictionary mapping column names to fields
-                for field in arrow_schema:
+                arrow_fields = list(arrow_schema)
+                # Retain a name fallback for malformed/partial schemas, but use
+                # result-column position whenever it is available. SQL permits
+                # duplicate aliases, for which a name lookup is ambiguous.
+                for field in arrow_fields:
                     field_dict[field.name] = field
             except Exception as e:
                 logger.debug(f"Could not parse arrow schema: {e}")
 
         return [
             ThriftDatabricksClient._col_to_description(
                 col,
-                field_dict.get(col.columnName) if field_dict else None,
+                (
+                    arrow_fields[index]
+                    if index < len(arrow_fields)
+                    else field_dict.get(col.columnName)
+                ),
                 host_url,
             )
-            for col in t_table_schema.columns
+            for index, col in enumerate(t_table_schema.columns)
         ]
 
     def _results_message_to_execute_response(

src/databricks/sql/backend/thrift_backend.py

@@ -194,6 +194,13 @@ def __init__(
                 decision. This is an intentional divergence from the
                 Thrift/SEA paths, where an explicit ``True`` can still
                 be suppressed by the feature flag.
+            :param enable_geospatial_support: `bool`, optional (default is False)
+                Kernel backend only. Controls the public representation of
+                ``GEOMETRY`` and ``GEOGRAPHY`` result values. ``True`` returns
+                ``{"srid": int, "wkb": bytes}``; ``False`` returns WKT / EWKT
+                strings (for example ``"SRID=4326;POINT(1 2)"``). The
+                conversion is local to the kernel/driver and this option is
+                never sent to the SQL Execution API.
             :param use_hybrid_disposition: `bool`, optional (default is False)
                 Use the hybrid disposition instead of the inline disposition.
             :param server_hostname: Databricks instance host name.

src/databricks/sql/client.py

@@ -332,6 +332,9 @@ def _create_backend(
                 retry_options=kernel_retry_options,
                 request_timeout_secs=kwargs.get("_socket_timeout"),
                 max_connections=kwargs.get("_pool_maxsize") or None,
+                enable_geospatial_support=kwargs.get(
+                    "enable_geospatial_support", False
+                ),
                 telemetry_options=kernel_telemetry_options,
                 escape_catalog_name=kwargs.get("escape_catalog_name", False),
             )