diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 7ae494d1f..cd134d6b5 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -24,6 +24,9 @@ This page tracks significant updates to the QuestDB documentation. ### Updated +- [Python client](/docs/connect/clients/python/) - Documented 5.1.0 QWP-only row types, UUID/LONG256 binary column overrides and byte-order migration, and pandas type-preserving round trips including DATE and read-only type metadata +- [Python](/docs/connect/clients/python/#dataframe-ingestion) and [Rust](/docs/connect/clients/rust/#arrow-and-polars-ingestion) clients - Clarified DataFrame-wide `in_doubt` status versus individual flushes and the risk of duplicating committed rows on retry +- [Store-and-forward](/docs/high-availability/store-and-forward/concepts/#reconnect-and-replay) - Corrected replay guidance: connection-local sequence numbers do not deduplicate writes; table-level deduplication with stable row identity is required to suppress duplicates - [query_activity()](/docs/query/functions/meta/#query_activity) - Documented the `is_wal`, `memory_used`, and `memory_limit` columns - [wal_tables()](/docs/query/functions/meta/#wal_tables) - Documented the `errorTag`, `errorMessage`, and `memoryPressure` columns; `errorTag` reads `OUT OF MEMORY` after a WAL apply memory limit breach - [SHOW](/docs/query/sql/show/) - `SHOW USERS`, `SHOW GROUPS`, and `SHOW SERVICE ACCOUNTS`, including their filtered forms, gain a trailing `memory_limit` column. Clients that read these results by position need [updating](/docs/security/rbac/#memory-limit-upgrade) diff --git a/documentation/concepts/delivery-semantics.md b/documentation/concepts/delivery-semantics.md index 597c1910c..9f2c70b84 100644 --- a/documentation/concepts/delivery-semantics.md +++ b/documentation/concepts/delivery-semantics.md @@ -38,6 +38,11 @@ the server confirms a batch, the client reconnects and re-sends. If the server had already committed the batch but the acknowledgement was lost in flight, the second send produces duplicates. +QWP's `wireSeq` cannot suppress this replay: it is assigned by receive order, +exists only for response correlation on one connection, and resets after a +reconnect. Requests carry no persistent message identifier that the server can +use to recognize the same frame on the next connection. + This path applies to every QuestDB client deployment. ### Multi-host failover replay diff --git a/documentation/connect/clients/c-and-cpp.md b/documentation/connect/clients/c-and-cpp.md index 9b82a307c..13bd9f6d5 100644 --- a/documentation/connect/clients/c-and-cpp.md +++ b/documentation/connect/clients/c-and-cpp.md @@ -696,7 +696,9 @@ on_error:; the 2 MiB target; if 8 rows still exceed 4 MiB — which takes very large string, binary, or array values — the flush fails instead of splitting. - **Recovery depends on `in_doubt`, not on the error code.** Check - `line_sender_error_in_doubt` (C++: `e.in_doubt()`). False means the queue + `line_sender_error_in_doubt` (C++: `e.in_doubt()`). This describes the + failed operation's input, not earlier independent flushes or replay from an + application checkpoint. False means the queue never took the frame and the chunk is intact: re-flush it. True means delivery is uncertain, so `wait` for what the queue already holds, and resend the chunk only where the table's dedup keys make duplicate rows harmless. A @@ -776,13 +778,18 @@ See `qwp_sender.h` for the exact signatures. Complete list: | `column_bool` | LSB-first packed bitmap | `BOOLEAN` | | `column_ts` + `qwp_ts_unit` (`_micros` / `_nanos`) | int64 since epoch | `TIMESTAMP` / `TIMESTAMP_NS` | | `column_date` | int64 millis since epoch | `DATE` | -| `column_uuid` | 16 bytes, canonical RFC-4122 order | `UUID` | -| `column_long256` | 32 bytes (4 LE limbs) | `LONG256` | +| `column_uuid` | 16 bytes, canonical RFC 4122 big-endian | `UUID` | +| `column_long256` | 32 bytes (4 LE limbs, low limb first) | `LONG256` | | `column_ipv4` | uint32 | `IPV4` | | `column_str` | Arrow Utf8 offsets + bytes | `VARCHAR` | | `column_binary` | Arrow Binary offsets + bytes | `BINARY` | | `symbol_i8` / `_i16` / `_i32` | dict codes + Utf8 dictionary | `SYMBOL` | +`column_uuid` takes the UUID's canonical RFC 4122 bytes, exactly as they are +written in the textual form, and byte-swaps them into QWP wire order for you. +The row-oriented `line_sender_buffer_column_uuid` is the exception: it takes +the two 64-bit wire halves, `(lo, hi)`. + Designated timestamp (exactly once per chunk, before flush): `at_nanos` / `at_micros` / `at_millis` / `at_seconds` (millis and seconds are widened to micros on the wire). Decimals, geohash, arrays, and the @@ -873,17 +880,137 @@ bool ingest(questdb_db* db, struct ArrowArray* array, caller keeps `schema`. On failure check `array->release != NULL` before invoking it. - Per-column wire-type hints (`qwp_arrow_override`: force - SYMBOL/VARCHAR, IPv4, char, geohash precision) steer encoding without - touching the Arrow schema. + SYMBOL/VARCHAR, IPv4, char, geohash precision, UUID, LONG256) choose the + wire type without touching the Arrow schema. An override wins for its + column over any field metadata the schema carries. - To append Arrow **columns** into a chunk alongside hand-built ones, use `qwp_chunk_append_arrow_column`, or `qwp_arrow_import_new` + `..._append_arrow_import` to import once - and slice across many chunks. + and slice across many chunks. Neither takes an overrides array, so an + IPv4, char, geohash, UUID, or LONG256 column has to carry its claim as + field metadata instead. The SYMBOL choice is still available on the import + path: `qwp_arrow_import_new` takes a `symbol_mode` argument + (`qwp_symbol_mode_auto`, `_symbol`, `_not_symbol`). - Dictionary-encoded string columns map to `SYMBOL` by default; plain Utf8 to `VARCHAR`. `qwp_sender.h` lists every Arrow type the client accepts, and the kinds it rejects (`Struct`, `Map`, `Interval`, ...); a rejected type fails with `line_sender_error_arrow_unsupported_column_kind`. +### Binary columns: UUID, LONG256, and opaque bytes + +Binary Arrow columns land as `BINARY` unless the column *claims* a richer +type. The width of a column claims nothing on its own: a bare +`FixedSizeBinary(16)` is opaque bytes, not a UUID. A claim comes from the +schema or from an override: + +| Claim | Lands as | +| --- | --- | +| `ARROW:extension:name = arrow.uuid` on `FixedSizeBinary(16)` | `UUID` | +| `questdb.column_type = uuid` field metadata | `UUID` | +| `questdb.column_type = long256` field metadata | `LONG256` | +| `qwp_arrow_override_uuid` / `qwp_arrow_override_long256` | `UUID` / `LONG256` | + +UUID bytes are canonical RFC 4122 big-endian and the client byte-swaps them +into QWP wire order; LONG256 bytes are little-endian limbs, low limb first, +and go out verbatim. The `questdb.column_type` claims and the two overrides +also apply to variable-length `Binary` / `LargeBinary` / `BinaryView` +columns, where every non-null value must then be exactly 16 or 32 bytes. The +`arrow.uuid` extension is the exception: the Arrow spec fixes its storage to +`FixedSizeBinary(16)`, so the client rejects the label on any other type. A +claim whose width doesn't match fails with +`line_sender_error_arrow_ingest`. + +:::caution Behaviour change + +Before client 7.0.0 a bare `FixedSizeBinary(16)` or `(32)` column became +`UUID` or `LONG256` on width alone, with no claim needed. It is now `BINARY` +unless the column carries one of the claims above, so a batch that used to +produce a `UUID` column now produces a `BINARY` one and reports no error. + +The UUID byte order at the API boundary changed in the same release. Code +written against an earlier version passed QWP wire-order bytes to +`qwp_chunk_column_uuid` and `qwp_reader_query_bind_uuid`; those values are +now stored with their bytes reversed, also with no error. + +::: + +#### Claiming in the schema + +Both metadata claims are attached to the Arrow `Field`, so you make them +wherever the batch is built. In Arrow C++: + +```cpp +// The standard Arrow extension label. FixedSizeBinary(16) only. +auto trade_id = arrow::field("trade_id", arrow::fixed_size_binary(16)) + ->WithMetadata(arrow::key_value_metadata( + {"ARROW:extension:name"}, {"arrow.uuid"})); + +// The QuestDB claim, also valid on Binary / LargeBinary / BinaryView. +auto order_hash = arrow::field("order_hash", arrow::fixed_size_binary(32)) + ->WithMetadata(arrow::key_value_metadata( + {"questdb.column_type"}, {"long256"})); + +auto batch_schema = arrow::schema({ + arrow::field("ts", arrow::timestamp(arrow::TimeUnit::NANO)), + trade_id, + order_hash}); +``` + +`questdb.column_type = uuid` has the same shape with `uuid` as the value. Use +it in place of `arrow.uuid` when the bytes sit in a variable-length binary +column, which the extension label doesn't allow. + +Export the batch built against that schema through `arrow::ExportRecordBatch` +and flush it exactly as above — the claims travel with it, and the flush call +needs no extra arguments. + +#### Claiming at the call site + +An override claims the type per flush and leaves the schema alone. Fill in a +`qwp_arrow_override` per column and pass the array to any +`flush_arrow_batch*` call, where the example above passes no overrides: + + + + +```cpp +using namespace questdb::ingress::literals; + +const ::qwp_arrow_override overrides[] = { + {"trade_id", sizeof("trade_id") - 1, qwp_arrow_override_uuid, 0}, + {"order_hash", sizeof("order_hash") - 1, qwp_arrow_override_long256, 0}, +}; + +sender.flush_arrow_batch_and_wait( + "trades"_tn, array, schema, "ts"_cn, + overrides, std::size(overrides)); +``` + + + + +```c +const qwp_arrow_override overrides[] = { + {"trade_id", sizeof("trade_id") - 1, qwp_arrow_override_uuid, 0}, + {"order_hash", sizeof("order_hash") - 1, qwp_arrow_override_long256, 0}, +}; + +bool ok = qwp_sender_flush_arrow_batch_at_column_and_wait( + sender, QDB_TABLE_NAME_LITERAL("trades"), array, schema, + QDB_COLUMN_NAME_LITERAL("ts"), + overrides, sizeof(overrides) / sizeof(overrides[0]), + qwpws_ack_level_ok, &err); +``` + + + + +`arg` (the trailing `0`) carries the geohash precision for +`qwp_arrow_override_geohash` and is unused by every other kind. An override +that names a column the batch doesn't have, repeats another override's +column, or carries an unknown kind fails with +`line_sender_error_invalid_api_call`. + ## Querying data Get a reader (QWP/WebSocket only), prepare/execute SQL, then stream batches and @@ -1084,6 +1211,12 @@ For width-independent access, use `column::visit` and mantissa as little-endian two's-complement bytes. Check for null before decoding it. +`qwp_reader_column_data_get_bytes` also serves `UUID` and `LONG256`. It hands +back UUID values as 16 canonical RFC 4122 big-endian bytes — the decoder has +already reversed them out of wire order, so they match what +`qwp_chunk_column_uuid` and `bind_uuid` take — and LONG256 values as 32 +little-endian limb bytes, low limb first, verbatim from the wire. + ### Parameterised queries Prepare then bind: C `qwp_reader_prepare` + `qwp_reader_query_bind_*` + @@ -1103,8 +1236,8 @@ outlive any cursor it produces. The complete bind surface (C | `bind_decimal64` / `bind_decimal128` / `bind_decimal256` | unscaled value + scale | `DECIMAL` | | `bind_geohash` | bits + precision | `GEOHASH` | | `bind_varchar` | UTF-8 string | `VARCHAR` | -| `bind_uuid` | 16 bytes | `UUID` | -| `bind_long256` | 32 bytes | `LONG256` | +| `bind_uuid` | 16 bytes, canonical RFC 4122 big-endian | `UUID` | +| `bind_long256` | 32 bytes (4 LE limbs, low limb first) | `LONG256` | | `bind_binary` | bytes + length | `BINARY` (not yet accepted server-side) | | `bind_ipv4` | uint32, host order | `IPV4` (not yet accepted server-side) | diff --git a/documentation/connect/clients/python.md b/documentation/connect/clients/python.md index be8d79aa2..3c4c5a1a3 100644 --- a/documentation/connect/clients/python.md +++ b/documentation/connect/clients/python.md @@ -277,6 +277,13 @@ The Python value type selects the QuestDB column type: | `TimestampMicros`, `TimestampNanos`, `datetime.datetime` | `TIMESTAMP`, `TIMESTAMP_NS` | | `numpy.ndarray` of `float64`, any number of dimensions | `DOUBLE[]`, `DOUBLE[][]`, ... matching the array's shape. QuestDB 9.0.0 or later | | `decimal.Decimal` | `DECIMAL`, QuestDB 9.2.0 or later | +| `uuid.UUID` | `UUID`, QWP only | +| `ipaddress.IPv4Address` | `IPV4`, QWP only | +| `bytes`, `bytearray`, `memoryview` | `BINARY`, QWP only | +| `Char` | `CHAR`, QWP only | +| `DateMillis` | `DATE`, QWP only | +| `Long256` | `LONG256`, QWP only | +| `Geohash` | `GEOHASH`, QWP only | | `None` | Column omitted for this row, stored as null | Nulls are written by omission: skip the key or pass `None`; there is no @@ -287,10 +294,62 @@ strings in `columns` become `VARCHAR`. `DECIMAL` columns must be created ahead of time with `CREATE TABLE ... (price DECIMAL(18, 2), ...)`; the server does not auto-create them. -`UUID`, `IPv4`, `GEOHASH`, `LONG256`, `CHAR`, `DATE`, and `BINARY` columns -have no `row()` value type. Route them through -[`dataframe()`](#dataframe-ingestion), whose `schema_overrides` covers -`symbol`, `ipv4`, `char`, and `geohash`, or through a SQL `INSERT` via +The seven types marked "QWP only" need Python client 5.1.0 or later, +QuestDB 10 or later, and a `udp`, `ws`, or `wss` connection. On a `tcp`, +`tcps`, `http`, or `https` sender they raise `QuestDBError`. + +Three of them map to a Python type you already have: `uuid.UUID`, +`ipaddress.IPv4Address`, and any bytes-like value. The other four have no +obvious Python equivalent, so the client gives you a small wrapper for each: + +| Wrapper | Takes | +| --- | --- | +| `Char("A")` | a one-character string | +| `DateMillis(1735689600000)` | milliseconds since the Unix epoch | +| `Long256(0xdeadbeef)` | an unsigned 256-bit `int` | +| `Geohash(bits, precision)` | the hash bits and how many bits they use | +| `Geohash.from_string("u33d8b12")` | the text form, 1 to 12 characters | + +A geohash column uses one precision for every row. The first row you write +fixes it, and if a later row has a different precision, `row()` raises +`QuestDBError` straight away. The bad row is removed, so the rest of the +buffer is untouched and you can carry on writing. + +```python +import uuid +from questdb import Char, DateMillis, Geohash, Long256, TimestampNanos + +sender.row( + "events", + columns={ + "id": uuid.UUID("123e4567-e89b-12d3-a456-426614174000"), + "payload": b"\x00\x01", + "grade": Char("A"), + "day": DateMillis(1735689600000), + "hash": Long256(0xdeadbeef), + "loc": Geohash.from_string("u33d8b12"), + }, + at=TimestampNanos.now(), +) +``` + +Some of these types have one value that QuestDB uses to mean `NULL`. The +client writes that value if you pass it, so it goes in fine and comes back +out as `NULL`: + +| Type | Value that reads back as `NULL` | +| --- | --- | +| `IPV4` | `0.0.0.0` | +| `DATE` | `INT64_MIN` | +| `UUID` | `80000000-0000-0000-8000-000000000000` | +| `LONG256` | all four 64-bit limbs set to `0x8000000000000000` | + +`CHAR` and `BINARY` have no such value. `Char("\x00")` is stored as code unit +0, although some SQL functions treat that as absent, and empty `BINARY` +(`b""`) is a real empty value that is not `NULL`. + +You can also write these types with +[`dataframe()`](#dataframe-ingestion), or with a SQL `INSERT` through [`query()`](#querying). QWP cannot preserve nulls for `BOOLEAN`, `BYTE`, or `SHORT`. An absent value @@ -401,6 +460,22 @@ with questdb.connect("ws::addr=localhost:9000;") as db: db.dataframe(df, table_name="trades", symbols=["symbol"], at="timestamp") ``` +The first successful batch on a fresh direct connection is already a commit +boundary. Later batches are pipelined until an explicit checkpoint or the final +commit. If a transient failure occurs before any batch is successfully +published, the client can replay a materialized source in full. Once any batch +may have committed, it raises instead of replaying from row zero and reports +`in_doubt=True` for the whole DataFrame call, even if the final native write +alone was provably not delivered. This also covers local validation and Arrow +stream errors after earlier batches were published. Internal checkpoints do +not reset the call's delivery status. This aggregation is specific to +`dataframe()`; a sender flush's flag does not summarize earlier independent +flushes. An application-level retry can then duplicate +an already committed prefix unless the table uses suitable `DEDUP UPSERT KEYS`. +A consumed one-shot Arrow stream can also be impossible to replay; when no +batch could have landed, that separate error has `in_doubt=False` and asks for +a fresh reader. + `df` accepts pandas `DataFrame`, polars `DataFrame` and `LazyFrame`, pyarrow `Table`, `RecordBatch`, and `RecordBatchReader`, and any object exposing the Arrow C Data Interface: @@ -428,7 +503,7 @@ Parameters: | `symbols` | `"auto"` (default: categorical and dictionary columns become `SYMBOL`), a bool, or a list of column names or indices. | | `at` | The designated timestamp column (by name or index), a fixed `TimestampNanos` or `datetime` shared by every row, or `questdb.ServerTimestamp`. | | `max_rows_per_batch` | Rows per published batch, default 16384. Sets pipelining granularity, not a safety limit — see below. | -| `schema_overrides` | Per-column wire-type overrides, e.g. `{"addr": "ipv4", "loc": ("geohash", 20)}`; values are `symbol`, `ipv4`, `char`, or `geohash`. | +| `schema_overrides` | Per-column type, e.g. `{"addr": "ipv4", "loc": ("geohash", 20)}`. Values are `symbol`, `ipv4`, `char`, `uuid`, `long256`, or `("geohash", bits)` with `bits` from 1 to 60. Beats any Arrow field metadata on the column. Needs a frame where every column is Arrow-backed; otherwise it raises `UnsupportedDataFrameShapeError`. | `max_rows_per_batch` decides how the frame is cut into published batches, and each batch is one unit of encoding, memory, and server-side apply. @@ -437,9 +512,10 @@ negotiated per-batch byte cap regardless of this setting, and a single row is never bounded by it. What it does control: - Peak client memory: each batch is encoded and held as one frame. -- Recovery quantum: a commit checkpoint fires every 100 batches, so - `max_rows_per_batch × 100` rows is the replay window on a transient - failover. +- Checkpoint spacing: sliceable Arrow inputs add a commit checkpoint about + every 100 batches. `max_rows_per_batch × 100` approximates the maximum + periodic uncommitted tail, not a safe whole-source replay window; the first + successful batch on a fresh connection is already a commit boundary. - Per-batch overhead: very small batches pay framing and server-side apply costs per batch. @@ -459,6 +535,144 @@ row ingestion. A frame the columnar path cannot express raises `UnsupportedDataFrameShapeError` with per-column failures in `column_failures`. +### Binary, UUID, and LONG256 columns + +Binary columns (`pyarrow.binary()`, `pyarrow.large_binary()`, fixed-size +`pyarrow.binary(n)`, and polars `Binary`) are written as `BINARY`. The width +of a column does not decide its type, so a 16-byte column is treated as +plain bytes, not as a UUID. + +:::caution Upgrading to Python client 5.1.0 + +Earlier clients inferred UUID or LONG256 from a 16- or 32-byte Arrow +column's width. In 5.1.0 and later, explicitly identify these columns using +`schema_overrides`, Arrow field metadata, or the `arrow.uuid` extension +for UUIDs. Otherwise, the fully Arrow-backed path writes `BINARY`; mixed +Arrow/NumPy frames reject these unlabelled fixed-size columns. + +UUID raw bytes also changed to canonical RFC 4122 big-endian order. Remove +any byte-swapping used for the old QWP wire layout on both ingestion and +query results. For example, replace `value.int.to_bytes(16, "little")` +with `value.bytes`. Adding a UUID override without fixing the byte order +silently stores reversed UUIDs. Existing `uuid.UUID` object columns and +query binds need no byte-order change. The QWP wire format is unchanged. + +::: + +For UUIDs, an object column of `uuid.UUID` values needs no extra +configuration. The client handles the byte order: + +```python +import uuid + +df = pd.DataFrame({ + "trade_id": [uuid.uuid4(), uuid.uuid4()], + "price": [2615.54, 65432.10], + "timestamp": pd.to_datetime([ + "2025-01-01T00:00:00Z", + "2025-01-01T00:00:01Z", + ]), +}) + +db.dataframe(df, table_name="trades", at="timestamp") +``` + +For columns that are already binary, name the types with +`schema_overrides`. The input must be fully Arrow-backed. This complete +example builds a pyarrow table with both binary columns: + +```python +from datetime import datetime, timezone +import uuid + +import pyarrow as pa +import questdb + +trade_uuids = [ + uuid.UUID("123e4567-e89b-12d3-a456-426614174000"), + uuid.UUID("123e4567-e89b-12d3-a456-426614174001"), +] +table = pa.table({ + "trade_id": pa.array([u.bytes for u in trade_uuids], type=pa.binary(16)), + "order_hash": pa.array( + [n.to_bytes(32, "little") for n in (0xdeadbeef, 0xcafebabe)], + type=pa.binary(32), + ), + "price": pa.array([2615.54, 65432.10], type=pa.float64()), + "timestamp": pa.array([ + datetime(2025, 1, 1, tzinfo=timezone.utc), + datetime(2025, 1, 1, 0, 0, 1, tzinfo=timezone.utc), + ], type=pa.timestamp("us", "UTC")), +}) + +with questdb.connect("ws::addr=localhost:9000;") as db: + db.dataframe( + table, + table_name="trades", + at="timestamp", + schema_overrides={"trade_id": "uuid", "order_hash": "long256"}, + ) +``` + +Every non-null value must then be exactly 16 bytes for `uuid`, or 32 bytes +for `long256`. UUID bytes are canonical RFC 4122 big-endian — the same bytes +`uuid.UUID.bytes` gives you, and the same bytes a `UUID` result column reads +back. LONG256 bytes are little-endian limbs, least significant first, and are +sent unchanged. + +The third way is a pyarrow column built with the `arrow.uuid` extension type, +which carries the claim itself. Using `trade_uuids` from the example above: + +```python +import pyarrow as pa + +trade_ids = pa.ExtensionArray.from_storage( + pa.uuid(), + pa.array([u.bytes for u in trade_uuids], type=pa.binary(16)), +) +``` + +This one needs pyarrow 18 or later, where `pa.uuid()` was added. Build the +column from `pa.uuid()` itself — writing `ARROW:extension:name` as field +metadata is not the same thing, because a pandas column keeps the Arrow type +but not the field, so the label is lost on the way in. + +:::note + +A frame where every column is Arrow-backed takes a different code path from +one that mixes Arrow and NumPy columns, and the two treat an unlabelled +16- or 32-byte column differently. + +On a fully Arrow-backed frame it is written as `BINARY`, because +`schema_overrides` is there if you meant something else. If any column is not +Arrow-backed, `schema_overrides` is unavailable, and rather than guess +between "plain bytes" and "a UUID whose label was lost", the client refuses +the column and tells you how to say which you meant. To send those widths as +plain bytes there, pass them as an object column of `bytes`. + +::: + +### DATE columns + +`row()` writes a `DATE` with the `DateMillis` wrapper. For `dataframe()`, +use an Arrow column of `pyarrow.timestamp("ms")`, `pyarrow.date32()`, or +`pyarrow.date64()`. These types are written as `DATE` in fully Arrow-backed +frames and frames that mix Arrow and NumPy columns. There is no `date` +value for `schema_overrides`: the Arrow type identifies the column. + +Without a DATE claim, a NumPy `datetime64[ms]` column widens to a +microsecond `TIMESTAMP`, and its timezone-aware `datetime64[ms, tz]` form +is rejected. + +Query results preserve the type. `to_arrow()` and +`to_pandas(dtype_backend="pyarrow")` return an Arrow +`timestamp("ms", "UTC")` column, which can be written straight back as +`DATE`. Plain `to_pandas()` returns timezone-naive `datetime64[ms]` +holding UTC instants and records a `{"kind": "date"}` claim in +`df.attrs["questdb"]`. Python client 5.1.0 and later reads that claim to +restore the Arrow DATE type before ingestion. Keep the claim when +[writing a result back](#writing-a-result-back). + Naive timestamps — DataFrame columns and the scalar `at` alike — are interpreted as UTC, matching the numpy `datetime64` convention. Prefer timezone-aware values throughout. @@ -583,12 +797,76 @@ thread that created it; `db.close()` waits for open leases. | `SYMBOL` | `Categorical` sharing one dictionary across batches | | `VARCHAR` | Strings with `None` for null | | `DECIMAL`, `UUID`, `BINARY` | `object` columns of `decimal.Decimal`, `uuid.UUID`, `bytes` | +| `LONG256` | `object` column of Python `int` — the only type wide enough without pyarrow | +| `IPV4`, `CHAR` | `uint32`, `uint16` | +| `DATE` | Timezone-naive `datetime64[ms]` holding UTC instants, `NaT` for null | +| `GEOHASH` | A signed integer wide enough for the column's precision: `int8` up to 7 bits, `int16` to 15, `int32` to 31, `int64` to 60 | QuestDB's sentinel values (for example `NaN` doubles and `INT64_MIN` longs) are decoded as nulls rather than leaking as magic numbers. -`to_pandas(dtype_backend="pyarrow")`, `dtype_backend="numpy_nullable"`, or -a `types_mapper=` callable select pyarrow-backed dtypes instead, matching -the `pd.read_sql` convention. +`to_pandas(dtype_backend="pyarrow")` selects Arrow-backed dtypes; +`dtype_backend="numpy_nullable"` selects pandas nullable dtypes where +available. A `types_mapper=` callable provides custom Arrow-to-pandas +dtype mapping. + +### Writing a result back + +Python client 5.1.0 and later preserves UUID, LONG256, IPV4, CHAR, GEOHASH, +and DATE types when you read a table into pandas, change values, and write +it back: + +```python +df = db.query("SELECT * FROM trades").to_pandas() +df["price"] *= 1.01 +db.dataframe(df, table_name="trades_adjusted", at="timestamp") +``` + +A pandas dtype does not always identify the QuestDB type. For example, +`IPV4` arrives as `uint32`, which would otherwise be written as `LONG`. +`to_pandas()` records the source types in `df.attrs["questdb"]`, and +`dataframe()` reads that metadata to preserve them. + +Plain `to_pandas()` returns `UUID` values as `uuid.UUID` objects and +`LONG256` values as Python integers. With `dtype_backend="numpy_nullable"`, +both are object columns of `bytes` instead. The `"pyarrow"` backend keeps +Arrow-backed columns. All three backends attach the metadata needed to +preserve the six QuestDB types above. + +Editing the frame is safe. A column you drop, rename, or convert to another +type simply loses its record, and the write goes ahead with whatever the +column now is. `symbols` and `schema_overrides` win over it, so you can +always state a type yourself. Two types do not survive unchanged: `BYTE` and +`SHORT` columns come back as `INT`, and `INT` as `LONG`. + +If a recorded type cannot apply to the column as it now stands — say the +column is an unsigned integer and the record says `geohash` — the client +warns and writes the column as its own type implies, rather than failing. + +The metadata returned by `to_pandas()` is a read-only dictionary shared +by copies of the frame. Its nested mappings are also read-only; editing +any of them in place raises `TypeError`. + +To change the metadata, assign a new dictionary to `df.attrs["questdb"]`. +Unpack the existing `columns` mapping to retain its other entries: + +```python +df.attrs["questdb"] = { + "version": 1, + "columns": { + **df.attrs["questdb"]["columns"], + "src_ip": {"kind": "ipv4"}, + "pos": {"kind": "geohash", "precision_bits": 20}, + "traded_on": {"kind": "date"}, + }, +} +``` + +For a hand-built frame with no existing claim, omit the unpacking line. +`version` is required and must be `1`. `kind` is one of `uuid`, `long256`, +`ipv4`, `char`, `geohash`, or `date`; `precision_bits` goes with `geohash` +only. Naming a column that is not in the frame does no harm; it is ignored. +A dictionary without `version`, or with a version this client does not +know, is ignored completely. ### DDL, DML, and cancellation @@ -844,7 +1122,7 @@ All failures raise `QuestDBError` (or a subclass). Inspect: | Property | Meaning | | --- | --- | | `code` | A `QuestDBErrorCode` member; compare by identity, e.g. `err.code is QuestDBErrorCode.Cancelled`. | -| `in_doubt` | `True` when the failed operation may already have delivered its input; retrying can duplicate rows without deduplication. | +| `in_doubt` | `True` when the failed ingestion operation may already have delivered its input. For QWP `dataframe()`, this includes earlier batches from the same call, even on a later validation error. Retrying can duplicate rows without deduplication. | | `sender_error` | Structured server diagnostic for QWP sender failures, or `None`. | Codes you will most often dispatch on: diff --git a/documentation/connect/clients/rust.md b/documentation/connect/clients/rust.md index 20fb11dbb..4e0a8dfc8 100644 --- a/documentation/connect/clients/rust.md +++ b/documentation/connect/clients/rust.md @@ -271,6 +271,18 @@ QWP cannot preserve nulls for `BOOLEAN`, `BYTE`, or `SHORT`. An absent value in one of those columns is received as `false` or `0`; use a wider nullable type when the distinction matters. +Everywhere in the API a UUID is 16 canonical RFC 4122 big-endian bytes — the +bytes `uuid::Uuid::as_bytes()` gives you — with one exception. `column_uuid` +on the row buffer takes the two 64-bit halves of the QWP wire encoding, +`(lo, hi)`, and is the only place you have to think about wire order. + +`Chunk::column_uuid` takes one 16-byte array per row, so wrap a single value +rather than splitting it yourself: + +```rust +chunk.column_uuid("trade_id", std::slice::from_ref(u.as_bytes()), None)?; +``` + ## Chunk ingestion {#sending-data-column-major} Use a `Chunk` when values already live in column slices. All columns and the @@ -356,8 +368,8 @@ needs an entry in `data`, which the encoder ignores. | `BOOLEAN` | `column_bool(name, bits, row_count, validity)` | LSB-first bit-packed values | | `TIMESTAMP`, `TIMESTAMP_NS` | `column_ts(name, data, TimestampUnit, validity)` | Epoch `i64` values | | `DATE` | `column_date` | Epoch milliseconds | -| `UUID` | `column_uuid` | `&[[u8; 16]]` in canonical RFC-4122 order | -| `LONG256` | `column_long256` | `&[[u8; 32]]` in little-endian limb order | +| `UUID` | `column_uuid` | `&[[u8; 16]]`, one 16-byte value per row, in canonical RFC 4122 big-endian order — each element is what `uuid::Uuid::as_bytes()` gives you | +| `LONG256` | `column_long256` | `&[[u8; 32]]` in little-endian limb order, least-significant limb first | | `IPv4` | `column_ipv4` | Host-order `u32` values | | `VARCHAR` | `column_str`, `column_str_large` | Arrow Utf8 offsets and bytes | | `BINARY` | `column_binary` | Arrow Binary offsets and bytes | @@ -426,7 +438,7 @@ db.flush_arrow_batch( "trades", &record_batch, None, // server-assigned designated timestamp - &[], // no Arrow column overrides + &[], // per-column wire-type overrides; see below Some(AckLevel::Ok), )?; ``` @@ -445,7 +457,10 @@ use questdb::ingress::{ ColumnName, }; -let overrides: [ArrowColumnOverride<'_>; 0] = []; +let overrides = [ + ArrowColumnOverride::Uuid { column: "trade_id" }, + ArrowColumnOverride::Long256 { column: "order_hash" }, +]; let options = PolarsIngestOptions::new() .max_rows(50_000) .timestamp_column(ColumnName::new("timestamp")?) @@ -455,8 +470,9 @@ let options = PolarsIngestOptions::new() db.flush_polars_dataframe("trades", &dataframe, &options)?; ``` -`max_rows(0)` uses the default batch size. Omitting `timestamp_column` asks the -server to assign timestamps. Omitting `ack_level` uses the pool default. +Pass `&[]` for `overrides` when no column needs one. `max_rows(0)` uses the +default batch size. Omitting `timestamp_column` asks the server to assign +timestamps. Omitting `ack_level` uses the pool default. `flush_polars_dataframe` checkpoints the frame and automatically retries the uncommitted tail after a transient failover. `flush_arrow_batch` returns a @@ -466,6 +482,52 @@ uncertain failure can also duplicate rows. Use [deduplication](/docs/concepts/deduplication/) when duplicates would be harmful. +On a failed `flush_polars_dataframe` call, `err.in_doubt()` covers the whole +DataFrame. It is true if any batch may have been delivered, including batches +confirmed by an earlier checkpoint, even when a later batch fails validation. +Internal retries and connection replacements retain this call-level status. +The flag is conservative: it does not identify a safe row offset for resuming +the load. A false flag does not make a validation error retryable; correct the +input first. Low-level Arrow and chunk flushes retain their current-operation +scope and do not aggregate earlier independent calls. + +### Binary columns: UUID, LONG256, and opaque bytes + +Binary columns land as `BINARY` unless the column claims a richer type, and a +byte width claims nothing on its own: a bare `FixedSizeBinary(16)` is opaque +bytes, not a UUID. A claim comes either from the Arrow schema — the +`ARROW:extension:name = arrow.uuid` extension label on `FixedSizeBinary(16)`, +or `questdb.column_type = uuid` / `= long256` field metadata — or from an +`ArrowColumnOverride::Uuid` / `::Long256` entry as above, which wins over any +metadata on that column. + +Polars has no fixed-size binary dtype, so it needs the override. Its `Binary` +columns export as Arrow `BinaryView`, and every non-null value must then be +exactly 16 or 32 bytes. UUID bytes are canonical RFC 4122 big-endian, and the +client byte-swaps them into wire order for you; LONG256 bytes are +little-endian limbs, low limb first, and are sent unchanged. A value of the +wrong width fails with `ErrorCode::ArrowIngest`. + +Polars `Object` columns are rejected outright. They export as +`FixedSizeBinary(8)` holding in-process handles, which is indistinguishable +from ordinary opaque binary once converted, so the client refuses them rather +than storing meaningless addresses. Cast the column to a supported dtype +first. + +:::caution Behaviour change + +Before client 7.0.0 a bare `FixedSizeBinary(16)` or `(32)` column became +`UUID` or `LONG256` on width alone. It is now `BINARY` unless the column +carries one of the claims above. A batch that used to produce a `UUID` column +now produces a `BINARY` one, with no error. + +The UUID byte order at the API boundary changed in the same release. Code +written against an earlier version passed wire-order bytes to +`Chunk::column_uuid` and `bind_uuid`; those values are now stored reversed, +also with no error. + +::: + ## Querying Borrow a reader, prepare SQL, bind values, execute, and pull typed batches: @@ -537,6 +599,11 @@ Available builders include: - `bind_geohash` - `bind_null` and the typed `bind_null_*` variants +`bind_uuid` takes 16 canonical RFC 4122 big-endian bytes by value, so pass +`*u.as_bytes()` or `u.into_bytes()`. `bind_long256` takes 32 little-endian +limb bytes, low limb first. Both are the byte orders the matching result +columns are read back in. + ### Reading columns `BatchView::column(index)` returns a non-exhaustive `ColumnView`. Match the @@ -549,7 +616,7 @@ variant before reading values: | `Symbol` | `resolve(row) -> Option<&str>` | | `Varchar` | `value(row) -> Option<&str>` | | `Binary` | `value(row) -> Option<&[u8]>` | -| `Uuid`, `Long256` | Fixed-size byte-array reference | +| `Uuid`, `Long256` | Fixed-size byte-array reference: `Uuid` yields 16 canonical RFC 4122 big-endian bytes, `Long256` 32 little-endian limb bytes, low limb first | | `Decimal64`, `Decimal128`, `Decimal256` | Integer value plus the column scale | | `Geohash` | Bits plus precision | | `DoubleArray`, `LongArray` | Per-row shape and element data | @@ -689,6 +756,11 @@ publishes or completes its first frame, so treat `None` from `acked_fsn()` as Recovery turns on `err.in_doubt()`, not on `err.code()` and not on whether the buffer or chunk still holds rows. +For these low-level APIs, the flag describes the failed operation's input. It +does not summarize earlier independent flushes or authorize replaying all data +since an application checkpoint. The DataFrame-level aggregation described +above applies specifically to `flush_polars_dataframe`. + When `in_doubt()` is `false`, the flush failed before the queue took the frame, so the rows never entered the send path and your input is intact: re-flush it. When `in_doubt()` is `true`, delivery is diff --git a/documentation/connect/wire-protocols/qwp-ingress-websocket.md b/documentation/connect/wire-protocols/qwp-ingress-websocket.md index e27c7ea5e..63adc3537 100644 --- a/documentation/connect/wire-protocols/qwp-ingress-websocket.md +++ b/documentation/connect/wire-protocols/qwp-ingress-websocket.md @@ -420,7 +420,7 @@ columns, which keeps every message self-contained for store-and-forward replay. | 9 | `0x09` | SYMBOL | var | Dictionary-encoded string | | 10 | `0x0A` | TIMESTAMP | 8 | Microseconds since Unix epoch | | 11 | `0x0B` | DATE | 8 | Milliseconds since Unix epoch | -| 12 | `0x0C` | UUID | 16 | RFC 4122 UUID | +| 12 | `0x0C` | UUID | 16 | UUID, two LE int64 halves, lo first | | 13 | `0x0D` | LONG256 | 32 | 256-bit integer | | 14 | `0x0E` | GEOHASH | var | Geospatial hash | | 15 | `0x0F` | VARCHAR | var | Length-prefixed UTF-8 | @@ -812,7 +812,15 @@ uncompressed mode. ### UUID 16 bytes per value: 8 bytes for the low 64 bits, then 8 bytes for the high -64 bits, both little-endian. +64 bits, both little-endian. Reverse all 16 bytes and you get the canonical +RFC 4122 order. + +The C, C++, Rust, and Python clients take RFC 4122 bytes in their APIs, so +they reverse each value on the way to the wire and again on the way back. The +one exception is the row-buffer call that takes the two 64-bit halves +directly — `line_sender_buffer_column_uuid(buffer, name, lo, hi, err)` and +its Rust equivalent — which is already in wire order and is passed through +unchanged. ### LONG256 diff --git a/documentation/high-availability/store-and-forward/concepts.md b/documentation/high-availability/store-and-forward/concepts.md index 6acafca19..6591c5bfc 100644 --- a/documentation/high-availability/store-and-forward/concepts.md +++ b/documentation/high-availability/store-and-forward/concepts.md @@ -62,8 +62,9 @@ Two distinct counters track frame identity: - **FSN** (frame-sequence-number) — a monotonic counter assigned when a frame is appended to the substrate. FSN survives reconnects and (in SF mode) restarts. It is the substrate's permanent identifier for a frame. -- **wireSeq** — the per-connection counter the server uses for - deduplication, reset to `0` on every successful WebSocket upgrade. +- **wireSeq** — a per-connection counter used to correlate cumulative QWP + responses with sent frames. It resets to `0` on every successful WebSocket + upgrade. On every (re)connect the relationship is pinned: @@ -82,9 +83,11 @@ Two consequences: - Frames **must** be sent in strict order. The wire format does not serialise `wireSeq` — the server assigns it implicitly from receive order. Reordering breaks the FSN mapping. -- After a reconnect, the server sees the **same payloads** at new - `wireSeq` values. Server-side dedup keys off `messageSequence` inside - the payload, not `wireSeq`, so replay does not produce double-writes. +- After a reconnect, the server sees the **same payloads** at new `wireSeq` + values. QWP requests contain neither `wireSeq` nor a persistent message ID, + so the protocol does not deduplicate those payloads across connections. If + the server committed a frame but its acknowledgement was lost, replay can + write the rows again. ## Trim: how unacked data is reclaimed @@ -151,8 +154,10 @@ On every successful (re)connect: 2. `wireSeq` resets to `0`. 3. The read cursor rewinds to the first un-acked frame on disk (or in memory). -4. Frames stream to the wire in FSN order. The server's dedup window - absorbs any frames that landed before the disconnect. +4. Frames stream to the wire in FSN order. A frame that landed before the + disconnect but was not acknowledged is sent again and can duplicate rows; + use table-level `DEDUP UPSERT KEYS` with stable row identity when this must + be suppressed. 5. New frames appended by the producer during replay are picked up automatically — the I/O loop watches a volatile `publishedFsn` cursor. diff --git a/documentation/high-availability/store-and-forward/when-to-use.md b/documentation/high-availability/store-and-forward/when-to-use.md index 8897ed6a9..8d5683825 100644 --- a/documentation/high-availability/store-and-forward/when-to-use.md +++ b/documentation/high-availability/store-and-forward/when-to-use.md @@ -170,7 +170,13 @@ If you are currently using HTTP or TCP ILP ingest, the comparison is: | Server outage tolerance | Best-effort retry | None | Reconnect loop with multi-minute budget | | Multi-host failover | Yes (HTTP only) | No | Yes | | Cross-region durability ack | No | No | Yes (`request_durable_ack=on`) | -| Cluster-wide ordering | Best-effort | Best-effort | FSN-driven, server-deduplicated | +| Cluster-wide ordering | Best-effort | Best-effort | FSN order within each sender stream, not across senders | + +QWP replay is at-least-once: FSNs track local progress but do not suppress +duplicate writes after a reconnect. Use table-level `DEDUP UPSERT KEYS` +with stable event timestamps and row identity to suppress replayed rows. +See [Delivery semantics](/docs/concepts/delivery-semantics/) for the +[deduplication](/docs/concepts/deduplication/) requirements. The transition is application-transparent — `Sender.fromConfig` accepts a `ws::` or `wss::` connect string and the public builder API is the