diff --git a/documentation/architecture/storage-engine.md b/documentation/architecture/storage-engine.md index 47c1985d5..1e0c83787 100644 --- a/documentation/architecture/storage-engine.md +++ b/documentation/architecture/storage-engine.md @@ -125,15 +125,31 @@ deduplicated on additional columns. ### Durability -By default, QuestDB relies on OS-level durability, letting the OS write dirty pages to disk. -For stronger guarantees, enable sync commit mode: +By default, QuestDB uses `nosync` commit mode and lets the OS write dirty pages +to disk. This provides the highest throughput, but an OS crash or power loss can +lose acknowledged writes. + +For WAL workloads that need local durability without flushing the whole +materialized table on every commit, use adaptive commit mode: ```ini title="server.conf" -cairo.commit.mode=sync +cairo.commit.mode=adaptive ``` -This invokes `fsync()` on each commit, ensuring data survives OS crashes or power loss -at the cost of reduced write throughput. +Adaptive mode makes the WAL authoritative, applies it lazily to table files, and +periodically creates a durable epoch. Recovery restores the latest valid epoch +and replays the durable WAL tail. Ordinary acknowledgements have a bounded RPO: +the configured group-commit window plus the background flush-sweep scheduling +delay. Set `cairo.adaptive.commit.group.window=0` or use QWP local durable +acknowledgements for a zero-loss acknowledgement boundary. + +Use `sync` when every commit to a non-WAL table, or every materialized-table +commit, must be locally durable. `async` schedules writeback without waiting and +does not provide an acknowledgement durability guarantee. + +See +[Cairo commit and write behavior](/docs/configuration/cairo-engine/#commit-and-write-behavior) +for a mode comparison, adaptive tuning, and rollback guidance. ## Next up diff --git a/documentation/configuration/cairo-engine.md b/documentation/configuration/cairo-engine.md index 8ebbf7f74..39a820f53 100644 --- a/documentation/configuration/cairo-engine.md +++ b/documentation/configuration/cairo-engine.md @@ -99,11 +99,158 @@ the same as for `query.timeout`. - **Default**: `nosync` - **Reloadable**: no -How changes are flushed to disk upon commit. Options: +Selects the instance-wide durability policy. It is not configurable per table. +Changing it requires a restart. + +| Mode | Commit behavior | Crash guarantee | Typical use | +| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | +| `nosync` | Does not explicitly flush table or WAL files. | A process crash usually leaves the OS page cache intact, but an OS crash or power loss can lose acknowledged writes. | Maximum throughput when the upstream source can replay data. This is the default. | +| `async` | Requests a flush after each commit but does not wait for it. | Reduces the dirty-data backlog but does not make an acknowledgement a durability boundary. | Workloads that want background writeback without synchronous commit latency. | +| `sync` | Flushes materialized table state and waits on every commit. | A returned commit is locally durable. | Non-WAL tables, or workloads that require the materialized table itself to be durable at every commit and can accept the throughput cost. | +| `adaptive` | Makes WAL data durable, batches sequencer flushes, applies WAL lazily, and periodically creates a durable epoch of materialized table state. | Ordinary commit acknowledgements have a bounded RPO: the group window plus the background flush-sweep scheduling delay. A QWP `local` durable acknowledgement is zero-loss. | WAL ingestion that needs local durability at substantially lower cost than `sync`. | + +`adaptive` separates the small, authoritative WAL from the larger materialized +table: + +1. QuestDB flushes each writer's private WAL data and event records. +2. It batches shared sequencer flushes for up to + [`cairo.adaptive.commit.group.window`](#cairoadaptivecommitgroupwindow). +3. WAL apply treats the table files, indexes, `_txn`, and `_cv` as a rebuildable + cache. +4. A durable epoch periodically flushes that cache. After an unclean shutdown, + QuestDB restores the last valid epoch and replays the durable WAL tail. + +The default 50 ms group window means a normal commit may return before its +sequencer record reaches the device. Set the window to `0` if every returned +commit must survive power loss; this gives zero-loss semantics with `sync`-class +per-commit latency. Alternatively, a QWP sender can request the +[`local` durable-ack tier](/docs/connect/wire-protocols/qwp-ingress-websocket/#durable-acknowledgement) +and retain its store-and-forward copy until QuestDB confirms the sequencer +record is durable. + +:::warning WAL tables only + +The `adaptive` recovery guarantee applies to WAL tables. A non-WAL table has no +WAL to replay, so its regular apply path has `nosync`-grade durability under +`adaptive`. Use `sync` when non-WAL commits must be locally durable. Structural +writes that cannot be reconstructed from WAL are flushed synchronously under +`adaptive`. -- `nosync`: no explicit flush (relies on OS page cache) -- `async`: flush call is scheduled but returns immediately -- `sync`: waits for flush on appended column files to complete +::: + +#### Choosing a mode + +- Keep `nosync` when throughput is the priority and producers can replay the + possible loss window. +- Use `async` to encourage writeback without paying synchronous latency, not as + a durability guarantee. +- Use `sync` for locally durable non-WAL writes or when every materialized-table + commit must be durable immediately. +- Use `adaptive` for WAL ingestion with a bounded RPO. Set its group window to + `0`, or wait for QWP local durable acknowledgements, when the relevant + acknowledgement must be zero-loss. +- Replication and local commit mode protect against different failures. Local + durability survives power loss; replicated QWP acknowledgements survive loss + of the server's disk or node. + +#### Migration and rollback + +Commit mode changes are supported across restarts. When a table first opens in +`adaptive` mode, QuestDB records that its materialized state may be ahead of its +last durable epoch. If the process crashes and you restart with another mode, +QuestDB still performs the required adaptive recovery before reconciling the +table to the new mode. Switching from `adaptive` to `nosync`, `async`, or `sync` +is therefore safe on a version that supports adaptive recovery. + +A **binary rollback** to a QuestDB version that predates `adaptive` is a +different operation. The new epoch and checksum sidecars are designed to be +ignored by older binaries, but the in-process compatibility tests do not replace +a real cross-version rollback matrix. In particular, a pre-adaptive binary +cannot recover a data directory left by an unclean adaptive shutdown. + +Before a binary rollback, use the adaptive-capable version to recover from any +unclean shutdown, restart it with a non-adaptive commit mode, allow every +adaptive WAL table to open for writing and reconcile to that mode, then stop it +cleanly and take a volume snapshot. Treat rollback to an older binary as +unverified unless the release notes for both versions explicitly support that +path. + +### cairo.adaptive.commit.group.window + +- **Default**: `50ms` +- **Reloadable**: no + +Maximum batching window for flushing adaptive sequencer records. The RPO for an +ordinary commit acknowledgement is bounded by this window plus the background +flush-sweep scheduling delay. `0` flushes every sequencer commit before +returning and gives zero-loss commit acknowledgements at higher latency. +Negative values are treated as `0`. + +This setting does not affect other commit modes. Materialized-view refresh WAL +continues to flush each commit synchronously. + +### cairo.adaptive.epoch.interval + +- **Default**: `60s` +- **Reloadable**: no + +Minimum time between durable materialized-state epochs for an adaptive table. +`0` takes an epoch after every WAL apply batch. A negative value disables both +time- and row-triggered epochs, causes unbounded WAL retention, and makes +recovery replay the WAL from its base; use that only for diagnostics or test +isolation. + +### cairo.adaptive.epoch.max.rows + +- **Default**: `5000000` +- **Reloadable**: no + +Takes a durable epoch after this many rows have been applied since the previous +epoch, even if the interval has not elapsed. This bounds retained WAL and +recovery replay under sustained ingestion. A value less than or equal to `0` +disables the row trigger while leaving the time trigger active. + +### cairo.adaptive.epoch.flush.on.close + +- **Default**: `true` +- **Reloadable**: no + +Makes a best-effort final durable epoch when an adaptive table writer closes +cleanly, including idle eviction and graceful shutdown. A negative +`cairo.adaptive.epoch.interval` disables epochs, including this close-time one. +Disable this setting only if the extra close-time I/O is unacceptable; the next +startup must then replay the tail since the previous epoch. + +### cairo.adaptive.epoch.column.sync.batched + +- **Default**: `true` +- **Reloadable**: no + +Uses a batched filesystem flush for adaptive epoch columns where the platform +supports it. QuestDB automatically disables this optimization on filesystems +where it cannot provide the required guarantee. Set it to `false` to force +per-file flushing. + +### cairo.adaptive.recovery.roll.forward.enabled + +- **Default**: `true` +- **Reloadable**: no + +Restores adaptive tables to their last valid durable epoch and replays the WAL +tail during startup. This is a recovery kill switch, not a way to accept weaker +recovery: when set to `false`, QuestDB refuses to start if an adaptive table +requires roll-forward. + +### cairo.wal.commit.writeback.drain + +- **Default**: `true` +- **Reloadable**: no + +Starts writeback across an adaptive WAL segment before taking the per-file +durability barriers. On supported filesystems this lets files write back in +parallel and reduces commit latency. It is only an optimization: QuestDB still +flushes every file, and filesystems that do not support effective range +writeback simply skip the drain. ### cairo.max.uncommitted.rows diff --git a/documentation/connect/clients/c-and-cpp.md b/documentation/connect/clients/c-and-cpp.md index 9b82a307c..b49b6c8a3 100644 --- a/documentation/connect/clients/c-and-cpp.md +++ b/documentation/connect/clients/c-and-cpp.md @@ -1191,15 +1191,18 @@ queue and returns before the server acks. 2. **`wait` = observation.** `qwp_sender_wait` (C++ `wait()`) blocks until everything published so far is acknowledged. - **Ack levels.** `qwpws_ack_level_ok` means the server accepted the - frames. `qwpws_ack_level_durable` additionally waits until they are - uploaded to object storage, not just in the server's WAL (Enterprise - with replication; see the protocol page's + frames. `qwpws_ack_level_local_durable` waits for local-disk durability + and requires `request_durable_ack=local`, WAL tables, and + `cairo.commit.mode=adaptive`. + `qwpws_ack_level_durable` waits for replicated/object-store durability + and requires `request_durable_ack=replicated` or the legacy alias `on`. + `local,replicated` is protocol-defined but current servers deny it (see + the protocol page's [durable acknowledgement](/docs/connect/wire-protocols/qwp-ingress-websocket/#durable-acknowledgement) - section). Durable acks must be requested at pool open with - `request_durable_ack=on`; the connect fails with - `protocol_version_error` when the server cannot provide them, and - without the key any durable-level `wait` or `flush_and_wait` fails up - front with `invalid_api_call`, leaving the buffer or chunk untouched. + section). An unavailable or partial grant fails the connect with + `protocol_version_error`; a level not selected by the connect string + fails `wait` or `flush_and_wait` up front with `invalid_api_call`, leaving + the buffer or chunk untouched. - **Ack is not visibility.** Rows become visible to queries after WAL apply, typically within milliseconds of the ack, so a query issued right after the ack can miss the newest rows. An empty read-back is @@ -1436,7 +1439,7 @@ Dispatch on `line_sender_error_get_code(err)` (C++ | `failover_retry` | Transient transport failure; frames may be in doubt | Dead — every later call fails | **Drop** the borrow, then re-borrow with `borrow_sender_with_retry(reconnect_max_duration_ms())`. With `sf_dir`, unresolved frames replay automatically. | | `server_rejection` | Server refused the data (schema/type conflict, bad name) | Dead — every later call fails | Plain **return** is safe; the pool retires the connection. Fix the data before re-sending; blind retry re-fails. | | `server_flush_error` | Backpressure deadline hit: queue full for `sf_append_deadline_millis` | Usable | Retry later, shed load, or raise the deadline. Nothing was dropped. See [backpressure](#durability-and-backpressure). | -| `invalid_api_call` | Borrow still at the pool cap after `acquire_timeout_ms`, pool closed, an operation on a borrow after close, or a durable-level wait without `request_durable_ack=on` | n/a | At-cap: treat as backpressure (see [Sizing the pool](#sizing-the-pool)). Closed pool: stop borrowing. | +| `invalid_api_call` | Borrow still at the pool cap after `acquire_timeout_ms`, pool closed, an operation on a borrow after close, or a durable-level wait without its matching `request_durable_ack` tier | n/a | At-cap: treat as backpressure (see [Sizing the pool](#sizing-the-pool)). Closed pool: stop borrowing. | If you are unsure which case you hit, **return is always safe**: the pool inspects the connection and retires it if unhealthy, so a broken connection diff --git a/documentation/connect/clients/connect-string.md b/documentation/connect/clients/connect-string.md index c027700dd..8f2d96a9b 100644 --- a/documentation/connect/clients/connect-string.md +++ b/documentation/connect/clients/connect-string.md @@ -665,35 +665,33 @@ Requires QuestDB Enterprise (multi-host). *Applies to: ingress.* -:::note QuestDB Enterprise - -Durable ACK requires QuestDB Enterprise. OSS is single-node and does not -ship WALs off-box, so the server-side durability-acknowledgement signal -that drives this protocol is enterprise-only. - -::: - -QuestDB Enterprise ships Write-Ahead Logs (WALs) from the primary to an -object store or another file system — typically over the network. After -durably shipping a WAL, the server emits a `STATUS_DURABLE_ACK` frame to -the store-and-forward client; the client marks that frame's FSN as durable -only after this acknowledgement arrives. - -The benefit: if the primary dies before shipping a WAL, the client still -holds the corresponding frames in its SF buffer and replays them against -the new primary on failover — closing the data-loss window that a -transport-level OK ACK alone cannot close. - -- `request_durable_ack` — when `on`, the client gates trim on - `STATUS_DURABLE_ACK` frames from the server, suppressing OK-driven trim. - Default: `off`. -- `durable_ack_keepalive_interval_millis` — interval at which the client - emits keepalive PINGs while waiting for durable-ack frames. Required - because the server only flushes pending durable acks on inbound recv - events. Default: `200` (ms). Set to `0` or a negative value to disable. - -See the [QWP Egress (WebSocket)](/docs/connect/wire-protocols/qwp-egress-websocket/) -wire protocol for the underlying mechanism. +Durable acknowledgements let a store-and-forward sender retain each frame until +QuestDB confirms the requested durability boundary: + +- `request_durable_ack=local` waits for `STATUS_LOCAL_DURABLE_ACK`. The WAL + transaction survives power loss on that server, but not loss of its disk. + This requires WAL tables and `cairo.commit.mode=adaptive`; a `nosync` server + can accept the handshake without producing local progress. +- `request_durable_ack=replicated` waits for `STATUS_DURABLE_ACK` after the WAL + reaches the configured object store. This requires QuestDB Enterprise with + replication. +- `request_durable_ack=local,replicated` requests both streams. It is + protocol-defined but not currently granted by servers. When supported, only + the stronger replicated stream may drive trim. +- `request_durable_ack=on` is the legacy alias for `replicated`. It retains the + original `true` request and `enabled` confirmation on the wire. +- `request_durable_ack=off` (the default) trims on ordinary OK responses. + +A server grants the complete requested tier set or fails the connection; it +never silently substitutes a weaker guarantee. + +`durable_ack_keepalive_interval_millis` controls how often the client sends a +WebSocket PING while durable work is pending. This is required because the +server emits pending durable progress only while handling inbound traffic. +Default: `200` ms. Set to `0` or a negative value to disable. + +See the [QWP ingress WebSocket protocol](/docs/connect/wire-protocols/qwp-ingress-websocket/#durable-acknowledgement) +for the wire-level contract. ## Query client keys {#egress-keys} @@ -901,7 +899,7 @@ description and behaviour notes. | `reconnect_initial_backoff_millis` | int (ms) | `100` | [Ingress reconnect](#reconnect-keys) | | `reconnect_max_backoff_millis` | int (ms) | `5000` | [Ingress reconnect](#reconnect-keys) | | `reconnect_max_duration_millis` | int (ms) | `300000` (5 min) | [Ingress reconnect](#reconnect-keys) | -| `request_durable_ack` | enum (`on` / `off`) | `off` | [Durable ACK](#durable-ack) | +| `request_durable_ack` | enum (`off` / `on` / `local` / `replicated` / `local,replicated`) | `off` | [Durable ACK](#durable-ack) | | `sender_id` | string | `default` | [Store-and-forward](#sf-keys) | | `sender_pool_max` | int | `4` | [Connection pool](#pool-keys) | | `sender_pool_min` | int | `1` | [Connection pool](#pool-keys) | diff --git a/documentation/connect/clients/java.md b/documentation/connect/clients/java.md index 1df7d8239..fb297be7b 100644 --- a/documentation/connect/clients/java.md +++ b/documentation/connect/clients/java.md @@ -836,21 +836,24 @@ borrow (see [Ingestion errors](#ingestion-errors)). ### Durable acknowledgement -:::note Enterprise - -Durable acknowledgement requires QuestDB Enterprise with primary replication -configured. - -::: - -By default, the server confirms a batch when it is committed to the local -[WAL](/docs/concepts/write-ahead-log/). To wait for the batch to be durably -uploaded to object storage: +By default, the server's OK response confirms a WAL commit but not a durable +storage boundary. A store-and-forward sender can retain its copy until the +requested tier is acknowledged: ```text -ws::addr=localhost:9000;sf_dir=/var/lib/questdb/sf;request_durable_ack=on; +# Local disk durability; requires adaptive commit mode +ws::addr=localhost:9000;sf_dir=/var/lib/questdb/sf;request_durable_ack=local; + +# Object-store durability; requires Enterprise replication +ws::addr=localhost:9000;sf_dir=/var/lib/questdb/sf;request_durable_ack=replicated; ``` +`request_durable_ack=on` remains the legacy alias for `replicated`. The Java +builder also accepts `requestDurableAck("local")`, +`requestDurableAck("replicated")`, and `requestDurableAck("local,replicated")`. +The combined tier is protocol-defined but current servers do not grant it. A +server must grant the complete requested set or the connection fails. + ### Awaiting acknowledgements `flush()` returns once the batch is handed to the send engine, not once the @@ -878,11 +881,12 @@ Related accessors: | `getAckedFsn()` | Highest FSN the server has acknowledged. `-1` if no batch has been published yet. | | `awaitAckedFsn(fsn, timeoutMillis)` | Block until `getAckedFsn()` reaches `fsn`, or the timeout elapses. | -When `request_durable_ack=on` is set, `getAckedFsn()` advances after the -durable upload to object storage, not on the ordinary commit ACK. The same -FSN span is reported on `SenderError.getFromFsn()` / `getToFsn()` for -rejected batches, so the value returned by `flushAndGetSequence()` is also -the correlation key for async error reports. +When a durable tier is requested, `getAckedFsn()` advances only after the +strongest requested tier covers the batch: local disk for `local`, or object +storage for `replicated` and the combined request. The same FSN span is reported +on `SenderError.getFromFsn()` / `getToFsn()` for rejected batches, so the value +returned by `flushAndGetSequence()` is also the correlation key for async error +reports. These methods are no-ops on transports that do not track frame sequence numbers (HTTP, TCP, UDP): `flushAndGetSequence()` and `getAckedFsn()` @@ -1586,7 +1590,7 @@ Common WebSocket-specific options: | `auto_flush_bytes` | disabled | Bytes before auto-flush. | | `sf_dir` | unset | Store-and-forward directory. | | `sender_id` | `default` | Sender slot identity for SF. | -| `request_durable_ack` | `off` | Request durable upload ACK (Enterprise). | +| `request_durable_ack` | `off` | Durable tier: `local`, `replicated`, `local,replicated`, legacy `on`, or `off`. | | `reconnect_max_duration_millis` | `300000` | Sync initial-connect budget only. | | `failover` | `on` | Egress per-query reconnect switch. | | `compression` | `raw` | Egress batch compression (`raw`, `zstd`, `auto`). | diff --git a/documentation/connect/clients/python.md b/documentation/connect/clients/python.md index be8d79aa2..d33a18296 100644 --- a/documentation/connect/clients/python.md +++ b/documentation/connect/clients/python.md @@ -642,15 +642,18 @@ A `wait()` timeout is a no-progress timeout: the data remains queued and background delivery continues, so retry `wait()` rather than flushing the same rows again. -`flush(wait=True)` and `wait()` observe the accepted (`Ok`) acknowledgement: -the server took responsibility for the frames. They are pure barriers — -data-fate notification is a separate channel, the -[rejection handler](#server-rejections). A durable-level wait (waiting -for object-storage upload on Enterprise deployments) is not exposed on the -pooled Python API. The `request_durable_ack=on` connect key is accepted, and -a server without durable-ack support rejects the first operation with -`QuestDBErrorCode.ProtocolVersionError`; see the -[connect string reference](/docs/connect/clients/connect-string/). +`flush(wait=True)`, `wait()`, and the acknowledged FSN use the connection's +configured barrier. With the default `request_durable_ack=off`, this is the +accepted (`Ok`) acknowledgement. `request_durable_ack=local` waits for +local-disk durability and requires WAL tables with `cairo.commit.mode=adaptive`; +`replicated` or the legacy alias `on` waits for the replication/object-store +boundary. `local,replicated` is accepted by the client but current servers deny +that combined request. The pooled Python API does not select a different tier +per call. A server that cannot grant the complete request fails +the connection with `QuestDBErrorCode.ProtocolVersionError`; see the +[connect string reference](/docs/connect/clients/connect-string/). Data-fate +notification remains a separate channel, the +[rejection handler](#server-rejections). Store-and-forward is bounded. When producers continuously outrun the server, publication can wait for ack-driven space and then raise diff --git a/documentation/connect/clients/rust.md b/documentation/connect/clients/rust.md index 20fb11dbb..3ba4bd929 100644 --- a/documentation/connect/clients/rust.md +++ b/documentation/connect/clients/rust.md @@ -433,7 +433,10 @@ db.flush_arrow_batch( Pass `Some(ColumnName)` instead of `None` to source the designated timestamp from an Arrow timestamp column. Passing `None` for the ACK level selects -`Durable` when `request_durable_ack=on`, otherwise `Ok`. +`LocalDurable` for `request_durable_ack=local`, `Durable` for `on`, +`replicated`, or `local,replicated`, and `Ok` for `off`. Local durability also +requires WAL tables with `cairo.commit.mode=adaptive`; current servers deny the +combined request. For Polars, configure ingestion with `PolarsIngestOptions`: @@ -666,9 +669,13 @@ rejected at build time. | Non-blocking progress | Buffer: `flush_buffer_and_get_fsn`; chunk: `flush_and_get_fsn`; then `acked_fsn` | Observe a boundary while retaining the same borrow. | `AckLevel::Ok` means the server accepted all frames through the boundary. -`AckLevel::Durable` requires `request_durable_ack=on` and Enterprise server -support. Requesting `Durable` without opting in is rejected before the buffer -or chunk is changed. +`AckLevel::LocalDurable` waits for the server's local disk and requires +`request_durable_ack=local`, WAL tables, and `cairo.commit.mode=adaptive`. +`AckLevel::Durable` waits for the replication/object-store boundary and requires +`request_durable_ack=on` or `replicated`. `local,replicated` is accepted by the +client but current servers deny the combined request. Requesting a level not +selected by the connect string is rejected before the buffer or chunk is +changed. A `wait` timeout is a no-progress timeout. The data remains queued and its background delivery continues. Retry `wait()` or keep observing `acked_fsn()`; diff --git a/documentation/connect/wire-protocols/qwp-ingress-websocket.md b/documentation/connect/wire-protocols/qwp-ingress-websocket.md index e27c7ea5e..c2504a5c6 100644 --- a/documentation/connect/wire-protocols/qwp-ingress-websocket.md +++ b/documentation/connect/wire-protocols/qwp-ingress-websocket.md @@ -51,9 +51,10 @@ both ends: - **Multi-table batches.** A single WebSocket frame can carry rows for many tables in one trip across the wire. - **Server-acknowledged commits.** Every batch gets an OK frame carrying the - per-table sequencer transaction it landed in, so the client knows - precisely what's durable. An optional `X-QWP-Request-Durable-Ack` opt-in - on the upgrade extends this to cluster-durable acks (Enterprise only). + per-table sequencer transaction it landed in. An optional + `X-QWP-Request-Durable-Ack` upgrade header adds cumulative local-disk and/or + replicated durability watermarks, so a store-and-forward client knows when + it can discard its copy. A minimum-viable client that supports BOOLEAN, LONG, DOUBLE, TIMESTAMP, and VARCHAR — the five types that cover most real workloads — is on the order of @@ -102,22 +103,29 @@ using custom headers. **Client request headers:** -| Header | Required | Description | -|---------------------|----------|--------------------------------------------------------------------------------------| -| `X-QWP-Max-Version` | No | Maximum QWP version the client supports (positive integer). Defaults to 1 if absent. | -| `X-QWP-Client-Id` | No | Free-form client identifier (e.g., `java/1.0.2`, `zig/0.1.0`). | +| Header | Required | Description | +| --------------------------- | -------- | ----------------------------------------------------------------------------------------------------------- | +| `X-QWP-Max-Version` | No | Maximum QWP version the client supports (positive integer). Defaults to 1 if absent. | +| `X-QWP-Client-Id` | No | Free-form client identifier (e.g., `java/1.0.2`, `zig/0.1.0`). | +| `X-QWP-Request-Durable-Ack` | No | Durable-ack tier request: `true` (legacy replicated request), `local`, `replicated`, or `local,replicated`. | **Server response headers:** -| Header | Description | -|------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `X-QWP-Version` | The QWP version selected for this connection. | +| Header | Description | +| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `X-QWP-Version` | The QWP version selected for this connection. | | `X-QWP-Max-Batch-Size` | Server's effective per-message payload cap in bytes, computed as `min(http.recv.buffer.size − 14, 16 MiB)` (the protocol ceiling clamped by the actual WebSocket recv buffer minus the worst-case frame header). Clients should size batches to stay under this value. Absent on servers older than the introduction of this header — clients fall back to their locally configured byte budget. | -| `X-QWP-Durable-Ack` | `enabled` when the connection will emit `STATUS_DURABLE_ACK` frames. Sent only when the client opted in via `X-QWP-Request-Durable-Ack: true` *and* the server has durable-ack support configured. Absent in every other case. | +| `X-QWP-Durable-Ack` | Exact durable-ack grant. The legacy `true` request is confirmed as `enabled`; explicit requests are confirmed as `local`, `replicated`, or `local,replicated`. The server omits the header when it cannot grant the complete request. | The server selects the version as `min(clientMax, serverMax)`. The selected -version is never higher than either side's maximum. The server may also -consider the `X-QWP-Client-Id` when selecting the version. +version is never higher than either side's maximum. The server may also consider +the `X-QWP-Client-Id` when selecting the version. + +Browser clients can offer the `questdb.qwp.durable-ack.v1` WebSocket +subprotocol. Because a subprotocol token cannot carry a tier parameter, it has +the same legacy replicated semantics as `X-QWP-Request-Durable-Ack: true`. The +server echoes the subprotocol even when it cannot grant that tier; capability is +confirmed only by `X-QWP-Durable-Ack: enabled`. ### Connection-level contract @@ -157,38 +165,45 @@ The end-to-end shape of a QWP client session, before the encoding details: - `X-QWP-Max-Version: 1` — highest version supported. - `X-QWP-Client-Id: /` — recommended, helps server-side diagnostics and version negotiation. - - Authentication header (`Authorization: Basic …` or `Authorization: Bearer …`). - - `X-QWP-Request-Durable-Ack: true` — optional, opt-in for cluster-durable - acks (Enterprise). + - Authentication header (`Authorization: Basic …` or + `Authorization: Bearer …`). + - `X-QWP-Request-Durable-Ack: ` — optional. Request `local`, + `replicated`, or `local,replicated`. The legacy value `true` requests the + replicated tier. 2. **Verify the upgrade.** On `101 Switching Protocols`, read the response headers: - `X-QWP-Version` — the version the connection runs on. Use it for the - `version` byte in every outgoing message header. Reject the connection - if it's outside the range your client supports. - - `X-QWP-Durable-Ack: enabled` — confirms durable-ack frames will follow, - iff you opted in. If you opted in and this header is absent, fail the - connection (don't silently wait for acks the server will never send). + `version` byte in every outgoing message header. Reject the connection if + it's outside the range your client supports. + - `X-QWP-Durable-Ack` — must exactly confirm the requested tier set. A legacy + `true` request expects `enabled`; explicit requests expect their canonical + tier token. If the header is absent, partial, or different, fail the + connection rather than silently weakening the guarantee. - `X-QWP-Max-Batch-Size` (optional, older servers omit it) — server's effective per-message payload cap in bytes. Clients should clamp their batch-size triggers to fit under this value (a safety margin of ~10% absorbs encoding overhead such as schema and dict-delta bytes). When - absent, fall back to a locally configured budget or a conservative - default such as 1.9 MiB to stay under the typical 2 MiB recv buffer. -3. **Send binary frames.** Each frame is one QWP message: - `12-byte header` + payload (`Delta Symbol Dictionary` if any, then one or - more `Table Block`s). Each table block carries its column schema inline. + absent, fall back to a locally configured budget or a conservative default + such as 1.9 MiB to stay under the typical 2 MiB recv buffer. +3. **Send binary frames.** Each frame is one QWP message: `12-byte header` + + payload (`Delta Symbol Dictionary` if any, then one or more `Table Block`s). + Each table block carries its column schema inline. 4. **Drain server responses.** The server sends an OK (or error) binary frame - per request, in send order. Match responses to requests by their position - in your in-flight queue — the server-assigned `sequence` field in each - response is the authoritative confirmation. If you opted in to durable - ack, you'll also receive periodic `STATUS_DURABLE_ACK` frames carrying - cumulative per-table watermarks. -5. **Close.** Send a WebSocket `Close` frame after the last expected OK has - been drained. + per request, in send order. Match responses to requests by their position in + your in-flight queue — the server-assigned `sequence` field in each response + is the authoritative confirmation. If you opted in to durable + acknowledgements, continue until the requested-tier watermarks cover every + committed table `seqTxn`. The server sends pending durable progress only + while processing inbound traffic, so an idle client must send periodic + WebSocket PINGs (200 ms is the client default) and keep draining + `STATUS_LOCAL_DURABLE_ACK` and/or `STATUS_DURABLE_ACK` frames. +5. **Close.** Without durable acknowledgement, close after the final expected OK + is drained. With durable acknowledgement, close only after the applicable + watermarks cover the final committed batch. Every reconnect resets connection-scoped state on both sides: the symbol -dictionary and sequence counter. Clients that want sender-restart -durability layer a store-and-forward buffer on top — see the +dictionary and sequence counter. Clients that want sender-restart durability +layer a store-and-forward buffer on top — see the [connect string reference](/docs/connect/clients/connect-string#sf-keys). ## Encoding primitives @@ -977,6 +992,7 @@ table that committed data in the acknowledged batch. `tableCount` is 0 when no | 11 | `0x0B` | LIMIT_EXCEEDED | Egress-only. Query aborted because a server-side limit was hit: query timeout, memory cap, circuit breaker, or OOM. | | 12 | `0x0C` | NOT_WRITABLE | **Reserved.** Node cannot accept writes (read-only replica, or a demoting primary). | | 13 | `0x0D` | DICTIONARY_GAP | A delta symbol dictionary whose start id runs past the server's connection dictionary. | +| 14 | `0x0E` | LOCAL_DURABLE_ACK | Batch WAL is durable on the server's local disk. | The status namespace is shared between ingress and egress, which is why the two egress-only codes appear here. @@ -995,53 +1011,81 @@ Two of these carry classification instructions a client cannot infer: ### Durable acknowledgement -:::note Enterprise +A standard OK confirms that the batch was committed to the server's WAL and +reports its per-table sequencer transaction. It is not, by itself, a promise +that the transaction has reached durable storage. -Durable acknowledgement (status code 0x02) is available in QuestDB Enterprise -with primary replication configured. Open source QuestDB returns OK (0x00) or -error responses only. +A client can request one or both durability tiers during the WebSocket upgrade: -::: +| Request value | Confirmation | Status stream | Guarantee | +| ------------------ | ------------------ | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `local` | `local` | `LOCAL_DURABLE_ACK` (`0x0E`) | The WAL transaction and sequencer record are durable on the server's disk. Survives process, OS, and power failure, but not loss of that disk. Local progress is produced for WAL tables using `adaptive` commit mode. | +| `replicated` | `replicated` | `DURABLE_ACK` (`0x02`) | The transaction has reached the configured object store and can survive loss of the server's disk or node. Requires a replication-capable QuestDB Enterprise configuration. | +| `local,replicated` | `local,replicated` | Both | Protocol-defined combined request. Local frames provide earlier progress, while replicated frames provide the stronger guarantee. Current servers do not yet grant this combination. | +| `true` | `enabled` | `DURABLE_ACK` (`0x02`) | Legacy alias for `replicated`, retained byte-for-byte for existing clients and servers. | -A standard OK confirms the batch was committed to the server's local WAL. To -receive a second acknowledgement after the WAL has been durably uploaded to the -configured object store, include `X-QWP-Request-Durable-Ack: true` -(case-insensitive) in the WebSocket upgrade request. +Header values are case-insensitive. Servers also accept `replicated,local`, but +clients should send and expect the canonical `local,replicated` spelling. -If the server accepts the opt-in, it echoes `X-QWP-Durable-Ack: enabled` in -the 101 response. Clients that opt in **must** verify this header is present -and fail the connect attempt if it is absent. +The grant is **all-or-nothing**. The server confirms the full requested set or +omits `X-QWP-Durable-Ack`; it never substitutes a weaker tier. A client that +requested durable acknowledgement must compare the response with the expected +confirmation and fail the connection on a missing, partial, or different value. +This intentionally makes explicit tier requests fail against servers that +predate tier negotiation. The legacy `true`/`enabled` exchange remains +compatible with those servers. -**Durable-ack response format:** +Tier availability depends on server configuration. Open source QuestDB can grant +`local`; it denies requests containing `replicated`. Enterprise can grant +`replicated` when replication is configured. `local,replicated` is reserved for +a server that can provide both streams; current servers deny the combined +request rather than granting only one tier. + +:::warning Local acknowledgements require adaptive commit mode + +The server can confirm a `local` handshake independently of table commit mode, +but local durable watermarks advance only for WAL tables using +[`cairo.commit.mode=adaptive`](/docs/configuration/cairo-engine/#cairocommitmode). +Because the server default is `nosync`, requesting `local` without enabling +`adaptive` connects successfully but produces no local durable-ack progress. + +::: + +Both status codes use the same response layout: ```text +------------------------------------------------------+ -| status: uint8 (0x02) | +| status: uint8 (0x02 or 0x0E) | | tableCount: uint16 Number of table entries | | Repeated tableCount times: | | nameLen: uint16 Table name length | | name: bytes UTF-8 table name | -| seqTxn: int64 Durably-uploaded seqTxn | +| seqTxn: int64 Durable seqTxn | +------------------------------------------------------+ ``` -The durable-ack has no sequence field. It carries cumulative per-table -watermarks that advance as uploads complete. Only tables whose durable -watermark advanced since the last durable-ack are included. +Durable-ack frames have no request sequence field. They carry cumulative +per-table watermarks and include only tables whose watermark has advanced since +the previous frame in that stream. -The durable-ack watermark always trails the regular OK watermark. Empty -messages (those that produced no WAL commit, for example messages that only -reference materialized views) are trivially durable; their sequence advances -the durable watermark as soon as all preceding messages are durable. +A store-and-forward sender must trim on the strongest requested guarantee: -Reconnects discard any in-flight durable-ack tracking. The new connection -re-OKs replayed batches and the server re-emits cumulative durable-ack -watermarks from scratch, so the client's trim watermark must restart against -the new connection's wire sequencing. +- With `local` only, `LOCAL_DURABLE_ACK` advances the trim watermark. +- With `replicated` only, `DURABLE_ACK` advances the trim watermark. +- With both tiers, local frames are progress signals only. The sender must keep + its copy until the corresponding replicated watermark arrives. + +This rule prevents a combined request from being silently weakened to local +storage. The applicable durable watermark trails the regular OK watermark. Empty +messages (those that produced no WAL commit, for example messages that only +reference materialized views) are trivially durable once all preceding messages +reach the requested tier. -Servers without replication silently ignore the request header and never emit -durable-ack frames. There is no durable-failure status; persistent upload -failures surface only as absence of a durable-ack frame. +Reconnects discard connection-local durable-ack tracking. The new connection +re-OKs replayed batches and the server re-emits cumulative durable watermarks, +so clients must rebuild their trim state against the new connection's request +ordering. There is no durable-failure status; a stalled local flush or upload +appears as the absence of further durable-ack progress. ## Protocol limits @@ -1296,15 +1340,15 @@ XX XX XX XX # Payload length ## Reference implementation The reference client implementation is -[`java-questdb-client`](https://github.com/questdb/java-questdb-client) -at commit -[`67bb5e4`](https://github.com/questdb/java-questdb-client/commit/67bb5e49feea7e63b813ea08189c23ea11486131). +[`java-questdb-client`](https://github.com/questdb/java-questdb-client) at +commit +[`c329caa`](https://github.com/questdb/java-questdb-client/commit/c329caa3fc4b3015b160744eebc9995bcc812424). The server-side protocol parser lives in the QuestDB server repository under `core/src/main/java/io/questdb/cutlass/qwp/protocol/`. ## Version history -| Version | Description | -|------------|---------------------------------| -| 1 (`0x01`) | Initial binary protocol release | +| Version | Description | +| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 (`0x01`) | Initial binary protocol release. Optional capabilities, including tiered durable acknowledgements, are negotiated with WebSocket upgrade headers and do not change the message version. | diff --git a/documentation/high-availability/store-and-forward/concepts.md b/documentation/high-availability/store-and-forward/concepts.md index 6acafca19..e0d31c995 100644 --- a/documentation/high-availability/store-and-forward/concepts.md +++ b/documentation/high-availability/store-and-forward/concepts.md @@ -106,33 +106,32 @@ batch updated. On receipt: 3. Any segment whose last FSN is `≤ ackedFsn` is unlinked and its bytes returned to the available pool. -This is the default and is sufficient when "data is in the server's WAL" -is the durability bar you need. +This is the default and is sufficient when a WAL commit, without a durable +storage guarantee, is the acknowledgement bar you need. -### `request_durable_ack=on` — WAL-durable trim +### Tiered durable trim -When the connect string sets `request_durable_ack=on`, trim is driven by -a separate frame: `STATUS_DURABLE_ACK`. These carry per-table watermarks -for data the server has **already uploaded from the WAL to the configured -object store** (S3, Azure Blob, GCS, or NFS). +`request_durable_ack=local` drives trim with `STATUS_LOCAL_DURABLE_ACK` +watermarks after the WAL transaction is durable on the server's disk. This +requires adaptive commit mode. `request_durable_ack=replicated` drives trim with +`STATUS_DURABLE_ACK` after the WAL reaches the configured object store (S3, +Azure Blob, GCS, or NFS). The legacy value `on` means `replicated`. -- OK frames still arrive on every batch, but they no longer advance the - trim watermark. Instead, they are stashed alongside their per-table - `seqTxn` values. -- A `STATUS_DURABLE_ACK` frame names tables and their durable `seqTxn` +- OK frames still arrive on every batch, but they no longer advance the trim + watermark. Instead, they are stashed alongside their per-table `seqTxn` + values. +- The applicable durable frame names tables and their cumulative `seqTxn` watermarks. The client matches the head of the OK queue against these - watermarks; each fully-covered head entry pops, and `ackedFsn` - advances to the highest covered wireSeq. -- The client opt-in is mandatory — the connect fails loudly if the server - does not echo `X-QWP-Durable-Ack: enabled` on the upgrade response. - This avoids the silent failure mode where the producer waits forever - for ack frames that will never arrive. - -Durable-ack mode is the right choice when "data is in the object store" -is the durability bar, but it has two costs: a longer time-to-trim (so -larger steady-state disk usage in SF mode), and a small WebSocket PING -sent every `durable_ack_keepalive_interval_millis` to nudge the server's -flush path when the client is idle but has pending confirmations. + watermarks; each fully-covered head entry pops, and `ackedFsn` advances to the + highest covered wire sequence. +- The server must echo the complete requested tier set. Missing or partial + confirmation fails the connection instead of silently weakening the + guarantee. + +Durable-ack mode has two costs: a longer time-to-trim, and therefore larger +steady-state SF storage use, plus a small WebSocket PING every +`durable_ack_keepalive_interval_millis` to prompt the passive server when the +client is idle but has pending confirmations. See [When to use](/docs/high-availability/store-and-forward/when-to-use/) for the decision. diff --git a/documentation/high-availability/store-and-forward/configuration.md b/documentation/high-availability/store-and-forward/configuration.md index 6d11a1330..137c21a11 100644 --- a/documentation/high-availability/store-and-forward/configuration.md +++ b/documentation/high-availability/store-and-forward/configuration.md @@ -59,12 +59,12 @@ Cross-reference: ## Durable-ack keys -Opt in to object-store-durable trim. See +Choose the local-disk or replicated durability boundary that drives trim. See [Durable-ack: when to opt in](/docs/high-availability/store-and-forward/when-to-use/#durable-ack-when-to-opt-in). | Key | Type | Default | Description | |---|---|---|---| -| `request_durable_ack` | bool | `off` | Opt-in via the upgrade header `X-QWP-Request-Durable-Ack: true`. Trim is then driven by `STATUS_DURABLE_ACK` frames only; OK frames no longer advance the trim watermark. Connect fails loudly if the server does not echo `X-QWP-Durable-Ack: enabled`. WebSocket transports only. | +| `request_durable_ack` | enum: `off`, `on`, `local`, `replicated`, `local,replicated` | `off` | `local` trims on local-disk durable acknowledgements and requires adaptive commit mode. `replicated` trims after object-store replication. `on` is the legacy replicated alias. The combined value is protocol-defined but current servers do not grant it. A missing or partial grant fails the connection. WebSocket transports only. | | `durable_ack_keepalive_interval_millis` | int (ms) | `200` | Cadence of WebSocket PING the I/O loop sends while there are pending durable confirmations and the producer is idle. `0` or negative disables. | ## Error-handling keys @@ -109,7 +109,7 @@ The parser rejects: - `sf_durability` values other than `memory`, `flush`, `append`. `flush` and `append` parse but are rejected at build time today. - `sender_id` containing path separators or empty. -- `request_durable_ack=on` on non-WebSocket transports. +- Any non-`off` `request_durable_ack` value on non-WebSocket transports. ## Worked examples diff --git a/documentation/high-availability/store-and-forward/when-to-use.md b/documentation/high-availability/store-and-forward/when-to-use.md index 8897ed6a9..c36b706ec 100644 --- a/documentation/high-availability/store-and-forward/when-to-use.md +++ b/documentation/high-availability/store-and-forward/when-to-use.md @@ -73,51 +73,47 @@ string. ## Durable-ack: when to opt in -By default the substrate trims unacked data on OK ack from the server. -That means the substrate releases a frame once the server has acknowledged -it into the WAL. The frame is durable on the **primary's** disk; whether -it has been replicated to the object store or to replicas is a separate -matter. +By default the substrate trims unacked data after the server's OK response. +An OK confirms a WAL commit, but is not itself a promise that the transaction +has reached durable storage. -When the connect string sets `request_durable_ack=on`, trim is held back -until a separate `STATUS_DURABLE_ACK` frame confirms the data has been -uploaded from the WAL to the **configured object store** (S3, Azure Blob, -GCS, or NFS). +Set `request_durable_ack=local` to retain the frame until QuestDB confirms that +the WAL transaction is durable on the primary's disk. This requires a WAL table +and `cairo.commit.mode=adaptive`. Set `request_durable_ack=replicated` (or its +legacy alias, `on`) to retain the frame until `STATUS_DURABLE_ACK` confirms that +the WAL reached the configured object store (S3, Azure Blob, GCS, or NFS). ### Choose durable-ack when -- You require object-store durability before considering a write - acknowledged — e.g. compliance requirements, end-to-end exactly-once - pipelines with cross-region recovery. -- Loss of an entire primary node (and its local disk) must not lose - in-flight data — replicas haven't downloaded the WAL yet, only the - object store has. -- You are willing to trade later trim (and so larger steady-state SF - disk usage) for the stronger guarantee. +- Use `local` when power-loss-safe durability on one server is sufficient and + you want to retain the client copy until that boundary. +- Use `replicated` when loss of the primary and its disk must not lose in-flight + data, or for compliance and cross-region recovery requirements. +- You are willing to trade later trim, and therefore larger steady-state SF + storage use, for the selected guarantee. ### Stay on the default OK trim when -- WAL-local durability on the primary is sufficient. -- You want minimum steady-state disk usage. -- You are running OSS or a build that does not support durable-ack. - (The handshake fails loudly if you opt in but the server cannot - deliver — see below.) +- Your upstream source can replay the server's local durability window. +- You want minimum steady-state storage use. +- The server does not support the durability tier you require. ### Caveats -- **Server support is required.** The client sends - `X-QWP-Request-Durable-Ack: true` on the upgrade. The server must echo - back `X-QWP-Durable-Ack: enabled`. If it does not — OSS build, - uninitialised primary, missing registry, hitting a replica — the - connect **fails loudly**, by design. Silently waiting for ack frames - that never arrive would let the SF disk fill up. -- **Idle keepalive.** The OSS server only flushes pending durable-ack - frames during inbound recv events. The client sends a WebSocket PING - every `durable_ack_keepalive_interval_millis` (default 200 ms) when - there are pending confirmations and the producer is idle. +- **Server support is required.** The server must echo the complete requested + tier set in `X-QWP-Durable-Ack`. A missing, partial, or different grant makes + the connection fail loudly. The legacy `on` request uses `true` on the wire + and expects `enabled`. +- **Local mode prerequisite.** OSS can grant `local`, but the watermark advances + only for adaptive WAL tables. Since `nosync` is the server default, enabling + `local` without adaptive mode can leave the sender waiting indefinitely. +- **Idle keepalive.** The server only flushes pending durable-ack frames during + inbound receive events. The client sends a WebSocket PING every + `durable_ack_keepalive_interval_millis` (default 200 ms) while confirmations + are pending and the producer is idle. - **Disk pressure.** Steady-state SF disk usage is roughly - `ingest_rate × time_to_object_store_durability`. Size - `sf_max_total_bytes` accordingly. + `ingest_rate × time_to_requested_durability`. Size `sf_max_total_bytes` + accordingly. ## Orphan adoption: when to enable