Skip to content

docs: explain adaptive commits and durable ack tiers #559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nwoolmer wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: explain adaptive commits and durable ack tiers #559

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 21 additions & 5 deletions documentation/architecture/storage-engine.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,15 +125,31 @@ deduplicated on additional columns.

### Durability

By default, QuestDB relies on OS-level durability, letting the OS write dirty pages to disk.
For stronger guarantees, enable sync commit mode:
By default, QuestDB uses `nosync` commit mode and lets the OS write dirty pages
to disk. This provides the highest throughput, but an OS crash or power loss can
lose acknowledged writes.

For WAL workloads that need local durability without flushing the whole
materialized table on every commit, use adaptive commit mode:

```ini title="server.conf"
cairo.commit.mode=sync
cairo.commit.mode=adaptive
```

This invokes `fsync()` on each commit, ensuring data survives OS crashes or power loss
at the cost of reduced write throughput.
Adaptive mode makes the WAL authoritative, applies it lazily to table files, and
periodically creates a durable epoch. Recovery restores the latest valid epoch
and replays the durable WAL tail. Ordinary acknowledgements have a bounded RPO:
the configured group-commit window plus the background flush-sweep scheduling
delay. Set `cairo.adaptive.commit.group.window=0` or use QWP local durable
acknowledgements for a zero-loss acknowledgement boundary.

Use `sync` when every commit to a non-WAL table, or every materialized-table
commit, must be locally durable. `async` schedules writeback without waiting and
does not provide an acknowledgement durability guarantee.

See
[Cairo commit and write behavior](/docs/configuration/cairo-engine/#commit-and-write-behavior)
for a mode comparison, adaptive tuning, and rollback guidance.

## Next up

Expand Down
155 changes: 151 additions & 4 deletions documentation/configuration/cairo-engine.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,11 +99,158 @@ the same as for `query.timeout`.
- **Default**: `nosync`
- **Reloadable**: no

How changes are flushed to disk upon commit. Options:
Selects the instance-wide durability policy. It is not configurable per table.
Changing it requires a restart.

| Mode | Commit behavior | Crash guarantee | Typical use |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `nosync` | Does not explicitly flush table or WAL files. | A process crash usually leaves the OS page cache intact, but an OS crash or power loss can lose acknowledged writes. | Maximum throughput when the upstream source can replay data. This is the default. |
| `async` | Requests a flush after each commit but does not wait for it. | Reduces the dirty-data backlog but does not make an acknowledgement a durability boundary. | Workloads that want background writeback without synchronous commit latency. |
| `sync` | Flushes materialized table state and waits on every commit. | A returned commit is locally durable. | Non-WAL tables, or workloads that require the materialized table itself to be durable at every commit and can accept the throughput cost. |
| `adaptive` | Makes WAL data durable, batches sequencer flushes, applies WAL lazily, and periodically creates a durable epoch of materialized table state. | Ordinary commit acknowledgements have a bounded RPO: the group window plus the background flush-sweep scheduling delay. A QWP `local` durable acknowledgement is zero-loss. | WAL ingestion that needs local durability at substantially lower cost than `sync`. |

`adaptive` separates the small, authoritative WAL from the larger materialized
table:

1. QuestDB flushes each writer's private WAL data and event records.
2. It batches shared sequencer flushes for up to
[`cairo.adaptive.commit.group.window`](#cairoadaptivecommitgroupwindow).
3. WAL apply treats the table files, indexes, `_txn`, and `_cv` as a rebuildable
cache.
4. A durable epoch periodically flushes that cache. After an unclean shutdown,
QuestDB restores the last valid epoch and replays the durable WAL tail.

The default 50 ms group window means a normal commit may return before its
sequencer record reaches the device. Set the window to `0` if every returned
commit must survive power loss; this gives zero-loss semantics with `sync`-class
per-commit latency. Alternatively, a QWP sender can request the
[`local` durable-ack tier](/docs/connect/wire-protocols/qwp-ingress-websocket/#durable-acknowledgement)
and retain its store-and-forward copy until QuestDB confirms the sequencer
record is durable.

:::warning WAL tables only

The `adaptive` recovery guarantee applies to WAL tables. A non-WAL table has no
WAL to replay, so its regular apply path has `nosync`-grade durability under
`adaptive`. Use `sync` when non-WAL commits must be locally durable. Structural
writes that cannot be reconstructed from WAL are flushed synchronously under
`adaptive`.

- `nosync`: no explicit flush (relies on OS page cache)
- `async`: flush call is scheduled but returns immediately
- `sync`: waits for flush on appended column files to complete
:::

#### Choosing a mode

- Keep `nosync` when throughput is the priority and producers can replay the
possible loss window.
- Use `async` to encourage writeback without paying synchronous latency, not as
a durability guarantee.
- Use `sync` for locally durable non-WAL writes or when every materialized-table
commit must be durable immediately.
- Use `adaptive` for WAL ingestion with a bounded RPO. Set its group window to
`0`, or wait for QWP local durable acknowledgements, when the relevant
acknowledgement must be zero-loss.
- Replication and local commit mode protect against different failures. Local
durability survives power loss; replicated QWP acknowledgements survive loss
of the server's disk or node.

#### Migration and rollback

Commit mode changes are supported across restarts. When a table first opens in
`adaptive` mode, QuestDB records that its materialized state may be ahead of its
last durable epoch. If the process crashes and you restart with another mode,
QuestDB still performs the required adaptive recovery before reconciling the
table to the new mode. Switching from `adaptive` to `nosync`, `async`, or `sync`
is therefore safe on a version that supports adaptive recovery.

A **binary rollback** to a QuestDB version that predates `adaptive` is a
different operation. The new epoch and checksum sidecars are designed to be
ignored by older binaries, but the in-process compatibility tests do not replace
a real cross-version rollback matrix. In particular, a pre-adaptive binary
cannot recover a data directory left by an unclean adaptive shutdown.

Before a binary rollback, use the adaptive-capable version to recover from any
unclean shutdown, restart it with a non-adaptive commit mode, allow every
adaptive WAL table to open for writing and reconcile to that mode, then stop it
cleanly and take a volume snapshot. Treat rollback to an older binary as
unverified unless the release notes for both versions explicitly support that
path.

### cairo.adaptive.commit.group.window

- **Default**: `50ms`
- **Reloadable**: no

Maximum batching window for flushing adaptive sequencer records. The RPO for an
ordinary commit acknowledgement is bounded by this window plus the background
flush-sweep scheduling delay. `0` flushes every sequencer commit before
returning and gives zero-loss commit acknowledgements at higher latency.
Negative values are treated as `0`.

This setting does not affect other commit modes. Materialized-view refresh WAL
continues to flush each commit synchronously.

### cairo.adaptive.epoch.interval

- **Default**: `60s`
- **Reloadable**: no

Minimum time between durable materialized-state epochs for an adaptive table.
`0` takes an epoch after every WAL apply batch. A negative value disables both
time- and row-triggered epochs, causes unbounded WAL retention, and makes
recovery replay the WAL from its base; use that only for diagnostics or test
isolation.

### cairo.adaptive.epoch.max.rows

- **Default**: `5000000`
- **Reloadable**: no

Takes a durable epoch after this many rows have been applied since the previous
epoch, even if the interval has not elapsed. This bounds retained WAL and
recovery replay under sustained ingestion. A value less than or equal to `0`
disables the row trigger while leaving the time trigger active.

### cairo.adaptive.epoch.flush.on.close

- **Default**: `true`
- **Reloadable**: no

Makes a best-effort final durable epoch when an adaptive table writer closes
cleanly, including idle eviction and graceful shutdown. A negative
`cairo.adaptive.epoch.interval` disables epochs, including this close-time one.
Disable this setting only if the extra close-time I/O is unacceptable; the next
startup must then replay the tail since the previous epoch.

### cairo.adaptive.epoch.column.sync.batched

- **Default**: `true`
- **Reloadable**: no

Uses a batched filesystem flush for adaptive epoch columns where the platform
supports it. QuestDB automatically disables this optimization on filesystems
where it cannot provide the required guarantee. Set it to `false` to force
per-file flushing.

### cairo.adaptive.recovery.roll.forward.enabled

- **Default**: `true`
- **Reloadable**: no

Restores adaptive tables to their last valid durable epoch and replays the WAL
tail during startup. This is a recovery kill switch, not a way to accept weaker
recovery: when set to `false`, QuestDB refuses to start if an adaptive table
requires roll-forward.

### cairo.wal.commit.writeback.drain

- **Default**: `true`
- **Reloadable**: no

Starts writeback across an adaptive WAL segment before taking the per-file
durability barriers. On supported filesystems this lets files write back in
parallel and reduces commit latency. It is only an optimization: QuestDB still
flushes every file, and filesystems that do not support effective range
writeback simply skip the drain.

### cairo.max.uncommitted.rows

Expand Down
21 changes: 12 additions & 9 deletions documentation/connect/clients/c-and-cpp.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1191,15 +1191,18 @@ queue and returns before the server acks.
2. **`wait` = observation.** `qwp_sender_wait` (C++ `wait()`) blocks until
everything published so far is acknowledged.
- **Ack levels.** `qwpws_ack_level_ok` means the server accepted the
frames. `qwpws_ack_level_durable` additionally waits until they are
uploaded to object storage, not just in the server's WAL (Enterprise
with replication; see the protocol page's
frames. `qwpws_ack_level_local_durable` waits for local-disk durability
and requires `request_durable_ack=local`, WAL tables, and
`cairo.commit.mode=adaptive`.
`qwpws_ack_level_durable` waits for replicated/object-store durability
and requires `request_durable_ack=replicated` or the legacy alias `on`.
`local,replicated` is protocol-defined but current servers deny it (see
the protocol page's
[durable acknowledgement](/docs/connect/wire-protocols/qwp-ingress-websocket/#durable-acknowledgement)
section). Durable acks must be requested at pool open with
`request_durable_ack=on`; the connect fails with
`protocol_version_error` when the server cannot provide them, and
without the key any durable-level `wait` or `flush_and_wait` fails up
front with `invalid_api_call`, leaving the buffer or chunk untouched.
section). An unavailable or partial grant fails the connect with
`protocol_version_error`; a level not selected by the connect string
fails `wait` or `flush_and_wait` up front with `invalid_api_call`, leaving
the buffer or chunk untouched.
- **Ack is not visibility.** Rows become visible to queries after WAL
apply, typically within milliseconds of the ack, so a query issued
right after the ack can miss the newest rows. An empty read-back is
Expand Down Expand Up @@ -1436,7 +1439,7 @@ Dispatch on `line_sender_error_get_code(err)` (C++
| `failover_retry` | Transient transport failure; frames may be in doubt | Dead — every later call fails | **Drop** the borrow, then re-borrow with `borrow_sender_with_retry(reconnect_max_duration_ms())`. With `sf_dir`, unresolved frames replay automatically. |
| `server_rejection` | Server refused the data (schema/type conflict, bad name) | Dead — every later call fails | Plain **return** is safe; the pool retires the connection. Fix the data before re-sending; blind retry re-fails. |
| `server_flush_error` | Backpressure deadline hit: queue full for `sf_append_deadline_millis` | Usable | Retry later, shed load, or raise the deadline. Nothing was dropped. See [backpressure](#durability-and-backpressure). |
| `invalid_api_call` | Borrow still at the pool cap after `acquire_timeout_ms`, pool closed, an operation on a borrow after close, or a durable-level wait without `request_durable_ack=on` | n/a | At-cap: treat as backpressure (see [Sizing the pool](#sizing-the-pool)). Closed pool: stop borrowing. |
| `invalid_api_call` | Borrow still at the pool cap after `acquire_timeout_ms`, pool closed, an operation on a borrow after close, or a durable-level wait without its matching `request_durable_ack` tier | n/a | At-cap: treat as backpressure (see [Sizing the pool](#sizing-the-pool)). Closed pool: stop borrowing. |

If you are unsure which case you hit, **return is always safe**: the pool
inspects the connection and retires it if unhealthy, so a broken connection
Expand Down
58 changes: 28 additions & 30 deletions documentation/connect/clients/connect-string.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -665,35 +665,33 @@ Requires QuestDB Enterprise (multi-host).

*Applies to: ingress.*

:::note QuestDB Enterprise

Durable ACK requires QuestDB Enterprise. OSS is single-node and does not
ship WALs off-box, so the server-side durability-acknowledgement signal
that drives this protocol is enterprise-only.

:::

QuestDB Enterprise ships Write-Ahead Logs (WALs) from the primary to an
object store or another file system — typically over the network. After
durably shipping a WAL, the server emits a `STATUS_DURABLE_ACK` frame to
the store-and-forward client; the client marks that frame's FSN as durable
only after this acknowledgement arrives.

The benefit: if the primary dies before shipping a WAL, the client still
holds the corresponding frames in its SF buffer and replays them against
the new primary on failover — closing the data-loss window that a
transport-level OK ACK alone cannot close.

- `request_durable_ack` — when `on`, the client gates trim on
`STATUS_DURABLE_ACK` frames from the server, suppressing OK-driven trim.
Default: `off`.
- `durable_ack_keepalive_interval_millis` — interval at which the client
emits keepalive PINGs while waiting for durable-ack frames. Required
because the server only flushes pending durable acks on inbound recv
events. Default: `200` (ms). Set to `0` or a negative value to disable.

See the [QWP Egress (WebSocket)](/docs/connect/wire-protocols/qwp-egress-websocket/)
wire protocol for the underlying mechanism.
Durable acknowledgements let a store-and-forward sender retain each frame until
QuestDB confirms the requested durability boundary:

- `request_durable_ack=local` waits for `STATUS_LOCAL_DURABLE_ACK`. The WAL
transaction survives power loss on that server, but not loss of its disk.
This requires WAL tables and `cairo.commit.mode=adaptive`; a `nosync` server
can accept the handshake without producing local progress.
- `request_durable_ack=replicated` waits for `STATUS_DURABLE_ACK` after the WAL
reaches the configured object store. This requires QuestDB Enterprise with
replication.
- `request_durable_ack=local,replicated` requests both streams. It is
protocol-defined but not currently granted by servers. When supported, only
the stronger replicated stream may drive trim.
- `request_durable_ack=on` is the legacy alias for `replicated`. It retains the
original `true` request and `enabled` confirmation on the wire.
- `request_durable_ack=off` (the default) trims on ordinary OK responses.

A server grants the complete requested tier set or fails the connection; it
never silently substitutes a weaker guarantee.

`durable_ack_keepalive_interval_millis` controls how often the client sends a
WebSocket PING while durable work is pending. This is required because the
server emits pending durable progress only while handling inbound traffic.
Default: `200` ms. Set to `0` or a negative value to disable.

See the [QWP ingress WebSocket protocol](/docs/connect/wire-protocols/qwp-ingress-websocket/#durable-acknowledgement)
for the wire-level contract.

## Query client keys {#egress-keys}

Expand Down Expand Up @@ -901,7 +899,7 @@ description and behaviour notes.
| `reconnect_initial_backoff_millis` | int (ms) | `100` | [Ingress reconnect](#reconnect-keys) |
| `reconnect_max_backoff_millis` | int (ms) | `5000` | [Ingress reconnect](#reconnect-keys) |
| `reconnect_max_duration_millis` | int (ms) | `300000` (5 min) | [Ingress reconnect](#reconnect-keys) |
| `request_durable_ack` | enum (`on` / `off`) | `off` | [Durable ACK](#durable-ack) |
| `request_durable_ack` | enum (`off` / `on` / `local` / `replicated` / `local,replicated`) | `off` | [Durable ACK](#durable-ack) |
| `sender_id` | string | `default` | [Store-and-forward](#sf-keys) |
| `sender_pool_max` | int | `4` | [Connection pool](#pool-keys) |
| `sender_pool_min` | int | `1` | [Connection pool](#pool-keys) |
Expand Down
Loading
Loading