diff --git a/.github/workflows/validate-links.yml b/.github/workflows/validate-links.yml index bb51f12c7..3cea5447c 100644 --- a/.github/workflows/validate-links.yml +++ b/.github/workflows/validate-links.yml @@ -36,5 +36,8 @@ jobs: - name: Install dependencies run: yarn install --frozen-lockfile + - name: Run plugin tests + run: yarn test + - name: Build site for broken link validation run: yarn build diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 833739bcc..f4b2fb6dc 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -8,6 +8,14 @@ description: Recent updates and improvements to the QuestDB documentation. This page tracks significant updates to the QuestDB documentation. +## October 2026 + +### Updated + +- [Node.js client](/docs/connect/clients/nodejs/) - Rewrote the page for QWP support in `@questdb/nodejs-client` 5.0.0: pooled ingestion and streaming SQL queries from one connect string, every column type, compiled object-row writers, acknowledgements, transactions, store-and-forward, UDP, failover, error handling, and migration from ILP and from 4.x. The [connect string reference](/docs/connect/clients/connect-string/) and the high-availability and wire-protocol pages now note where the Node.js client's keys, defaults, and behavior differ +- [Store-and-forward](/docs/high-availability/store-and-forward/concepts/) - Corrected the replay semantics for every client: replay is at least once and can insert duplicate rows unless the table uses `DEDUP UPSERT KEYS`. The error policy table now shows the real defaults, which include no drop policy, and the [Java](/docs/connect/clients/java/#ingestion-errors) and [.NET](/docs/connect/clients/dotnet/#how-errors-surface) client pages now document their retriable, terminal, and abandoned policies. [Client failover](/docs/high-availability/client-failover/concepts/#authentication-is-cluster-wide) now lists which clients retry authentication rejections after a sender's first connection +- [Replication tuning](/docs/high-availability/tuning/) - Corrected the default of `replication.primary.throttle.window.duration` to 1 second, its value since QuestDB Enterprise 3.3.1, in the tuning guide and the [replication configuration reference](/docs/configuration/database-replication/#replicationprimarythrottlewindowduration) + ## September 2026 ### QuestDB Enterprise Releases diff --git a/documentation/concepts/delivery-semantics.md b/documentation/concepts/delivery-semantics.md index 597c1910c..a72b78230 100644 --- a/documentation/concepts/delivery-semantics.md +++ b/documentation/concepts/delivery-semantics.md @@ -2,15 +2,25 @@ title: Delivery semantics sidebar_label: Delivery semantics description: - How QuestDB clients deliver data (at-least-once), where duplicate rows can - arise, and how to combine designated timestamps with deduplication for - exactly-once outcomes. + How QuestDB QWP/WebSocket senders replay unacknowledged writes, where + duplicates arise, and how to use deduplication for exactly-once outcomes. --- -QuestDB clients deliver data **at-least-once**: every row your application -publishes is guaranteed to reach the server, but under failure it may arrive -more than once. Storing each row exactly once is the application's -responsibility, and QuestDB provides the mechanisms to make it routine. +QuestDB QWP/WebSocket senders retry **published, unacknowledged batches** +while they retain them. This gives at-least-once delivery under transport +failures, but a row may be stored more than once. QWP/UDP is fire-and-forget: +it has no acknowledgement or retry, so rows can be lost. Storing each row +exactly once is the application's responsibility, and QuestDB provides the +mechanisms to make it routine. + +Without [store-and-forward](/docs/high-availability/store-and-forward/concepts/), +unacknowledged rows live in memory and are lost if the process exits, or the +sender closes, before the server acknowledges them. A Node.js sender in +default memory mode also gives up after `reconnect_max_duration_millis` of +outage; see the +[Node.js client](/docs/connect/clients/nodejs/#ingestion-reconnect). +Server rejections and exhausted buffer capacity can also stop delivery; handle +those errors rather than assuming every attempted write will arrive. This page explains where duplicates come from and how to suppress them. @@ -18,14 +28,14 @@ This page explains where duplicates come from and how to suppress them. | Property | Meaning | Where it comes from | |----------|---------|---------------------| -| **At-most-once** | Each row reaches the server zero or one times. Rows can be lost. | A "fire and forget" client that does not retransmit on failure. | -| **At-least-once** | Each row reaches the server one or more times. No row is lost; duplicates are possible. | A client that retransmits unacknowledged data after a transport error. **This is the QuestDB client default.** | +| **At-most-once** | Each row reaches the server zero or one times. Rows can be lost. | Fire-and-forget delivery without ACKs or retries, such as QWP/UDP. | +| **At-least-once** | Each row reaches the server one or more times. Duplicates are possible. | QWP/WebSocket senders retry published batches while they retain them; disk-backed store-and-forward extends replay across process restarts. | | **Exactly-once** | Each row is stored exactly once. | At-least-once delivery plus server-side deduplication on a key covering row identity. | -QuestDB's clients retransmit unacknowledged batches after transport errors, -host failovers, and process restarts. The trade-off is deliberate: losing -data silently is the worse failure mode. The cost is that the application -must tolerate or suppress duplicates. +QWP/WebSocket senders retransmit unacknowledged batches after transport errors +and host failovers, and across process restarts with store-and-forward. The +trade-off is deliberate: losing data silently is the worse failure mode. The +cost is that the application must tolerate or suppress duplicates. ## Where duplicates come from @@ -38,7 +48,7 @@ the server confirms a batch, the client reconnects and re-sends. If the server had already committed the batch but the acknowledgement was lost in flight, the second send produces duplicates. -This path applies to every QuestDB client deployment. +This path applies to QWP/WebSocket senders, not QWP/UDP. ### Multi-host failover replay @@ -112,12 +122,13 @@ If two distinct events can share `(ts, symbol, side)` and both should be preserved, widen `UPSERT KEYS` to include a column that distinguishes them — for example a `trade_id` or `seq` column. -:::warning DEDUP is required on tables behind multi-host failover +:::warning Use DEDUP behind multi-host failover when duplicates matter When the client fails over from one primary to another, unacknowledged batches are replayed against the new primary. Without `DEDUP UPSERT KEYS` -covering row identity, those replays produce duplicate rows in the target -table. +covering row identity, those replays can produce duplicate rows in the target +table. Enable DEDUP for exactly-once outcomes; applications that tolerate +occasional duplicates can skip it. ::: diff --git a/documentation/configuration/database-replication.md b/documentation/configuration/database-replication.md index d50b9b910..fef85cb1a 100644 --- a/documentation/configuration/database-replication.md +++ b/documentation/configuration/database-replication.md @@ -136,7 +136,7 @@ better for constrained networks but more costly. ### replication.primary.throttle.window.duration -- **Default**: `10000` +- **Default**: `1000` - **Reloadable**: no The millisecond duration of the sliding window used to process replication diff --git a/documentation/connect/clients/connect-string.md b/documentation/connect/clients/connect-string.md index c027700dd..b015b14db 100644 --- a/documentation/connect/clients/connect-string.md +++ b/documentation/connect/clients/connect-string.md @@ -16,8 +16,12 @@ deviates, the affected key section and client page call it out. One `ws::` / `wss::` connect string serves both the ingress sender and the egress query client. Each direction reads the keys relevant to it and ignores keys meant only for the other direction, so the same string -configures both without edits. The *Applies to:* tag on each section below -marks which direction a key affects. +configures both without edits. The Node.js client is the exception for +`target` and `zone`, which it also applies to ingress; see +[Role filter and zone preference](#role-filter-and-zone-preference). The +*Applies to:* tag on each section below marks which direction a key affects. +The [Node.js client page](/docs/connect/clients/nodejs/#differences-from-other-clients) +lists its behavioral differences from this reference. For legacy InfluxDB Line Protocol (ILP) transports (`http`, `https`, `tcp`, `tcps`), see the [ILP overview](/docs/connect/compatibility/ilp/overview/). @@ -128,10 +132,22 @@ wss::addr=questdb.example.com:443;username=admin;password=secret; ### Production with a custom trust store +For Node.js and clients that accept PEM roots, use a PEM file without a password: + +```text +wss::addr=questdb.example.com:443;username=admin;password=secret;tls_roots=/etc/questdb/ca.pem; ``` -wss::addr=questdb.example.com:443;username=admin;password=secret;tls_roots=/etc/questdb/ca-roots;tls_roots_password=changeit; + +For clients that accept password-protected JKS or PKCS#12 stores, supply the +password as well: + +```text +wss::addr=questdb.example.com:443;username=admin;password=secret;tls_roots=/etc/questdb/ca-roots.p12;tls_roots_password=changeit; ``` +Node.js rejects `tls_roots_password` and accepts only PEM roots; Go accepts +neither key and uses the OS trust store. See [TLS](#tls) for formats by client. + ### Ingest with store-and-forward across multiple nodes ``` @@ -144,17 +160,33 @@ wss::addr=node-a:9000,node-b:9000;sf_dir=/var/lib/myapp/qdb-sf;sender_id=ingest- wss::addr=node-a:443,node-b:443;target=replica;zone=eu-west-1a; ``` +Senders in other clients ignore `target`, so they can share this string. On the +Node.js client, `target=replica` in the connect string stops ingestion; +set the role with the typed `egress` option instead, as described under +[Role filter and zone preference](#role-filter-and-zone-preference). + ### Tolerate a slow or restarting server at startup ``` ws::addr=node-a:9000;reconnect_max_duration_millis=120000; ``` -The 2-minute reconnect budget covers both the *first* connect and any -subsequent reconnect: setting any explicit `reconnect_*` key implicitly -turns on `initial_connect_retry`. See +Setting any explicit `reconnect_*` key implicitly turns on +`initial_connect_retry`, so the sender retries its *first* connect for up to +2 minutes. A running sender retries later outages indefinitely. See [Ingress reconnect](#reconnect-keys). +:::caution Node.js client + +On the Node.js client, the retry covers senders only. +`connectQwpNodeClient()` also opens a query connection, which still gives up +almost at once, so add `query_pool_min=0`, or use `lazy_connect=on` to start +without waiting. In default memory mode, the budget also ends every later +outage after 2 minutes. See +[Node.js startup and outage modes](/docs/connect/clients/nodejs/#ingestion-modes). + +::: + ## Recipes {#recipes} Goal-to-keys mapping. For complete connect-string templates, see @@ -169,11 +201,11 @@ caveats), follow the section links from the [Key index](#key-index). | Bearer-token credentials | both | `token` | `auth_timeout_ms` | | Multi-host failover | both | `addr=h1,h2,…` | `target`, `zone`, `reconnect_*` (ingress), `failover_*` (egress) | | Query only the primary (freshest data) | egress | `target=primary` | — | -| Query only replicas (offload primary) | egress | `target=replica` | — | +| Query only replicas (offload primary) | egress | `target=replica` | Node.js: use the typed `egress.target` option, because the key also filters ingress | | Zone-aware routing with DR last-resort | egress | `zone=` | `target` | | Tune ingest batching | ingress | — | Clients with auto-flush: `auto_flush_rows`, `auto_flush_interval`, `auto_flush_bytes` | | Disable auto-flush (manual `flush()` only) | ingress | `auto_flush=off` | — | -| Memory-buffered ingest (no disk durability) | ingress | (omit `sf_dir`) | `init_buf_size`, `max_buf_size` | +| Memory-buffered ingest (no disk durability) | ingress | (omit `sf_dir`) | Node.js: `sf_max_total_bytes` (memory replay capacity), `auto_flush_rows` (batching); clients with row-buffer sizing: `init_buf_size`, `max_buf_size` | | Durable store-and-forward ingest | ingress | `sf_dir` | `sender_id`, `sf_max_segment_bytes`, `sf_max_total_bytes`, `sf_append_deadline_millis` | | Run multiple senders sharing one `sf_dir` | ingress | `sf_dir`, `sender_id` | unique `sender_id` per sender | | Orphan recovery for crashed senders | ingress | `drain_orphans=on` | `max_background_drainers` | @@ -222,12 +254,14 @@ WebSocket upgrade request. deployments. - `auth_timeout_ms` — per-host upper bound on the upgrade response read. Does not cover TLS handshake or post-upgrade frame reads, which use OS or - hard-coded defaults. Default: `15000` (15 s). + hard-coded defaults. Default: `15000` (15 s). On Node.js, it defaults to + `connect_timeout` when only that key is set. - `connect_timeout` — integer milliseconds, must be `> 0`. Applies to ingress and egress. Bounds the TCP connect phase for each endpoint, so a black-holed host in a multi-host `addr` no longer stalls the [endpoint walk](#failover-keys) until the OS connect timeout. Unset by - default. + default in most clients, which then use the OS timeout. Node.js defaults it + to `15000` (15 seconds) and also bounds DNS and the TLS handshake with it. **Mutual TLS (mTLS).** Not supported. The client validates the server's certificate against a trust store but cannot present a client certificate; @@ -248,7 +282,8 @@ Selecting the `wss` schema enables TLS. below. - `tls_roots` — path to a file of trusted root certificates, used instead of the system trust store. If omitted, the client uses the system default - trust store. The accepted on-disk formats are client-specific: + trust store, except the Node.js client, which uses the CA certificates + bundled with Node.js. The accepted on-disk formats are client-specific: | Client | Formats accepted at `tls_roots` | |---|---| @@ -256,6 +291,7 @@ Selecting the `wss` schema enables TLS. | Rust, C, C++, Python | PEM (default), JKS, PKCS#12 | | .NET | PKCS#12 / PFX | | Go | none — OS trust store only, both keys rejected at parse time | + | Node.js | PEM only; `tls_roots_password` is rejected at parse time | - `tls_roots_password` — password for the `tls_roots` file. Required only for a JKS or PKCS#12 trust store; PEM needs no password. Setting it without @@ -291,7 +327,9 @@ Existing JKS and PKCS#12 trust stores keep working through The Go client verifies against the operating-system trust store only and **rejects both keys at parse time**; to trust a private CA there, install it in -the host trust store. On Rust, C, C++ and Python, `tls_roots_password` switches +the host trust store. The Node.js client accepts PEM only and rejects +`tls_roots_password`: export a JKS or PKCS#12 trust store to PEM first. On +Rust, C, C++ and Python, `tls_roots_password` switches the file to a Java keystore and is QWP/WebSocket only: other transports keep PEM as the sole format. Check the relevant [client library page](/docs/connect/overview/#client-libraries) for @@ -320,15 +358,19 @@ act independently: whichever threshold trips first sends the batch. buffered rows. - `auto_flush_rows` — flush when the buffered row count reaches this threshold. Set to `off` to disable. Default where supported: `1000`. + The Node.js client rejects `off` here; set `0` to disable. - `auto_flush_interval` — flush when this many milliseconds have elapsed - since the first buffered row. The client evaluates the interval on the - next `at()` / `flush()` call, not on a wall-clock timer. Set to `off` to - disable. Default where supported: `100` (100 ms). + since the first buffered row in most clients. The client evaluates the + interval on the next `at()` / `flush()` call, not on a wall-clock timer. + Node.js instead measures from its last flush (or sender creation), so the + first row after an idle period may trigger a flush. Set to `off` to disable. + Default where supported: `100` (100 ms). The Node.js client rejects `off` + here; set `0` to disable. - `auto_flush_bytes` — flush when the encode buffer reaches this byte size. Set to `off` to disable. Accepts [size suffixes](#size-suffixes). **The default differs by client**: Java - ships it **disabled** (`0`), .NET defaults to `8m` (8 MiB), and Rust, C and - C++ reject the key outright. A Java application that assumes an 8 MiB byte + and Node.js ship it **disabled** (`0`), .NET defaults to `8m` (8 MiB), and + Rust, C and C++ reject the key outright. A Java application that assumes an 8 MiB byte trigger is active will size batches expecting a flush that never fires. When set to a positive value, the client clamps the effective threshold down to 90% of the server- @@ -356,7 +398,9 @@ applications must call `flush()` explicitly; see the *Applies to: ingress (encode buffer).* These keys control the in-memory row buffer that the client uses before -flushing. +flushing. The Node.js QWP `ws`/`wss` client rejects `init_buf_size` and +`max_buf_size` as legacy-transport options; use +[`auto_flush_rows`](#auto-flush) to control its row batches instead. - `init_buf_size` — initial buffer size in bytes. Default: `65536` (64 KiB). Accepts [size suffixes](#size-suffixes). @@ -381,10 +425,13 @@ case-insensitive and 1024-based, matching `-Xmx` conventions: | `g` or `gb` | GiB (× 1024³) | `1g`, `10gb` | | `t` or `tb` | TiB (× 1024⁴) | `1t` | +The Node.js client accepts only the single-letter suffixes `k`, `m`, `g`, +and `t`, and rejects `kb`, `mb`, `gb`, and `tb`. + ## Multi-host failover {#failover-keys} *Applies to: ingress and egress. The [Role filter and zone preference](#role-filter-and-zone-preference) -sub-section is egress only.* +sub-section is egress only, except on the Node.js client.* :::note QuestDB Enterprise @@ -421,7 +468,18 @@ backoff. Both `target` and `zone` apply to **egress only**. QuestDB is currently a single-primary cluster: ingress automatically follows the primary across the host list and adapts when the primary moves to another node. Ingress -silently accepts these keys and ignores them. +silently accepts these keys and ignores them, except on the Node.js client +(see below). + +:::caution Node.js client + +The Node.js client applies `target` and `zone` to ingress as well. With +`target=replica` in a shared connect string, its senders accept only replicas +and cannot ingest. Set the query-side role through the typed `egress` option +instead; see the +[Node.js client page](/docs/connect/clients/nodejs/#multiple-endpoints). + +::: - `target` — server-role filter applied per endpoint after the upgrade reads `SERVER_INFO`. Options: @@ -452,15 +510,15 @@ server-side HA separately. Related: [Reconnect and failover](#reconnect-keys), [Store-and-forward](#sf-keys). -:::warning Enable DEDUP on tables ingested through failover +:::warning Use DEDUP for exactly-once outcomes with failover On unplanned failover — when the primary dies before issuing a durable ACK — the client replays unacknowledged frames against the new primary. Without [DEDUP](/docs/concepts/deduplication/) on the target table, those -replays can produce duplicate rows. Tables ingested through a multi-host -failover connect string **must** declare `DEDUP UPSERT KEYS(...)` covering -row identity. See [Delivery semantics](/docs/concepts/delivery-semantics/) -for the full at-least-once / exactly-once model. +replays can produce duplicate rows. If your application requires exactly-once +outcomes, declare `DEDUP UPSERT KEYS(...)` covering row identity. Applications +that tolerate occasional duplicates can skip DEDUP. See +[Delivery semantics](/docs/concepts/delivery-semantics/) for the full model. ::: @@ -485,8 +543,9 @@ equivalent — same architecture, no durability across restarts. - Taken verbatim. Absolute paths recommended for production; relative paths resolve against the process working directory. - The client does **not** expand shell-style syntax such as `~`. - - The client creates the leaf directory if it is missing, but the parent - must already exist — it does not create paths recursively. + - Java, Rust, C, C++, and Python create `sf_dir` and its slot, but require + any parent directories of `sf_dir` to exist first. Go, .NET, and Node.js + create missing parent directories recursively as well as the slot. - `sender_id` — slot identity. The slot lives at `//`, used verbatim as the directory name. Allowed characters: letters, digits, `_`, `-`. No path separators, no `.`, no spaces. Two senders @@ -500,21 +559,18 @@ equivalent — same architecture, no durability across restarts. |---|---| | Java (`QuestDB` facade) | `/-/` | | Rust, C, C++ (`QuestDb` / `questdb::pool` / `questdb_db`) | `/-ingest-/` | + | Node.js (`connectQwpNodeClient`) | `/-/` | The minted names belong to that pool's namespace, so pools sharing one `sf_dir` need distinct bases; the slot-in-use error covers both cases (another process or pool holds the slot). -- `sf_durability` — disk durability mode. `memory` (the default) and - `periodic` both ship. `periodic` requires `sf_dir` and checkpoints published - frames in the background at `sf_sync_interval_millis`. `flush` and `append` - are reserved: they parse but are rejected at `build()`. - - Reach for `periodic` when you must survive host loss. `memory` mode is - process-crash durable but **not** host-crash durable, because the page cache - is lost on power failure. - - The .NET client is the exception: it accepts only `memory` and rejects - anything else at parse time. +- `sf_durability` — disk durability mode. `memory` (the default) relies on the + page cache; `periodic` requires `sf_dir` and checkpoints published frames + in the background at `sf_sync_interval_millis`. Reach for `periodic` when + you must survive host loss: `memory` survives a process crash but not a + power failure. Go and .NET accept only `memory`. Node.js also accepts + `append`, which makes each journal append durable before `flush()` resolves. + Other clients reject `append`; `flush` is not supported. - `sf_sync_interval_millis` — cadence at which `sf_durability=periodic` checkpoints published frames to stable storage. Default: `5000`. Requires `sf_durability=periodic`; rejected otherwise. The configured interval is a @@ -522,10 +578,15 @@ equivalent — same architecture, no durability across restarts. - `sf_max_segment_bytes` — per-segment rotation threshold. Must be ≥ the largest single flushed frame. Default: `4 MiB` (`4m`). Accepts [size suffixes](#size-suffixes). -- `sf_max_total_bytes` — hard cap on per-slot storage. When the slot - reaches the cap, `append()` blocks until ACKs trim space (see +- `sf_max_total_bytes` controls per-slot capacity for producer backpressure. + When capacity is exhausted, `append()` blocks until ACKs trim space (see `sf_append_deadline_millis`). Defaults: `10 GiB` (`10g`) in SF mode, `128 MiB` (`128m`) in memory mode. Accepts size suffixes. + On Node.js with `sf_dir`, this is a journal size target, not a hard disk + limit: transaction-closing batches and retained symbol dictionaries can + exceed it, and other metadata needs additional space. Provision disk + headroom; see the [Node.js capacity guidance](/docs/connect/clients/nodejs/#sf-capacity). + Without `sf_dir`, the key caps the in-memory replay queue. ### Sender restart and replay @@ -552,6 +613,18 @@ exit, SIGKILL, host crash, or reboot — instantiate a new sender with the parallel with the application's new `append()` calls — it does not block the application. +:::caution Node.js client + +The Node.js client does not use an OS lock. It locks a slot with a +`.lock.owner` directory, so a crashed Node.js sender can leave the slot +locked, and a new sender then fails with `QwpReplayStoreLockedError`. Node.js +and other clients do not see each other's locks: never let them use the same +`sf_dir` at the same time. See the +[Node.js client](/docs/connect/clients/nodejs/#sf-lock-recovery) for lock +recovery. + +::: + If `sf_dir` is a relative path, ensure the process resolves it the same way after restart (typically: use an absolute path). @@ -583,6 +656,11 @@ and releases it — **multiple orphans drain in parallel**, up to - `drain_orphans` — `on` enables the orphan drainer pool. Default: `off`. - `max_background_drainers` — maximum concurrent drainers. Default: `4`. +Without `drain_orphans=on`, the pooled Node.js client still replays slots of +its own `sender_id` that no running sender holds; the key adds other +`sender_id`s. See +[Node.js journal replay](/docs/connect/clients/nodejs/#replaying-the-journal-after-a-restart). + For delivery semantics, architecture, and tradeoffs (at-least-once guarantees, DEDUP requirements, segment-granular trim), see [Store-and-forward concepts](/docs/high-availability/store-and-forward/concepts/). @@ -602,7 +680,9 @@ These keys control the cursor-engine reconnect loop used by QWP ingest. SF mode and memory-only mode share the same loop. A **running** sender retries a transport outage indefinitely with capped exponential backoff — there is no wall-clock give-up: the whole point of the buffering -architecture is that a producer survives an arbitrarily long outage. +architecture is that a producer survives an arbitrarily long outage. The +exception is a Node.js sender in default memory mode, which gives up after +`reconnect_max_duration_millis`; see below. - `reconnect_initial_backoff_millis` — initial wait between reconnect attempts. Backoff grows exponentially up to `reconnect_max_backoff_millis`. @@ -616,14 +696,20 @@ architecture is that a producer survives an arbitrarily long outage. constructor gives up and returns the error. The running loop and the `async` initial connect never consult it. Default: `300000` (5 min). Setting this enables `initial_connect_retry=on` implicitly; see below. + The Node.js client differs: a sender in default memory mode, without + `sf_dir`, `initial_connect_retry=async`, or `lazy_connect=on`, applies this + budget to every outage, and fails with `QwpReconnectExhaustedError` when it + runs out. See the + [Node.js client](/docs/connect/clients/nodejs/#ingestion-reconnect). - `initial_connect_retry` — whether the client retries the initial connect attempt on failure. - `off` (default, alias `false`) — fail fast on initial connect failure. - `on` (aliases `sync`, `true`) — retry synchronously on the user thread, up to `reconnect_max_duration_millis`. - `async` — return the `Sender` immediately; the I/O thread retries in - the background indefinitely, surfacing only genuine terminal failures - (auth reject, durable-ack mismatch) via the error inbox. + the background indefinitely, surfacing terminal failures via the error + inbox. An authentication rejection before the first successful connection + is terminal; see the authentication note below. **Implicit promotion.** Setting any explicit `reconnect_*` key without also choosing an `initial_connect_retry` mode promotes @@ -636,14 +722,20 @@ architecture is that a producer survives an arbitrarily long outage. milliseconds waiting for buffered frames to drain. Set to `0` or `-1` for fast close (skip the drain). **The default differs by client**: `60000` (60 s) on Java and .NET, `5000` (5 s) on Rust, C, C++ and Python, which - share the same Rust core. + share the same Rust core, and on Go and Node.js. This is the shutdown data-loss window. Setting it to `0` skips the drain entirely and drops un-ACKed batches on every clean shutdown. -Auth failures during reconnect (authentication rejected, version mismatch, -durable-ack mismatch, non-101 upgrade without a role hint) are immediately -terminal — the loop does not retry them. +Authentication rejection (HTTP `401` / `403`) never moves the loop to another +host. Before a sender's first successful connection it is terminal in every +client. After that, clients differ: the Java client retries it indefinitely, +the Node.js client does so for senders with `sf_dir` or in background memory +mode (`initial_connect_retry=async` or `lazy_connect=on`), and the Rust, C, +C++, Python, Go, and .NET clients stop. Query connections and orphan drainers +stop too, except a Java orphan drainer whose token comes from a token +provider, which retries for a bounded time before quarantining the slot. See +[authentication during failover](/docs/high-availability/client-failover/concepts/#authentication-is-cluster-wide). ### Egress failover {#egress-failover} @@ -690,7 +782,10 @@ transport-level OK ACK alone cannot close. - `durable_ack_keepalive_interval_millis` — interval at which the client emits keepalive PINGs while waiting for durable-ack frames. Required because the server only flushes pending durable acks on inbound recv - events. Default: `200` (ms). Set to `0` or a negative value to disable. + events. Default: `200` (ms). Set to `0` or a negative value to disable + in clients that support it. In Node.js, explicitly setting this key also + requests durable ACK (even at `0`), and negative values are rejected; + see [Node.js differences](/docs/connect/clients/nodejs/#differences-from-other-clients). See the [QWP Egress (WebSocket)](/docs/connect/wire-protocols/qwp-egress-websocket/) wire protocol for the underlying mechanism. @@ -705,7 +800,8 @@ keys so that the Sender and the `QwpQueryClient` can share a single connect string without an "unknown configuration key" error — the Sender does not interpret the values. Range, enum, and type checks happen on the egress side; the Sender silently accepts even a value the -`QwpQueryClient` parser would reject. +`QwpQueryClient` parser would reject. The Node.js `Sender` accepts these keys +too and logs a warning for the ones it ignores; it applies `client_id`. - `compression` — result-batch compression the client advertises. Options: `raw` (default — no compression; the client omits the accept-encoding @@ -748,11 +844,13 @@ per-language names. *Applies to: the pooled facade (`QuestDB.connect`, `questdb::pool`, `QuestDb::connect`, `questdb.connect`, `qdb.NewQuestDB`, -`QuestDBClient.Connect`).* +`QuestDBClient.Connect`, `connectQwpNodeClient`).* Every client now leads with a pooled facade, so these keys are a first-contact concern. The `Sender` and query-client parsers accept and ignore them; the -facade reads them off the string. Each has an equivalent builder setter, and an +facade reads them off the string. The Node.js `Sender` logs a warning for the +pool keys it ignores, and applies `lazy_connect`, which gives it a +[background start](/docs/connect/clients/nodejs/#ingestion-modes). Each has an equivalent builder setter, and an explicit setter always wins over the string. - `sender_pool_min` — senders kept open even when idle. `0` lets the pool close @@ -768,13 +866,19 @@ explicit setter always wins over the string. forever. Default: `60000`. - `max_lifetime_ms` — maximum age of a connection; the housekeeper closes and reopens older ones once idle. `0` means no age limit. Default: `1800000` - (30 min). + (30 min). The Node.js client only closes idle connections above the pool + minimum, and does not recycle the minimum connections. - `housekeeper_interval_ms` — how often the housekeeper checks for idle and over-age connections. Default: `5000`. - `lazy_connect` — when `on`, the pool defers opening its first connection until the first borrow, so construction succeeds against a server that is down. This is the supported way to tolerate a server that starts after your - application. Default: `off`. + application. Default: `off`. The Node.js client instead starts its senders + connecting in the background (`initial_connect_retry=async`) and sets + `query_pool_min=0`, rejecting a positive value. Those senders also retry an + outage indefinitely rather than giving up after + `reconnect_max_duration_millis`; see + [Node.js startup and outage modes](/docs/connect/clients/nodejs/#ingestion-modes). ## Error handling {#error-handling} @@ -790,11 +894,22 @@ consumed by the application. :::caution Accepted, but not applied by every client Every client's parser accepts the six `on_*_error` keys below, but only -clients that implement the policy layer act on them. **In the Java reference -client they are currently accepted no-ops** — setting -`on_write_error=retriable_other` parses cleanly and changes nothing. .NET does -implement them, via `SenderErrorPolicy` and `SenderErrorCategory`. The -category table and precedence model below describe the target contract. +clients that implement the policy layer act on them, and the two that do +accept different values: + +- **Go** applies them as described here. `on_server_error` accepts `auto`, + `terminal`, `retriable`, or `retriable_other`, and the per-category keys + accept the same values except `auto`. +- **.NET** accepts only `halt` (alias `terminal`) and `retry` (alias + `retriable`), maps the legacy `drop` and `drop_and_continue` to `retry`, and + rejects any other value, including `auto` and `retriable_other`, with a + `ConfigError`. Its keys can only make a retriable category terminal; see the + [.NET client](/docs/connect/clients/dotnet/#per-category-policy). +- **In the Java reference client and the Node.js, Rust, C, C++, and Python + clients they are currently accepted no-ops**, so setting + `on_write_error=retriable_other` parses cleanly and changes nothing. + +The category table and precedence model below describe the target contract. ::: @@ -836,7 +951,9 @@ poison-frame detector (`max_frame_rejections`, default `4`). `PROTOCOL_VIOLATION` is always terminal and `UNKNOWN` always retriable (fail open: a status byte from a newer server degrades to retry, not to a dead -sender); neither can be overridden. Per-client wiring of the override surface +sender); the `on_*_error` keys cannot override either. The .NET client's +programmatic resolver is the exception: it can change the policy for +`UNKNOWN`. Per-client wiring of the override surface may lag the spec — check your client's documentation for which of the resolver / per-category / connect-string layers it exposes. For the full model see the @@ -854,17 +971,17 @@ description and behaviour notes. | `addr` | `host:port[,host:port…]` | required | [Multi-host failover](#failover-keys) | | `auth_timeout_ms` | int (ms) | `15000` | [Authentication](#auth) | | `auto_flush` | enum (`on` / `off`) | `on` (Rust: only `off`) | [Auto-flushing](#auto-flush) | -| `auto_flush_bytes` | size | Java `0` (off) / .NET `8m` (Rust: rejected) | [Auto-flushing](#auto-flush) | +| `auto_flush_bytes` | size | Java, Node.js `0` (off) / .NET `8m` (Rust: rejected) | [Auto-flushing](#auto-flush) | | `auto_flush_interval` | int (ms) / `off` | `100` (Rust: rejected) | [Auto-flushing](#auto-flush) | | `auto_flush_rows` | int / `off` | `1000` (Rust: rejected) | [Auto-flushing](#auto-flush) | | `buffer_pool_size` | int (≥ 1) | `4` | [Query client keys](#egress-keys) | | `catch_up_cap_gap_min_escalation_window_millis` | int (ms) | `300000` (5 min) | [Store-and-forward](#sf-keys) | | `client_id` | string | client-specific | [Query client keys](#egress-keys) | -| `close_flush_timeout_millis` | int (ms) | Java/.NET `60000` / Rust, C, C++, Python `5000` | [Ingress reconnect](#reconnect-keys) | +| `close_flush_timeout_millis` | int (ms) | Java/.NET `60000` / Rust, C, C++, Python, Go, Node.js `5000` | [Ingress reconnect](#reconnect-keys) | | `compression` | enum (`raw` / `zstd` / `auto`) | `raw` | [Query client keys](#egress-keys) | | `compression_level` | int (`1`–`22`) | `1` | [Query client keys](#egress-keys) | -| `connect_timeout` | int (ms, `> 0`) | unset | [Authentication](#auth) | -| `connection_listener_inbox_capacity` | int (≥ 1) | `64` (Java) · `256` (Go, .NET) · not supported by Rust, C/C++, Python | [Error handling](#error-handling) | +| `connect_timeout` | int (ms, `> 0`) | unset (Node.js: `15000`) | [Authentication](#auth) | +| `connection_listener_inbox_capacity` | int (≥ 1) | `64` (Java, Node.js) · `256` (Go, .NET) · not supported by Rust, C/C++, Python | [Error handling](#error-handling) | | `drain_orphans` | enum (`on` / `off`) | `off` | [Store-and-forward](#sf-keys) | | `durable_ack_keepalive_interval_millis` | int (ms) | `200` | [Durable ACK](#durable-ack) | | `error_inbox_capacity` | int (≥ 16) | `256` | [Error handling](#error-handling) | @@ -873,7 +990,7 @@ description and behaviour notes. | `failover_backoff_max_ms` | int (ms) | `1000` | [Egress failover](#reconnect-keys) | | `failover_max_attempts` | int | `8` | [Egress failover](#reconnect-keys) | | `failover_max_duration_ms` | int (ms) | `30000` | [Egress failover](#reconnect-keys) | -| `init_buf_size` | size | `65536` (64 KiB) | [Buffer sizing](#buffer) | +| `init_buf_size` | size | `65536` (Node.js QWP: unsupported) | [Buffer sizing](#buffer) | | `initial_connect_retry` | enum (`off` / `on` / `async`) | `off` (auto-promoted to `on` when any explicit `reconnect_*` key is set) | [Ingress reconnect](#reconnect-keys) | | `initial_credit` | int (bytes) | `0` (unbounded) | [Query client keys](#egress-keys) | | `housekeeper_interval_ms` | int (ms) | `5000` | [Connection pool](#pool-keys) | @@ -882,7 +999,7 @@ description and behaviour notes. | `max_background_drainers` | int | `4` | [Store-and-forward](#sf-keys) | | `max_batch_rows` | int (`1`–`1048576`) | server default | [Query client keys](#egress-keys) | | `max_lifetime_ms` | int (ms) | `1800000` (`0` ⇒ infinite) | [Connection pool](#pool-keys) | -| `max_buf_size` | size | `104857600` (100 MiB) | [Buffer sizing](#buffer) | +| `max_buf_size` | size | `104857600` (Node.js QWP: unsupported) | [Buffer sizing](#buffer) | | `max_datagram_size` | size | (UDP) below typical MTU | [Buffer sizing](#buffer) | | `max_name_len` | int | `127` | [Buffer sizing](#buffer) | | `max_frame_rejections` | int (≥ 1) | `4` | [Error handling](#error-handling) | @@ -894,7 +1011,7 @@ description and behaviour notes. | `on_write_error` | enum | `retriable` | [Error handling](#error-handling) | | `pass` | string | unset | [Authentication](#auth) (alias of `password`) | | `password` | string | unset | [Authentication](#auth) | -| `poison_min_escalation_window_millis` | int (ms) | `5000` | [Error handling](#error-handling) | +| `poison_min_escalation_window_millis` | int (ms) | `5000` (Node.js: `300000`; Go: not supported, its window is `reconnect_max_duration_millis`) | [Error handling](#error-handling) | | `query_close_timeout_ms` | int (ms) | `5000` | [Query client keys](#egress-keys) | | `query_pool_max` | int | `4` | [Connection pool](#pool-keys) | | `query_pool_min` | int | `1` | [Connection pool](#pool-keys) | @@ -907,12 +1024,12 @@ description and behaviour notes. | `sender_pool_min` | int | `1` | [Connection pool](#pool-keys) | | `sf_append_deadline_millis` | int (ms) | `30000` (30 s) | [Store-and-forward](#sf-keys) | | `sf_dir` | path | unset (memory mode) | [Store-and-forward](#sf-keys) | -| `sf_durability` | enum (`memory` / `periodic`) | `memory` (.NET: `memory` only) | [Store-and-forward](#sf-keys) | +| `sf_durability` | enum (`memory` / `periodic` / Node.js `append`) | `memory` (Go and .NET: `memory` only) | [Store-and-forward](#sf-keys) | | `sf_max_segment_bytes` | size | `4 MiB` | [Store-and-forward](#sf-keys) | | `sf_max_total_bytes` | size | `128 MiB` mem / `10 GiB` SF | [Store-and-forward](#sf-keys) | | `sf_sync_interval_millis` | int (ms) | `5000` | [Store-and-forward](#sf-keys) | | `target` | enum (`any` / `primary` / `replica`) | `any` | [Multi-host failover](#failover-keys) | -| `tls_roots` | path | system trust store | [TLS](#tls) | +| `tls_roots` | path | system trust store (Node.js: bundled CAs) | [TLS](#tls) | | `tls_roots_password` | string | unset (JKS / PKCS#12 only) | [TLS](#tls) | | `tls_verify` | enum (`on` / `unsafe_off`) | `on` | [TLS](#tls) | | `token` | string | unset | [Authentication](#auth) | diff --git a/documentation/connect/clients/date-to-timestamp-conversion.md b/documentation/connect/clients/date-to-timestamp-conversion.md index 9e6efb728..dc401b259 100644 --- a/documentation/connect/clients/date-to-timestamp-conversion.md +++ b/documentation/connect/clients/date-to-timestamp-conversion.md @@ -319,27 +319,42 @@ class Program ``` Learn more about the [QuestDB .NET Client](/docs/connect/clients/dotnet/) -## Date to Timestamp in JavasScript/Node.js + -The Date type stores both date and time information. +## Date to Timestamp in JavaScript/Node.js -The QuestDB Node.js client accepts an epoch in microseconds, which can be a `number` or `bigint`. +A JavaScript `Date` stores milliseconds since the Unix epoch. The QuestDB +Node.js client takes a timestamp as an integer `number` or a `bigint` together +with a unit: `"ms"`, `"us"` (the default), or `"ns"`. A `Date` therefore needs +no arithmetic: pass `getTime()` with the `"ms"` unit. ```javascript -const { Sender } = require("@questdb/nodejs-client") - -const dateStr = '2024-08-05'; -const dateObj = new Date(dateStr + 'T00:00:00Z'); - -// Convert to timestamp (milliseconds since Epoch) then convert to microseconds -const timestamp = BigInt(dateObj.getTime()) * 1000n; -console.log("Date:", dateObj.toISOString().split('T')[0]); -console.log("Timestamp (microseconds):", timestamp.toString()); - -// You can now add the column using QuestDB client, as in -// .timestampColumn("NonDesignatedTimestampColumnName", timestamp) +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const settlementDate = new Date("2024-08-05T00:00:00Z"); + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;"); +try { + const sender = await db.borrowSender(); + try { + await sender + .table("settlements") + .symbol("symbol", "ETH-USD") + .timestampColumn("settlement_date", settlementDate.getTime(), "ms") + .doubleColumn("amount", 0.5) + .at(Date.now(), "ms"); + } finally { + await sender.close(); + } +} finally { + await db.close(); +} ``` +For an explicit microsecond value, convert through `bigint`: +`BigInt(settlementDate.getTime()) * 1000n`. Nanosecond timestamps, with the +`"ns"` unit, must be a `bigint`. + Learn more about the [QuestDB Node.js Client](/docs/connect/clients/nodejs/) ## Date to Timestamp in Ruby diff --git a/documentation/connect/clients/dotnet.md b/documentation/connect/clients/dotnet.md index 684ea97cd..5f4d5318f 100644 --- a/documentation/connect/clients/dotnet.md +++ b/documentation/connect/clients/dotnet.md @@ -783,10 +783,12 @@ Each error is classified into a `SenderErrorCategory` and assigned a | Policy | Effect | Default categories | |---|---|---| -| `DropAndContinue` | The rejected batch is dropped; the sender keeps running. | `SchemaMismatch`, `WriteError` | -| `Halt` | The sender latches terminal; the next producer call throws `LineSenderServerException`. | `ParseError`, `InternalError`, `SecurityError`, `ProtocolViolation`, `Unknown` | +| `Retriable` | The client reconnects and resends the batch; the sender keeps running. A batch that keeps being rejected escalates to `Terminal`. | `WriteError`, `InternalError`, `NotWritable`, `DictionaryGap`, `Unknown` | +| `Terminal` | The sender latches terminal; the next producer call throws `LineSenderServerException`. | `SchemaMismatch`, `ParseError`, `SecurityError`, `ProtocolViolation` | +| `Abandoned` | Store-and-forward data that can never be sent was set aside at `QuarantinedPath` and is not retried. | `DataLoss` | -After a `Halt`, discard the sender and create a new one. +After a `Terminal` error, discard the sender and create a new one. There is no +drop policy: the client never discards a rejected batch silently. ### Error handler @@ -814,8 +816,8 @@ Each `SenderError` carries the following fields: | Field | Description | |---|---| -| `Category` | `SchemaMismatch`, `ParseError`, `InternalError`, `SecurityError`, `WriteError`, `ProtocolViolation`, `Unknown`. Use for programmatic dispatch. | -| `AppliedPolicy` | `DropAndContinue` (batch dropped, sender continues) or `Halt` (sender latched terminal; next API call throws `LineSenderServerException`). | +| `Category` | `SchemaMismatch`, `ParseError`, `InternalError`, `SecurityError`, `WriteError`, `NotWritable`, `DictionaryGap`, `ProtocolViolation`, `DataLoss`, `Unknown`. Use for programmatic dispatch. | +| `AppliedPolicy` | `Retriable` (batch resent, sender continues), `Terminal` (sender latched terminal; next API call throws `LineSenderServerException`), or `Abandoned` (only with `DataLoss`: the data was set aside). | | `ServerStatusByte` | Raw QWP status byte (e.g. `0x03` for `SchemaMismatch`). `-1` (`SenderError.NoStatusByte`) on `ProtocolViolation` and engine-internal terminal failures. | | `ServerMessage` | Human-readable server text (≤ 1024 UTF-8 bytes), or `null`. See [Message stability](#message-stability) and [PII safety](#message-pii). | | `MessageSequence` | Server's per-frame QWP wire sequence for the error frame. `-1` (`SenderError.NoMessageSequence`) for engine-internal failures. **Resets on reconnect** — only meaningful within one connection. | @@ -824,6 +826,7 @@ Each `SenderError` carries the following fields: | `DetectedAtUtc` | Wall-clock receipt time on the I/O thread; for ops timelines, not for correlation. | | `Exception` | Non-`null` for engine-internal failures (connect-budget exhaustion, fatal upgrade reject); `null` for server rejections. | | `IsInitialConnect` | `true` if the engine never reached a first successful connection (config / connectivity issue); always `false` for server-side rejections. | +| `QuarantinedPath` | For `DataLoss`, where the set-aside data was preserved; `null` otherwise. | #### Message stability {#message-stability} @@ -857,8 +860,8 @@ and the `(MessageSequence, FromFsn, ToFsn)` triple. ### Synchronous errors Misconfiguration and API-misuse errors surface synchronously as `IngressError` -(or its subclass `LineSenderServerException` for HALT-policy server -rejections). They are thrown directly from the call site: +(or its subclass `LineSenderServerException` for server rejections with the +`Terminal` policy). They are thrown directly from the call site: | Site | Throws when | |---|---| @@ -869,7 +872,7 @@ rejections). They are thrown directly from the call site: | Array `Column(...)` overloads | The `shape` does not match the element count, dimensionality exceeds 32, or the element type is not `double` / `long`. | | `ColumnGeohash(...)` | `precisionBits` is outside `[1, 60]`. | | `ColumnDecimal*(...)` with explicit `scale` | `scale` is outside `[0, 18]` (DECIMAL64), `[0, 38]` (DECIMAL128), or `[0, 76]` (DECIMAL256). | -| Producer-thread call after `Halt` policy fired | The next `Table`, `Column`, `AtAsync`, or `SendAsync` throws `LineSenderServerException` carrying the latched `SenderError`. Discard the sender and create a new one. | +| Producer-thread call after a `Terminal` policy fired | The next `Table`, `Column`, `AtAsync`, or `SendAsync` throws `LineSenderServerException` carrying the latched `SenderError`. Discard the sender and create a new one. | Authentication failures surface differently between paths: a `401` / `403` during the WebSocket upgrade returns synchronously from `Sender.New` / @@ -880,26 +883,31 @@ the sender latched terminal. ### Per-category policy -Override the default policy per category with the `on_*_error` connect-string -keys (values `halt` or `drop`): +Make a category stricter with the `on_*_error` connect-string keys. The +accepted values are `halt` (alias `terminal`) and `retry` (alias `retriable`). +The legacy values `drop` and `drop_and_continue` now mean `retry`, because the +client never drops a batch. Any other value, such as `auto` or +`retriable_other`, is rejected with a `ConfigError`: ```csharp -// Treat a schema mismatch as fatal instead of dropping the batch. +// Stop the sender on write errors instead of retrying them. using var sender = Sender.New( - "ws::addr=localhost:9000;on_schema_mismatch_error=halt;"); + "ws::addr=localhost:9000;on_write_error=halt;"); ``` | Key | Scope | |---|---| -| `on_server_error` | Catch-all default for every category. | -| `on_schema_mismatch_error` (alias: `on_schema_error`) | Schema-validation rejections. | -| `on_parse_error` | Client-side parse errors. | -| `on_internal_error` | Unexpected client-side errors. | -| `on_security_error` | Auth / TLS errors. | -| `on_write_error` | Transport write failures. | - -`ProtocolViolation` and `Unknown` are always `Halt`, regardless of these keys. -For programmatic control, set `SenderOptions.error_policy_resolver` to a +| `on_server_error` | Catch-all default for every category below. | +| `on_schema_mismatch_error` (alias: `on_schema_error`) | `SchemaMismatch`: the batch does not match the table schema. | +| `on_parse_error` | `ParseError`: the server could not parse the batch. | +| `on_internal_error` | `InternalError`: an unexpected server-side failure. | +| `on_security_error` | `SecurityError`: the server denied the write. | +| `on_write_error` | `WriteError`: the write failed, for example because the table is not accepting writes. | + +The keys can only make a category stricter. `SchemaMismatch`, `ParseError`, +`SecurityError`, and `ProtocolViolation` are always `Terminal`, even if their +key says `retry`, and `Unknown` stays `Retriable` regardless of these keys. For +programmatic control, set `SenderOptions.error_policy_resolver` to a `SenderErrorPolicyResolver` delegate. ### Connection-level errors @@ -934,9 +942,10 @@ A summary of how the engine treats each error class on the wire: | Auth (`401` / `403`) on any endpoint | Terminal | Halts the failover loop immediately; the sender / query client latches non-recoverable. | | Role reject (`421` + `X-QuestDB-Role`) | Topology-level (transient if `PRIMARY_CATCHUP`, otherwise terminal for the loop) | The client tries the next endpoint; if every endpoint rejects, surfaces as `QwpRoleMismatchException` (egress) or the sender's reconnect loop exhausts. | | Version mismatch during upgrade | Per-endpoint, **not** terminal | The client moves on to the next endpoint. | -| Server rejection of a batch (`SchemaMismatch`, `ParseError`, `WriteError`, etc.) | Per the `on_*_error` policy — default is `DropAndContinue` for `SchemaMismatch` / `WriteError`, `Halt` for everything else. | `DropAndContinue` keeps the sender alive; `Halt` latches the sender so the next producer call throws `LineSenderServerException`. | +| Server rejection of a batch (`SchemaMismatch`, `ParseError`, `WriteError`, etc.) | Per category: `Retriable` for `WriteError`, `InternalError`, `NotWritable`, and `DictionaryGap`; `Terminal` for `SchemaMismatch`, `ParseError`, and `SecurityError`. The `on_*_error` keys can make a retriable category terminal. | `Retriable` resends the batch and keeps the sender alive; `Terminal` latches the sender so the next producer call throws `LineSenderServerException`. | | TCP / TLS failure, `404`, `503`, mid-stream drop | Transient | Fed into the ingress reconnect loop (`reconnect_max_*` keys) or, on egress, the per-query failover loop (`failover_*` keys). | -| `ProtocolViolation`, `Unknown` | Terminal | Always `Halt`, regardless of `on_*_error` settings. | +| `ProtocolViolation` | Terminal | Always `Terminal`, regardless of `on_*_error` settings. | +| `Unknown` (a status this client does not know) | Retriable | Resent rather than stopping the sender, regardless of `on_*_error` settings. | ### Connection events @@ -962,7 +971,7 @@ Event kinds: `Connected`, `Disconnected`, `Reconnected`, `FailedOver`, `AuthFailed` and `ReconnectBudgetExhausted` are **terminal**: the sender latches a non-recoverable failure, the next producer-thread call (`Table`, `Column`, `AtAsync`, `SendAsync`) throws `IngressError` (or -`LineSenderServerException` if a HALT-policy error was latched alongside), +`LineSenderServerException` if a `Terminal`-policy error was latched alongside), and no further data can be sent. Discard the sender, build a new one, and replay any state your application owns. `DroppedConnectionNotifications` on `IQwpWebSocketSender` counts events that were dropped because a slow listener diff --git a/documentation/connect/clients/java.md b/documentation/connect/clients/java.md index 1df7d8239..cdee6b7ae 100644 --- a/documentation/connect/clients/java.md +++ b/documentation/connect/clients/java.md @@ -215,7 +215,8 @@ apply latency varies with load. This applies to every client, not just Java. See the equivalent poll in the [Python](/docs/connect/clients/python/), [Rust](/docs/connect/clients/rust/) and -[Go](/docs/connect/clients/go/) quick starts. +[Go](/docs/connect/clients/go/) quick starts, and in the Node.js client's +[Read-after-write](/docs/connect/clients/nodejs/#read-after-write) section. The `QuestDB` handle is a facade over two distinct kinds of client: a [`Sender`](#data-ingestion) for ingestion (`db.borrowSender()`) and a @@ -1242,18 +1243,24 @@ regardless of how the sender was created: | Field | Accessor | Description | |-------|----------|-------------| -| Category | `getCategory()` | `SCHEMA_MISMATCH`, `PARSE_ERROR`, `INTERNAL_ERROR`, `SECURITY_ERROR`, `WRITE_ERROR`, `PROTOCOL_VIOLATION`, or `UNKNOWN` | -| Policy | `getAppliedPolicy()` | `DROP_AND_CONTINUE` (batch dropped, sender continues) or `HALT` (next API call throws `LineSenderServerException`) | +| Category | `getCategory()` | `SCHEMA_MISMATCH`, `PARSE_ERROR`, `INTERNAL_ERROR`, `SECURITY_ERROR`, `WRITE_ERROR`, `NOT_WRITABLE`, `DICTIONARY_GAP`, `PROTOCOL_VIOLATION`, `DATA_LOSS`, or `UNKNOWN` | +| Policy | `getAppliedPolicy()` | `RETRIABLE` (the client reconnects and resends the batch), `RETRIABLE_OTHER` (resends it to another endpoint), `TERMINAL` (the sender stops; the next API call throws `LineSenderServerException`), or `ABANDONED` (only with `DATA_LOSS`: store-and-forward data that can never be sent was set aside) | | Server message | `getServerMessage()` | Human-readable error text from the server (may be null) | | Table name | `getTableName()` | The rejected table (null for multi-table batches) | | FSN range | `getFromFsn()` / `getToFsn()` | Frame sequence number span identifying the rejected batch | | Message sequence | `getMessageSequence()` | Server's per-frame sequence number (`-1` if not available) | | Status byte | `getServerStatusByte()` | Raw QWP status code (`-1` if not available) | +| Quarantined path | `getQuarantinedPath()` | For `DATA_LOSS`, where the set-aside data was preserved (null otherwise) | + +There is no drop policy: a rejected batch is resent, stops the sender, or, for +`DATA_LOSS` only, is set aside. See +[Error frames](/docs/high-availability/store-and-forward/concepts/#error-frames) +for the default policy of each category. The error handler runs on a dedicated dispatcher thread, never on the I/O or producer thread. -When a sender owned by `QuestDB` enters a terminal `HALT` state, the next +When a sender owned by `QuestDB` stops with a `TERMINAL` error, the next producer-thread call throws `LineSenderServerException`. The pool detects the failure on close/return and replaces the failed sender with a fresh one on the next borrow. @@ -1319,8 +1326,10 @@ What is and isn't carried on `onError`: ### Connection-level errors - **Authentication failure**: `401`/`403` HTTP response before the WebSocket - upgrade completes. Terminal across all endpoints. The borrow that - triggered the connect rethrows `LineSenderException`. + upgrade completes. Terminal across all endpoints for the query client and + for a sender's first connection: the borrow that triggered the connect + rethrows `LineSenderException`. A sender that has connected once retries + instead; see [Which failures are retried](#which-failures-are-retried). - **Malformed frames**: `QwpDecodeException` or WebSocket close with a terminal code. - **Role mismatch**: `QwpRoleMismatchException` when all endpoints report @@ -1421,8 +1430,12 @@ client and gives you a fresh one on your next query. ### Which failures are retried -- **Authentication failures** (a bad token or wrong credentials) stop - immediately on every host — retrying cannot help. +- **Authentication failures** (a bad token or wrong credentials) stop the + query client, and a sender's first connection, immediately on every host. A + sender that has connected once retries them indefinitely, keeping its data + buffered, and reports each rejection to its error handler as a `RETRIABLE` + `SECURITY_ERROR`; see + [Authentication is cluster-wide](/docs/high-availability/client-failover/concepts/#authentication-is-cluster-wide). - **Network and availability failures** (connection refused, TLS errors, a `5xx` from the server, a mid-query drop) are treated as temporary and fed into the reconnect and failover loops. diff --git a/documentation/connect/clients/nodejs.md b/documentation/connect/clients/nodejs.md index de60fe561..0149c5f89 100644 --- a/documentation/connect/clients/nodejs.md +++ b/documentation/connect/clients/nodejs.md @@ -1,269 +1,1929 @@ --- slug: /connect/clients/nodejs -title: Node.js Client Documentation -description: - "Get started with QuestDB using the Node.js client for efficient, - high-performance insert operations. Achieve unparalleled time series data - ingestion and query capabilities." +title: Node.js client for QuestDB +sidebar_label: Node.js +description: "Use @questdb/nodejs-client for QWP ingestion, streaming SQL queries, failover, and store-and-forward from TypeScript or JavaScript." --- -import { ILPClientsTable } from "@theme/ILPClientsTable" +import SfDedupWarning from "../../partials/_sf-dedup-warning.partial.mdx" -QuestDB offers Node.js developers a dedicated client designed for efficient and -high-performance data ingestion. +`@questdb/nodejs-client` ingests rows and streams SQL results over the +[QuestDB Wire Protocol (QWP)](/docs/connect/wire-protocols/qwp-ingress-websocket/). +One pooled client can serve both writers and queries. It also supports the +older ILP transports for existing applications. -:::note No QWP support yet +## Requirements -Unlike the other clients in this section, the Node.js client does not speak the -QuestDB Wire Protocol. It ingests over -[ILP](/docs/connect/compatibility/ilp/overview/) using `http::` connect strings, -and queries run over [PGWire](/docs/connect/compatibility/pgwire/nodejs/). The -`http::` examples below are current, not stale. QWP support is planned. +- `@questdb/nodejs-client` 5.0.0 or later (earlier versions support ILP only). +- Node.js 20.18.1 or later. +- QuestDB 10.0.0 or later, with QWP on its HTTP port (9000 by default). -::: + + +## Installation + +```shell +npm install @questdb/nodejs-client@^5 +npm install --save-dev tsx +``` + +The examples are TypeScript ES modules: save one as `example.mts` and run +`npx tsx example.mts`. For JavaScript, use `.mjs` or `"type": "module"` +and remove TypeScript annotations. + +## Quick start + +Create a table, publish a row, wait for QuestDB's acknowledgement, then poll +until the row is visible. An acknowledgement means QuestDB has committed the +row, but queries see it only after it is applied, which happens +asynchronously; see [Read-after-write](#read-after-write). + +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;"); +try { + const lease = await db.borrowQuery(); + try { + const ddl = await lease.query( + "CREATE TABLE IF NOT EXISTS trades (" + + "timestamp TIMESTAMP, symbol SYMBOL, side SYMBOL, " + + "price DOUBLE, amount DOUBLE" + + ") TIMESTAMP(timestamp) PARTITION BY DAY", + ); + await ddl.completion; + + const eventTime = Date.now(); // milliseconds; also used to find the row + const sender = await db.borrowSender(); + try { + await sender + .table("trades") + .symbol("symbol", "ETH-USD") + .symbol("side", "buy") + .doubleColumn("price", 2615.54) + .doubleColumn("amount", 0.5) + .at(eventTime, "ms"); + await sender.flush(); + await sender.waitForAcknowledged(sender.publishedSequence, 10_000); + } finally { + await sender.close(); // return it to the pool + } + + // Poll until the acknowledged row is visible, for up to 10 seconds. + const deadline = Date.now() + 10_000; + let found = false; + while (!found && Date.now() < deadline) { + const query = await lease.query( + "SELECT timestamp, symbol, side, price, amount FROM trades " + + "WHERE timestamp = $1", + { + binds: (binds) => + binds.setTimestampMicros(0, BigInt(eventTime) * 1000n), + }, + ); + for await (const batch of query) { + for (const row of batch.rows()) { + // [timestamp in microseconds, symbol, side, price, amount] + console.log(row); + found = true; + } + } + await query.completion; + if (!found) await new Promise((resolve) => setTimeout(resolve, 100)); + } + if (!found) throw new Error("row not visible after 10 seconds"); + } finally { + await lease.close(); + } +} finally { + await db.close(); +} +``` + +Use an event timestamp rather than `atNow()` if rows may be replayed. If a +`trades` table already exists with a different designated timestamp name, +change the SQL to match it; the +[PGWire Node.js guide](/docs/connect/compatibility/pgwire/nodejs/) uses `ts`. + +## Connecting + +`connectQwpNodeClient(conf, options?)` opens a pooled ingestion and query +connection and rejects if the server is unreachable. A `ws::` connect string +uses plain WebSocket, and `wss::` uses TLS. The same `addr`, credentials, and +TLS settings apply to both directions: + +```text +ws::addr=localhost:9000;sender_pool_max=2;query_pool_max=8; +``` + +`addr` accepts comma-separated or repeated hosts. A port omitted from an +address defaults to 9000. A key may appear only once (except `addr`); +unknown keys and duplicate keys fail validation. Escape a semicolon in a +value as `;;`. See the +[connect string reference](/docs/connect/clients/connect-string/) for the +shared keys and [Differences from other clients](#differences-from-other-clients) +for Node.js exceptions. A second argument takes typed options; see +[Programmatic options](#programmatic-options). + +`connectQwpNodeClient()` resolves to a `QwpClient`. Its `borrowSender()` +returns a `QwpSender`, and its `borrowQuery()` a `QwpQueryLease`. The package +exports all three classes, so you can use them to type your own variables. + +### Standalone Sender + +For ingestion without a pool, create a `Sender` from a connect string, call +`connect()` before writing, and close it in a `finally` block: + +```typescript +import { Sender } from "@questdb/nodejs-client"; + +const sender = await Sender.fromConfig("ws::addr=localhost:9000;"); +try { + await sender.connect(); + await sender + .table("trades") + .symbol("symbol", "ETH-USD") + .symbol("side", "buy") + .floatColumn("price", 2615.54) // DOUBLE: the Sender has no doubleColumn() + .floatColumn("amount", 0.5) + .at(Date.now(), "ms"); + await sender.flush(); +} finally { + await sender.close(); +} +``` + +The standalone `Sender` has only the nine column methods it shares with ILP +(see [Column methods](#column-methods)); use `sender.writer()` for other types, +or a pooled sender. It also speaks ILP over `http::` and `tcp::`; see +[ILP transports](#ilp-transports-legacy). For a typed standalone QWP sender +with every column method, use +`await connectQwpNodeSender({ url: "ws://localhost:9000/write/v4" })`. + + + +## Authentication and TLS -The Node.js client has solid benefits: +Use a bearer token (QuestDB Enterprise) or HTTP basic authentication. Supply +secrets through your application's configuration rather than source code: -- **Automatic table creation**: No need to define your schema upfront. -- **Concurrent schema changes**: Seamlessly handle multiple data streams with - on-the-fly schema modifications -- **Optimized batching**: Use strong defaults or curate the size of your batches -- **Health checks and feedback**: Ensure your system's integrity with built-in - health monitoring -- **Automatic write retries**: Reuse connections and retry after interruptions +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; -This quick start guide introduces the basic functionalities of the Node.js -client, including setting up a connection, inserting data, and flushing data to -QuestDB. +const token = process.env.QDB_TOKEN; +if (!token) throw new Error("QDB_TOKEN is not set"); +const db = await connectQwpNodeClient( + `wss::addr=db.example.com:9000;token=${token};`, +); +await db.close(); +``` - +Basic auth uses `username=...;password=...;`. With `wss`, the client checks +certificates against Node.js's bundled CA store. For a private CA, set +`tls_roots=/path/to/ca.pem` or `NODE_EXTRA_CA_CERTS`; only PEM roots are +supported. `tls_verify=unsafe_off` disables verification for development +and cannot be combined with `tls_roots`. + +| Path | Status | Alternative | +|---|---|---| +| OIDC token acquisition or refresh | Not supported. The client does not talk to an identity provider. | Get an access token from your identity provider, pass it as `token`, and create a new client before it expires. See [OpenID Connect](/docs/security/oidc/). | +| Client certificates (mTLS) | Not supported. QuestDB does not negotiate client certificates. | Use a token or basic auth over `wss`. | +| Token rotation on a running client | Not supported. Every connection, including reconnects, sends the token the client was created with. | Close the client and create a new one with the new token. For a token rejected on reconnect, see [Connection-level errors](#connection-level-errors). | + +## The connection pool + +Borrow one sender per concurrent producer and one query lease per concurrent +query. Each pool defaults to a minimum of 1 and a maximum of 4 connections; +set `sender_pool_max`, `query_pool_max`, and, if needed, their `_min` keys. +A borrowed sender's `close()` flushes completed rows and **returns it to the +pool**, but does not normally wait for their acknowledgements. A returned +lease or sender must not be reused. Size pools to the number of simultaneous +borrows, and close the client on shutdown. + +### Startup and outage modes {#ingestion-modes} + +To start while QuestDB is down and keep accepting rows during an outage, add +`lazy_connect=on;sf_max_segment_bytes=1m;` to the connect string. To also +keep unacknowledged rows across process restarts, add +`sf_dir=/var/lib/myapp/qdb-sf;sender_id=trades;`. These keys select one of +three modes: + +| Mode | Keys | QuestDB down at startup | During an outage | +|---|---|---|---| +| Default memory | None | `connectQwpNodeClient()` rejects | `flush()`, and `at()` when it triggers an auto-flush, wait for the reconnect for up to `reconnect_max_duration_millis` (5 minutes by default). Then the sender fails with `QwpReconnectExhaustedError` and its unacknowledged rows are lost | +| Background memory | `lazy_connect=on` | Starts; rows queue in memory | Rows queue in memory, up to `sf_max_total_bytes` (128 MiB by default); retries continue indefinitely | +| Store-and-forward | `sf_dir`, usually with `lazy_connect=on` | With `lazy_connect=on`, starts and journals rows; without it, rejects | Rows go to the disk journal and survive a process restart; retries continue indefinitely | + +:::caution Default memory mode blocks producers + +Auto-flush runs on the first `at()` call 100 ms or more after the last flush, +so in default memory mode a producer blocks almost as soon as an outage +starts. The blocked call throws if QuestDB is still unreachable after +`reconnect_max_duration_millis` (5 minutes by default). If a producer must +not block on QuestDB, for example an HTTP request handler, use a background +start: its calls block only when the replay queue is full; see +[Backpressure](#backpressure). -:::info +::: -This page focuses on our high-performance ingestion client, which is optimized for **writing** data to QuestDB. -For retrieving data, we recommend using a [PostgreSQL-compatible Node.js library](/docs/connect/compatibility/pgwire/nodejs/) or our -[HTTP query endpoint](/docs/query/overview/#rest-http-api). +How the keys combine: + +- **Foreground and background start.** By default, senders get a + *foreground start*: `connectQwpNodeClient()` connects them and rejects if + QuestDB is down. `lazy_connect=on` gives them a *background start* instead + and sets `query_pool_min=0`; combining it with a positive + `query_pool_min` is a configuration error. `initial_connect_retry=async` + also gives senders a background start, but `connectQwpNodeClient()` still + opens one query connection at startup and rejects while QuestDB is down, + unless you also set `query_pool_min=0`. A query borrowed before QuestDB is + reachable fails. A standalone `Sender` gets a background start from either + key. +- **Batch size.** With a background start, set `sf_max_segment_bytes=1m`, + with or without `sf_dir`; see [Batch size limits](#batch-size-limits). +- **Storage.** Without `sf_dir`, unacknowledged rows live in memory and are + lost if the process exits. With `sf_dir`, they are journaled to disk and + replayed after a restart; see [Store-and-forward](#store-and-forward). + `sf_dir` alone keeps a foreground start: startup rejects while QuestDB is + down, and outages after the first successful connection are retried + indefinitely. +- **Tables.** An ingester that starts while QuestDB is down cannot create + its tables first. Create tables that need DEDUP beforehand; see + [Store-and-forward](#store-and-forward). +- **First-connection retry.** `initial_connect_retry=on`, or any + `reconnect_*` key unless `initial_connect_retry=off` is set, makes senders + retry their first connection for up to `reconnect_max_duration_millis` + instead of failing at once. The sender stays in default memory mode. These + keys do not apply to query connections, so with the default + `query_pool_min=1`, `connectQwpNodeClient()` still rejects almost at once: + also set `query_pool_min=0`. +- **Locked journal.** A journal locked by another process fails startup, + even with a background start; see [Lock recovery](#sf-lock-recovery). + +### Closing the pooled client + +`db.close()` rejects new borrows, closes idle senders and queries, and waits +briefly for borrowed senders to be returned. It then waits up to +`close_flush_timeout_millis` (5 seconds by default) for acknowledgements and +resolves even if some batches are still unacknowledged: with `sf_dir`, they +stay in the journal and are replayed on the next start; without it, they are +lost. When an ACK is required, call +`await sender.waitForAcknowledged(sender.publishedSequence, timeoutMs)` before +returning a borrowed sender. + +A borrowed sender's `close()` flushes its completed rows. With a background +start or `sf_dir`, that hands them to the memory replay queue or the journal, +so `close()` returns without waiting for QuestDB, even during an outage, +unless the queue or journal is full (see [Backpressure](#backpressure)). In +[default memory mode](#ingestion-modes), `close()` can wait for a reconnect up +to `reconnect_max_duration_millis` (5 minutes by default); plan your shutdown +deadline accordingly. + +## Data ingestion + + + +Start a row with `table()`, add columns, and finish with +`await sender.at(timestamp, unit)` or `await sender.atNow()`. QuestDB creates +missing tables and columns automatically. The +[quick start](#quick-start) shows the full borrow/flush/close cycle. + +### Column methods + +The pooled QWP sender and `connectQwpNodeSender()` expose these methods: + +| Method | QuestDB type / value | +|---|---| +| `symbol(name, value)` | SYMBOL; use for bounded sets such as tickers and sides | +| `stringColumn(name, value)` | VARCHAR; use for high-cardinality IDs | +| `booleanColumn`, `byteColumn`, `shortColumn`, `int32Column` | BOOLEAN, BYTE, SHORT, INT | +| `longColumn`, `intColumn` | LONG; safe integer `number` or `bigint` | +| `float32Column`, `doubleColumn`, `floatColumn` | FLOAT, DOUBLE, DOUBLE | +| `timestampColumn(name, value, unit = "us")`, `dateColumn` | TIMESTAMP/TIMESTAMP_NS and DATE; for units, see [Designated timestamp](#designated-timestamp) | +| `charColumn`, `binaryColumn`, `uuidColumn` | CHAR, BINARY (`Uint8Array`), UUID | +| `long256Column`, `ipv4Column`, `geohashColumn` | LONG256, IPv4, GEOHASH | +| `decimalColumnText`, `decimalColumn`, `decimal64Column`, `decimal128Column`, `decimal256Column` | DECIMAL; see [Decimals](#decimals) | +| `arrayColumn(name, value)` | Nested DOUBLE arrays; see [Arrays](#arrays) | + +`floatColumn()` writes DOUBLE and `intColumn()` writes LONG; use the `32` +variants for FLOAT and INT. A standalone `Sender` offers the nine methods +shared with ILP: `symbol`, `stringColumn`, `booleanColumn`, `floatColumn`, +`intColumn`, `timestampColumn`, `arrayColumn`, `decimalColumn`, and +`decimalColumnText`. Its compiled writer supports the other types. See the +[API reference](https://questdb.github.io/nodejs-questdb-client/modules/_questdb_nodejs-client.html) +for method signatures and accepted values. + +:::caution Use SYMBOL for bounded sets + +A sender keeps distinct SYMBOL values in a dictionary across its lifetime; +high-cardinality IDs belong in VARCHAR or UUID instead. The dictionary is +limited to 2,000,000 values. If a flush exceeds it, discard the staged rows +with `reset()` and use a new sender for new symbol values. A pooled sender is +replaced only if its `close()` fails; resetting before returning it leaves its +dictionary full. ::: +### Designated timestamp + +`at(value, unit)` accepts `"us"` (the default), `"ms"`, or `"ns"`. +`Date.now()` is **milliseconds**, so use `.at(Date.now(), "ms")`; +without the unit the row lands in 1970. `timestampColumn(name, value, unit)` +takes the same units with the same `"us"` default, so pass the unit there +too: `.timestampColumn("exchange_ts", Date.now(), "ms")`. Nanoseconds require +a `bigint`. +`atNow()` asks QuestDB to assign arrival time, which changes on replay. Use +the event's timestamp for deduplication. For a newly created table, `"ns"` +creates a TIMESTAMP_NS designated timestamp; the other units create +TIMESTAMP. The default designated column name is `timestamp`. On an existing +table, `at()` writes the table's designated timestamp column, whatever its +name. + +### Null values + +Passing `null` or `undefined` omits that column. On an existing nullable +column this stores NULL; an omitted BOOLEAN becomes `false`, and BYTE and +SHORT become `0`. An all-null column does not create a new column. Local +value errors discard the row in progress: start again with `table()`. +`cancelRow()` drops an unfinished row; `reset()` also drops rows staged since +the last flush. + + + +### Decimals + +Pre-create a table if you need a specific precision: QWP auto-creation +chooses the maximum precision for the wire width. + +```questdb-sql +CREATE TABLE IF NOT EXISTS trade_fees ( + timestamp TIMESTAMP, + symbol SYMBOL, + settled_price DECIMAL(18, 2), + commission DECIMAL(18, 4) +) TIMESTAMP(timestamp) PARTITION BY DAY; +``` -## Requirements + -- Node.js v16 or newer. -- Assumes QuestDB is running. If it's not, refer to - [the general quick start](/docs/getting-started/quick-start/). +`decimalColumnText(name, value)` sends a decimal as text and preserves its +scale, including trailing zeros. Both strings and numbers accept exponents +such as `"1.5e-3"`; a JavaScript number cannot retain trailing zeros. -## Client installation + -Install the QuestDB Node.js client via npm: +The binary methods `decimal64Column(name, unscaled, scale)`, +`decimal128Column()`, and `decimal256Column()` take the unscaled value as a +`bigint`, followed by the scale: -```shell -npm i -s @questdb/nodejs-client +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;"); +try { + const sender = await db.borrowSender(); + try { + await sender + .table("trade_fees") + .symbol("symbol", "ETH-USD") + .decimalColumnText("settled_price", "2615.50") // keeps the trailing zero + .decimal64Column("commission", -750n, 4) // -0.0750 + .at(Date.now(), "ms"); + await sender.flush(); + } finally { + await sender.close(); + } +} finally { + await db.close(); +} ``` -## Authentication +Bind parameters take the scale **before** the unscaled value, the reverse of +`decimal64Column()`: `binds.setDecimal64(0, 2, 261550n)` binds `2615.50`; see +[Bind parameters](#bind-parameters). Query results return a DECIMAL as +`{ unscaled, scale }`. The server currently cannot return DECIMAL with +precision 9 or less over QWP; cast it to a wider precision when querying. -Passing in a configuration string with basic auth: +### Arrays -```javascript -const { Sender } = require("@questdb/nodejs-client"); +`arrayColumn(name, value)` sends a uniformly shaped nested array of numbers +as DOUBLE[], DOUBLE[][], and so on. `longArrayColumn()` exists for protocol +parity, but current servers reject LONG arrays. Query results expose DOUBLE +arrays as `{ dimensions, values }`. -const conf = "http::addr=localhost:9000;username=admin;password=quest;" -const sender = Sender.fromConfig(conf); - ... +### Compiled object-row writers + +For a stream of objects with a fixed shape, compile a writer once with +`sender.writer(table, schema)` and call `writer.row(object)` or +`writer.rows(iterable)`. The package exports the schema builders, such as +`symbol()`, `double()`, `varchar()`, and `designatedTimestamp("ms")`. Each +schema key names a column, except the `designatedTimestamp()` key: it supplies +the row's designated timestamp (named `timestamp` on a new table) and is +required in every row. + +```typescript +import { + connectQwpNodeClient, + designatedTimestamp, + double, + symbol, +} from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;"); +try { + const sender = await db.borrowSender(); + try { + const trades = sender.writer("trades", { + symbol: symbol(), + side: symbol(), + price: double(), + amount: double(), + timestamp: designatedTimestamp("ms"), + }); + await trades.rows([ + { symbol: "ETH-USD", side: "buy", price: 2615.54, amount: 0.5, timestamp: Date.now() }, + { symbol: "BTC-USD", side: "sell", price: 61234.5, amount: 0.01, timestamp: Date.now() }, + ]); + await sender.flush(); + } finally { + await sender.close(); // the writer cannot be used after its sender is closed + } +} finally { + await db.close(); +} ``` -Passing via the `QDB_CLIENT_CONF` env var: +The writer validates each row; its `QwpWriterRowError` names the offending +table, column, and row index. See the +[client API reference](https://questdb.github.io/nodejs-questdb-client/modules/_questdb_nodejs-client.html) +for all builders. + +### Flushing + +Auto-flush is enabled by default: 1,000 rows or 100 ms since the last flush +(checked when a row is added, not by a background timer). Call `flush()` at +the end of a burst. `auto_flush=off` disables triggers, or set +`auto_flush_rows=0` / `auto_flush_interval=0` separately. `flush()` publishes +rows, but does not by default wait for QuestDB to accept them. To confirm +delivery, see [Awaiting acknowledgements](#awaiting-acknowledgements). + +#### Backpressure + +The replay queue defaults to 128 MiB without `sf_dir`, and the disk journal +targets 10 GiB with it. When full, publishing waits up to 30 seconds by +default, then rejects with `QwpMemoryReplayAppendTimeoutError` or +`QwpReplayStoreAppendTimeoutError`. The batch stays staged: slow down and +retry `flush()`; do not write the rows again. If you close the sender +instead, it tries once more and, if there is still no room, drops the staged +rows with a warning. A borrowed sender's `close()` then rejects with the same +error, after up to `sf_append_deadline_millis` plus +`close_flush_timeout_millis` (35 seconds by default). A standalone `Sender`'s +`close()` waits at most `close_flush_timeout_millis` (5 seconds by default) +and, with the default deadlines, rejects with `QwpSenderCloseTimeoutError`. +Configure the cap with `sf_max_total_bytes` and the wait with +`sf_append_deadline_millis`. + +#### Batch size limits + +QuestDB advertises its maximum batch size on connection (about 2 MiB on a +default server). The client splits a larger batch into several frames at row +boundaries. A single row larger than the limit fails with +`QwpBatchTooLargeError`: call `reset()` and shrink that row, for example a +large VARCHAR or BINARY value. A sender with a [background start](#ingestion-modes) cannot know +the limit before its first connection, so a frame built while QuestDB is +down can exceed it. That frame is then never delivered: it is retried +indefinitely and blocks every later batch. With a background start, with or +without `sf_dir`, set `sf_max_segment_bytes=1m` to cap each frame at 1 MiB. + +### Awaiting acknowledgements + +After `flush()`, wait for the cumulative watermark: +`await sender.waitForAcknowledged(sender.publishedSequence, 10_000)`. + +`publishedSequence` includes batches sent by auto-flush; `acknowledgedSequence` +is the last accepted one. `waitForAcknowledged()` rejects with +`QwpIngressNackError` when QuestDB terminally rejects a batch (see +[Ingestion errors](#ingestion-errors)), or with `QwpIngressAckTimeoutError` on +timeout; without a timeout argument, it waits up to `ackTimeoutMs` +(15 seconds by default). A timeout +alone does not mean the batch was rejected: it may still be in flight. **Do +not use the return value of `flushAndGetSequence()` as the watermark for all +your rows**: it returns `-1n` if an earlier auto-flush already published them. + +To make each `flush()` wait for the acknowledgement, pass typed +`sender: { awaitServerAck: true }` as the second argument of +`connectQwpNodeClient()`; see [Programmatic options](#programmatic-options). +Each such flush waits up to `ackTimeoutMs`, then rejects with a plain `Error` +whose message starts with `timed out waiting for QWP ACK`. As with +`waitForAcknowledged()`, QuestDB may still acknowledge the batch later. Set the +deadline with typed `ingressSession: { ackTimeoutMs }`; it has no +connect-string key. + +#### Committing source offsets + +When consuming from Kafka or another source, record each batch's +`publishedSequence` with its last source offset. Commit only the newest offset +whose sequence is at or below `acknowledgedSequence`: -```bash -export QDB_CLIENT_CONF="http::addr=localhost:9000;username=admin;password=quest;" +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +interface Fill { offset: bigint; symbol: string; price: number; amount: number; tsMs: number } +// Stand-ins for your consumer: replace them with your Kafka client's calls. +const batches: Fill[][] = [ + [{ offset: 41n, symbol: "ETH-USD", price: 2615.54, amount: 0.5, tsMs: Date.now() }], + [{ offset: 42n, symbol: "BTC-USD", price: 61234.5, amount: 0.01, tsMs: Date.now() }], +]; +const commitOffset = async (offset: bigint) => console.log("commit", offset); + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;"); +try { + const sender = await db.borrowSender(); + const pending: { sequence: bigint; offset: bigint }[] = []; + const commitAcknowledged = async () => { + let newest: bigint | undefined; + while (pending.length > 0 && pending[0].sequence <= sender.acknowledgedSequence) { + newest = pending.shift()!.offset; + } + if (newest !== undefined) await commitOffset(newest); + }; + try { + for (const batch of batches) { + for (const fill of batch) { + await sender + .table("trades") + .symbol("symbol", fill.symbol) + .doubleColumn("price", fill.price) + .doubleColumn("amount", fill.amount) + .at(fill.tsMs, "ms"); + } + await sender.flush(); + pending.push({ sequence: sender.publishedSequence, offset: batch[batch.length - 1].offset }); + await commitAcknowledged(); + } + await sender.waitForAcknowledged(sender.publishedSequence, 10_000); + await commitAcknowledged(); + } finally { + await sender.close(); + } +} finally { + await db.close(); +} ``` -```javascript -const { Sender } = require("@questdb/nodejs-client"); +Request [durable acknowledgement](#durable-acknowledgement) if the offset must +also survive a primary failure. The watermark then advances only after the +upload to object storage, so allow for the upload interval in your +`waitForAcknowledged()` timeout. + +### Transactions + +Set `transaction=on` to defer server commits of auto-flushed batches until +`flush()` (or `commit()` on a pooled sender). Transactions are atomic per +table, not across tables, and QuestDB can commit early when the table exceeds +[`qwp.max.uncommitted.rows`](/docs/configuration/qwp/#qwpmaxuncommittedrows). + +Only a standalone sender can roll back: closing it without `flush()` discards +the open transaction. A pooled sender has no rollback. Returning it with +`close()` flushes and commits the open transaction, including batches that +auto-flush already sent; `reset()` drops only rows that have not been sent. +If your code throws partway through a batch, a pooled sender therefore +commits the rows written before the error. When a failed batch must leave no +rows, use a standalone sender: +```typescript +import { Sender } from "@questdb/nodejs-client"; + +const sender = await Sender.fromConfig("ws::addr=localhost:9000;transaction=on;"); +try { + await sender.connect(); + for (const [side, price] of [["buy", 2615.54], ["sell", 2615.62]] as const) { + await sender + .table("trades") + .symbol("symbol", "ETH-USD") + .symbol("side", side) + .floatColumn("price", price) // DOUBLE: the Sender has no doubleColumn() + .floatColumn("amount", 0.5) + .at(Date.now(), "ms"); + } + await sender.flush(); // commits: both rows become visible together +} finally { + await sender.close(); // rolls back the transaction if flush() was not reached +} +``` -const sender = Sender.fromEnv(); - ... +### Store-and-forward + +Set `sf_dir` to journal batches across process restarts. To start while +QuestDB is down, add a background start (`lazy_connect=on`), as below; see +[Startup and outage modes](#ingestion-modes). Keep event IDs and timestamps +stable across retries. + +If duplicates are unacceptable, create a deduplicated table **before** the +first ingester runs, for example as a deployment or migration step. An +ingester that starts while QuestDB is down cannot create the table itself +first: when QuestDB comes back, the sender replays its journal in the +background, QuestDB auto-creates a missing table without DEDUP, and a later +`CREATE TABLE IF NOT EXISTS` does nothing. To add DEDUP to an existing table, +use [`ALTER TABLE ... DEDUP ENABLE`](/docs/query/sql/alter-table-enable-deduplication/), +for example +`ALTER TABLE trades_sf DEDUP ENABLE UPSERT KEYS(timestamp, trade_id);`. It +does not remove duplicates that were written earlier. + + + +```questdb-sql +CREATE TABLE IF NOT EXISTS trades_sf ( + timestamp TIMESTAMP, + trade_id VARCHAR, + symbol SYMBOL, + price DOUBLE +) TIMESTAMP(timestamp) PARTITION BY DAY +DEDUP UPSERT KEYS(timestamp, trade_id); ``` -When using QuestDB Enterprise, authentication can also be done via REST token. -Please check the [RBAC docs](/docs/security/rbac/#authentication) for more -info. +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +// Persist this directory outside the container in production. +const db = await connectQwpNodeClient( + "ws::addr=localhost:9000;sf_dir=./questdb-sf;sender_id=trades;" + + "sf_max_segment_bytes=1m;lazy_connect=on;", +); +try { + const sender = await db.borrowSender(); + try { + await sender + .table("trades_sf") + .stringColumn("trade_id", "trade-12345") + .symbol("symbol", "ETH-USD") + .doubleColumn("price", 2615.54) + .at(1723000000000, "ms"); + await sender.flush(); // persisted locally; not necessarily acknowledged + } finally { + await sender.close(); + } +} finally { + await db.close(); +} +``` -## Basic insert +If this client closes before an ACK, the journal keeps the unacknowledged row +until a client reopens the same `sf_dir` and `sender_id`; see +[Replaying the journal after a restart](#replaying-the-journal-after-a-restart). + +`sf_durability=memory` (the default) survives a process crash, not a power +failure; `periodic` checkpoints and `append` syncs each append. `sf_dir` alone +keeps a foreground start; see [Startup and outage modes](#ingestion-modes). +A terminally rejected batch stays at the head of the journal and blocks later +rows until fixed; see +[Recovering from a terminal rejection](#recovering-from-a-terminal-rejection). +See also the +[store-and-forward concepts](/docs/high-availability/store-and-forward/concepts/) +and [operating guide](/docs/high-availability/store-and-forward/operating-and-tuning/). + +#### Replaying the journal after a restart + +Run a client with the same `sf_dir` and `sender_id` once QuestDB is reachable +again. It replays the unacknowledged frames in the background, whatever +`sender_pool_min` is: each pooled sender reopens its own slot (`trades-0` for +the first), and a background drainer replays every `trades-` slot that no +running sender holds, at startup and then every 30 seconds. That includes +slots left by more concurrent senders in an earlier run. Slots of other +`sender_id`s in the same `sf_dir` are replayed only with `drain_orphans=on`. +Then poll for a row you know was written: -Example: inserting executed trades for cryptocurrencies. +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; -Without authentication and using the current timestamp. +const db = await connectQwpNodeClient( + "ws::addr=localhost:9000;sf_dir=./questdb-sf;sender_id=trades;" + + "sf_max_segment_bytes=1m;failover=off;", +); +try { + const lease = await db.borrowQuery(); + try { + const deadline = Date.now() + 10_000; + let visible = false; + while (!visible && Date.now() < deadline) { + const query = await lease.query( + "SELECT trade_id FROM trades_sf WHERE trade_id = $1 LIMIT 1", + { + binds: (binds) => binds.setVarchar(0, "trade-12345"), + timeoutMs: Math.max(1, deadline - Date.now()), + }, + ); + for await (const batch of query) visible ||= batch.rowCount > 0; + await query.completion; + if (!visible) await new Promise((resolve) => setTimeout(resolve, 100)); + } + if (!visible) throw new Error("trade not visible in time"); + } finally { + await lease.close(); + } +} finally { + await db.close(); +} +``` -```javascript -const { Sender } = require("@questdb/nodejs-client") +`failover=off` prevents a replayed query from invalidating the result +mid-poll. A fixed sleep without checking for the row is not a visibility +guarantee. + +#### Journal capacity {#sf-capacity} + +`sf_max_total_bytes=10g` is a target, not a hard disk quota: transaction +completion and symbol dictionaries can exceed it. Provision extra space and +monitor the directory. Without `sf_dir`, the same key caps the memory replay +queue. + +#### Lock recovery {#sf-lock-recovery} + +Node.js uses a `.lock.owner` directory in each journal slot, not an OS file +lock, and a crashed process can leave one behind. The next sender reclaims it +automatically only when the lock was recorded on the same host and the +recorded process ID is no longer in use, as when a process restarts on the +same machine under a new process ID. It cannot reclaim a lock recorded on +another host, such as a container replaced under a new host name, or one whose +process ID is in use again. That includes a container restarted in place, +which usually gives the restarted process its previous process ID (often 1), +so the restarted process holds the recorded ID itself. Opening the journal +then fails with `QwpReplayStoreLockedError` (wrapped in `QwpPoolResourceError` +when pooled) on every start until the stale lock is removed. Verify that no +other process owns the slot **before** removing it; the +[Node.js lock-recovery runbook](/docs/high-availability/store-and-forward/operating-and-tuning/#nodejs-lock-recovery) +also shows when a startup step can remove it safely. +Do not let Node.js and another client's OS-lock-based sender use the same +`sf_dir` concurrently. + +### Durable acknowledgement + +On QuestDB Enterprise with replication, `request_durable_ack=on` makes the +acknowledgement watermark wait until the WAL has been uploaded to object +storage. Typed `sender: { awaitDurableAck: true }` also makes each `flush()` +wait for the upload, for up to `durableAckTimeoutMs` (by default +`ackTimeoutMs`, 15 seconds). Under light load, the primary uploads WAL data +only when +[`replication.primary.throttle.window.duration`](/docs/high-availability/tuning/#throttle-window) +expires: 1 second by default, or 60 seconds with the +[network-efficiency settings](/docs/high-availability/tuning/#network-efficiency). +Set the deadline well above the configured window: -async function run() { - // create a sender using HTTP protocol - const sender = Sender.fromConfig("http::addr=localhost:9000") +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; - // add rows to the buffer of the sender +const token = process.env.QDB_TOKEN; +if (!token) throw new Error("QDB_TOKEN is not set"); +const db = await connectQwpNodeClient( + `wss::addr=db.example.com:9000;token=${token};request_durable_ack=on;`, + { sender: { awaitDurableAck: true, durableAckTimeoutMs: 120_000 } }, +); +try { + const sender = await db.borrowSender(); + try { + await sender + .table("trades") + .symbol("symbol", "ETH-USD") + .doubleColumn("price", 2615.54) + .at(Date.now(), "ms"); + await sender.flush(); // waits up to 2 minutes for the upload + } finally { + await sender.close(); + } +} finally { + await db.close(); +} +``` + +When the deadline passes, `flush()` rejects with a plain `Error` whose message +starts with `timed out waiting for QWP durable ACK`. QuestDB has already +acknowledged the batch, so it is committed to the WAL on the primary: the +timeout means only that the upload was not confirmed in time. + +If the server does not support durable ACK: + +- A sender with a foreground start fails with + `QwpDurableAckUnavailableError` (wrapped in `QwpPoolResourceError` when + pooled). +- A sender with a background start retries from startup and emits + `durable-ack-unavailable` connection events, **even with `sf_dir`**. +- A sender with `sf_dir` and a foreground start fails at its first + connection, but after a successful connection it retries later mismatches. + +Monitor these events and journal capacity: a successful background start +does not prove durable ACK is available. + +### Fire-and-forget UDP + +The standalone `Sender` also accepts `udp::addr=localhost:9007;` for +fire-and-forget ingestion. Enable the server's +[`qwp.udp.enabled`](/docs/configuration/qwp/#udp-receiver) first. UDP has no +TLS, auth, ACK, retries, transactions, or store-and-forward; use WebSocket +for reliable writes. + +```typescript +import { Sender } from "@questdb/nodejs-client"; + +const sender = await Sender.fromConfig("udp::addr=localhost:9007;"); +try { + await sender.connect(); await sender .table("trades") .symbol("symbol", "ETH-USD") - .symbol("side", "sell") + .symbol("side", "buy") .floatColumn("price", 2615.54) - .floatColumn("amount", 0.00044) - .atNow() + .floatColumn("amount", 0.5) + .at(Date.now(), "ms"); + await sender.flush(); // sends datagrams; nothing confirms delivery +} finally { + await sender.close(); +} +``` - // flush the buffer of the sender, sending the data to QuestDB - // the buffer is cleared after the data is sent, and the sender is ready to accept new data - await sender.flush() +## Querying - // close the connection after all rows ingested - // unflushed data will be lost - await sender.close() +Borrow one query lease per concurrent query. A lease executes one query at a +time; close it in `finally`. + +### Running a SELECT + +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;"); +try { + const lease = await db.borrowQuery(); + try { + const query = await lease.query( + "SELECT timestamp, symbol, price FROM trades WHERE symbol = $1 LIMIT 100", + { + binds: (binds) => binds.setVarchar(0, "ETH-USD"), + timeoutMs: 30_000, + initialCredit: 1024 * 1024, + }, + ); + for await (const batch of query) { + for (const [timestamp, symbol, price] of batch.rows()) { + console.log(new Date(Number((timestamp as bigint) / 1000n)), symbol, price); + } + } + await query.completion; + } finally { + await lease.close(); + } +} finally { + await db.close(); } +``` + +`lease.query()` returns a handle with async result batches and a `completion` +promise. For DDL/DML, await `completion` without iterating. A batch has +`rowCount`, `columns`, `rows()`, `get(rowIndex, columnIndex)`, and +`batchSequence`. `rows()` yields one array per row, in SELECT column order: +destructure or index it, because a row has no named properties (`row.price` +is `undefined`). `batch.columns[i].name` gives the column names. A query can +fail during iteration as well as at completion. + +Query failover is on by default: if the connection is lost, the query runs +again from its first row, so code that accumulates rows must handle the +restart. See [Query failover](#query-failover). + +### Reading result values + +| QuestDB type | JavaScript value | +|---|---| +| BOOLEAN, INT, DOUBLE, FLOAT | `boolean` or `number` | +| LONG | `bigint` | +| TIMESTAMP / TIMESTAMP_NS / DATE | `bigint` in microseconds / nanoseconds / milliseconds | +| VARCHAR, SYMBOL, CHAR | `string` | +| BINARY | `Uint8Array` | +| UUID | `{ low: bigint, high: bigint }` | +| DECIMAL | `{ unscaled: bigint, scale: number }` | +| DOUBLE arrays | `{ dimensions: number[], values: number[] }` | +| Nullable values | `null` (except CHAR's zero marker, which can be `"\u0000"`) | + +Other types include IPv4 (signed 32-bit `number`), GEOHASH and LONG256 +objects. `JSON.stringify()` cannot serialize `bigint`: convert it to a string +first. Convert a TIMESTAMP to a `Date` with `new Date(Number(value / 1000n))`. +Current servers cannot return INTERVAL, an untyped NULL, or DECIMAL +with precision 9 or less over QWP; cast those in SQL to a supported type. + +### Bind parameters + +Set bind values in the `binds` callback of `query()`. Index 0 binds `$1`, and +indexes must be set in ascending order without gaps. A placeholder after the +last one you set is treated as NULL rather than rejected, so set every +placeholder. + +| QuestDB type | Setter | JavaScript value | +|---|---|---| +| BOOLEAN | `setBoolean(i, value)` | `boolean` | +| BYTE, SHORT, INT | `setByte`, `setShort`, `setInt` | `number` | +| LONG | `setLong(i, value)` | `bigint` or safe-integer `number` | +| FLOAT, DOUBLE | `setFloat`, `setDouble` | `number` | +| CHAR | `setChar(i, value)` | one-character `string` | +| VARCHAR, SYMBOL | `setVarchar(i, value)` | `string` or `null`; SYMBOL has no setter of its own | +| TIMESTAMP | `setTimestampMicros(i, value)` | microseconds, such as `BigInt(date.getTime()) * 1000n` | +| TIMESTAMP_NS | `setTimestampNanos(i, value)` | nanoseconds as a `bigint` | +| DATE | `setDate(i, value)` | milliseconds, such as `date.getTime()` | +| UUID | `setUuid(i, value)` or `setUuid(i, low, high)` | canonical UUID `string`, or two 64-bit halves | +| DECIMAL | `setDecimal64(i, scale, unscaled)`, `setDecimal128(i, scale, low, high)`, `setDecimal256(i, scale, lowLow, lowHigh, highLow, highHigh)` | unscaled value in 64-bit parts, after the scale | +| LONG256 | `setLong256(i, word0, word1, word2, word3)` | four 64-bit words | +| GEOHASH | `setGeohash(i, precisionBits, value)` | geohash bits as an integer | + +For a typed NULL, use `setNull(i, QWP_COLUMN_TYPE.DOUBLE)` (import +`QWP_COLUMN_TYPE` from the package), `setNullDecimal64/128/256(i, scale)`, or +`setNullGeohash(i, precisionBits)`. The decimal setters take the scale +**before** the unscaled value, the reverse of +`decimal64Column(name, unscaled, scale)`. BINARY, IPv4, arrays, and INTERVAL +have no bind setter. + +### DDL and DML statements + +`CREATE`, `ALTER`, `DROP`, `TRUNCATE`, `INSERT`, and `UPDATE` use `query()`. +For a statement without result batches, `completion.kind` is `"exec-done"`. +Only `INSERT` reliably provides a row count in `rowsAffected`, a `bigint`; a +WAL `UPDATE` can report a transaction number instead. -run().then(console.log).catch(console.error) +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +// failover=off: a lost connection fails the INSERT instead of running it again. +const db = await connectQwpNodeClient("ws::addr=localhost:9000;failover=off;"); +try { + const lease = await db.borrowQuery(); + try { + const query = await lease.query( + "INSERT INTO trades (timestamp, symbol, side, price, amount) " + + "VALUES (now(), 'ETH-USD', 'buy', 2615.54, 0.5)", + ); + const completion = await query.completion; + if (completion.kind === "exec-done") { + console.log(`inserted ${completion.rowsAffected} row(s)`); + } + } finally { + await lease.close(); + } +} finally { + await db.close(); +} ``` -In this case, the designated timestamp will be the one at execution time. Let's -see now an example with an explicit timestamp, custom auto-flushing, and basic -auth. +:::warning SQL writes can run twice -```javascript -const { Sender } = require("@questdb/nodejs-client") +Query failover is on by default, so a lost connection can re-execute in-flight +SQL, including `INSERT`. Use `failover=off` for non-idempotent SQL and check an +uncertain outcome before retrying, or make the statement idempotent. A typed +`egressSession.reconnect` object turns failover back on even when the connect +string says `failover=off`; see [Typed reconnect policy](#typed-reconnect-policy). -async function run() { - // create a sender using HTTP protocol - const sender = Sender.fromConfig( - "http::addr=localhost:9000;username=admin;password=quest;auto_flush_rows=100;auto_flush_interval=1000;", - ) +::: - // Calculate the current timestamp. You could also parse a date from your source data. - const timestamp = Date.now() +### Read-after-write - // add rows to the buffer of the sender - await sender - .table("trades") - .symbol("symbol", "ETH-USD") - .symbol("side", "sell") - .floatColumn("price", 2615.54) - .floatColumn("amount", 0.00044) - .at(timestamp, "ms") +An ACK confirms commitment to the WAL, **not** query visibility: WAL apply +is asynchronous. Create the table before writing, wait for the ACK, then poll +for the row with a deadline: - // add rows to the buffer of the sender - await sender - .table("trades") - .symbol("symbol", "BTC-USD") - .symbol("side", "sell") - .floatColumn("price", 39269.98) - .floatColumn("amount", 0.001) - .at(timestamp, "ms") +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;failover=off;"); +try { + const eventTime = Date.now(); // a stable key to find the row again + const sender = await db.borrowSender(); + try { + await sender + .table("trades") + .symbol("symbol", "ETH-USD") + .symbol("side", "sell") + .doubleColumn("price", 2615.62) + .doubleColumn("amount", 0.25) + .at(eventTime, "ms"); + await sender.flush(); + await sender.waitForAcknowledged(sender.publishedSequence, 10_000); + } finally { + await sender.close(); + } + + const lease = await db.borrowQuery(); + try { + const deadline = Date.now() + 10_000; + let visible = false; + while (!visible && Date.now() < deadline) { + const query = await lease.query( + "SELECT price FROM trades WHERE timestamp = $1 AND symbol = $2 LIMIT 1", + { + binds: (binds) => + binds.setTimestampMicros(0, BigInt(eventTime) * 1000n).setVarchar(1, "ETH-USD"), + timeoutMs: Math.max(1, deadline - Date.now()), + }, + ); + for await (const batch of query) visible ||= batch.rowCount > 0; + await query.completion; + if (!visible) await new Promise((resolve) => setTimeout(resolve, 100)); + } + if (!visible) throw new Error("row not visible in time"); + } finally { + await lease.close(); + } +} finally { + await db.close(); +} +``` - // flush the buffer of the sender, sending the data to QuestDB - // the buffer is cleared after the data is sent, and the sender is ready to accept new data - await sender.flush() +`failover=off` prevents a replayed query from invalidating the result mid-poll. +A fixed sleep without checking for the row is not a visibility guarantee. +After a store-and-forward restart, see +[Replaying the journal after a restart](#replaying-the-journal-after-a-restart). - // close the connection after all rows ingested - // unflushed data will be lost - await sender.close() +### Cancellation and timeouts + +Set `timeoutMs` per query (or `egressSession.queryTimeoutMs` by default). +A deadline cancels the query and reports `QwpEgressQueryTimeoutError`. +Leaving a `for await` loop early also cancels the query, and `completion` +then rejects with `QwpEgressQueryAbandonedError`. `query.cancel()` requests +cancellation but does not wait for it; the query then fails with +`QwpEgressQueryError` and `status` `QWP_STATUS.CANCELLED`. + +Cancellation is prompt only with a credit window (see +[Flow control](#flow-control)). Without one, the server keeps streaming after +a cancel: returning the lease waits up to `query_close_timeout_ms` (5 seconds +by default) for the stream to drain, can take about twice that in total, and +then discards the connection. Set `initialCredit` on any query you may time +out, cancel, or stop early: + +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;"); +try { + const lease = await db.borrowQuery(); + try { + const query = await lease.query("SELECT timestamp, symbol, price FROM trades", { + timeoutMs: 5_000, + initialCredit: 1024 * 1024, + }); + let seen = 0; + let stoppedEarly = false; + for await (const batch of query) { + seen += batch.rowCount; + if (seen >= 10_000) { + stoppedEarly = true; + break; // cancels the rest of the result + } + } + // After a break, completion rejects with QwpEgressQueryAbandonedError. + if (!stoppedEarly) await query.completion; + } finally { + await lease.close(); + } +} finally { + await db.close(); } +``` + +### Flow control -run().then(console.log).catch(console.error) +Without a credit window, the server streams as fast as it can: a slow consumer +can buffer a large result in memory, and cancelling the query is slow. Set +`initialCredit: 1024 * 1024` on a query, as above, or `initial_credit=1048576` +in the connect string; `initial_credit` takes plain bytes, not a size suffix +such as `1m`. The client replenishes credit as your loop consumes batches. Use +`autoCredit: false` and `query.grantCredit(bytes)` for manual control. + +### Zero-copy result views + +For hot paths, `lease.queryViews(sql, callback)` reads typed values directly +from received bytes instead of materializing arrays. Views and byte slices +are valid only until the callback returns; copy them if you need to retain +them. + +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +// failover=off: a re-run after a lost connection would count batches twice. +const db = await connectQwpNodeClient("ws::addr=localhost:9000;failover=off;"); +try { + const lease = await db.borrowQuery(); + try { + let notional = 0; + const query = await lease.queryViews( + "SELECT price, amount FROM trades WHERE symbol = 'ETH-USD'", + (batch) => { + const price = batch.column(0); + const amount = batch.column(1); + for (let row = 0; row < batch.rowCount; row++) { + if (!price.isNull(row) && !amount.isNull(row)) { + notional += price.getDouble(row) * amount.getDouble(row); + } + } + }, + { initialCredit: 1024 * 1024 }, + ); + await query.completion; + console.log("ETH-USD notional:", notional); + } finally { + await lease.close(); + } +} finally { + await db.close(); +} ``` -As you can see, both events now are using the same timestamp. We recommended to -use the original event timestamps when ingesting data into QuestDB. Using the -current timestamp hinder the ability to deduplicate rows which is -[important for exactly-once processing](/docs/connect/compatibility/ilp/overview/#exactly-once-delivery-vs-at-least-once-delivery). +The callback adds to a running total, so the example turns failover off: with +failover on, a lost connection re-runs the query and its batches are added +again. If the query fails with `QwpEgressSessionClosedError`, run it again +from an empty total; see [Query failover](#query-failover). See the +[client API reference](https://questdb.github.io/nodejs-questdb-client/modules/_questdb_nodejs-client.html) +for the typed column getters. -## Decimal insertion +### Compression -:::note -Decimal columns are available with ILP protocol version 3 (QuestDB v9.2.0+ and NodeJS client v4.2.0+). +Query results default to `compression=raw`. Use `compression=zstd` (or `auto`) +for large results; `compression_level=3` is accepted only with `zstd` or +`auto`. Compression does not affect ingestion. -HTTP/HTTPS connections negotiate this automatically (`protocol_version=auto`), while TCP/TCPS connections must opt in explicitly (for example `tcp::...;protocol_version=3`). Once on v3, you can choose between the textual helper and the binary helper. -::: +## Error handling -:::caution -QuestDB does not auto-create decimal columns. Define them ahead of ingestion with -`DECIMAL(precision, scale)` so the server knows how many digits to store, as explained in the -[decimal data type](/docs/query/datatypes/decimal/#creating-tables-with-decimals) guide. -::: +Handle the failure at the stage where it occurs. Every error class and +constant named on this page is exported from `@questdb/nodejs-client`, so you +can import it for `instanceof` checks: + +| Failure | Action | +|---|---| +| Local value validation | Fix the value; the row in progress was discarded. Test `QwpBatchTooLargeError` before `RangeError` because it extends `RangeError`. | +| `QwpMemoryReplayAppendTimeoutError` / `QwpReplayStoreAppendTimeoutError` | The batch stays staged. Slow down and retry the flush, not the rows. | +| ACK timeout: `QwpIngressAckTimeoutError` from `waitForAcknowledged()`, or a plain `Error` from a `flush()` that waits for an acknowledgement | Not a rejection: QuestDB may still acknowledge the batch. Do not write the rows again; wait again, or raise `ackTimeoutMs` or `durableAckTimeoutMs`. See [Awaiting acknowledgements](#awaiting-acknowledgements). | +| Server rejection: `QwpIngressNackError` from `waitForAcknowledged()`, or an `onSenderError` report | See [Ingestion errors](#ingestion-errors); a terminal rejection fails the sender. | +| `QwpEgressQueryError` | Check `status`: fix SQL or bind values for `PARSE_ERROR`. `CANCELLED` comes from your own `query.cancel()` or, with `failover=off`, from a server shutdown: retry on a new lease only a query you did not cancel. The lease remains usable after a SQL error. | +| `QwpEgressSessionClosedError` | The query connection was lost with failover off. Close the lease and retry the whole query. | +| `QwpReconnectExhaustedError` | Close the failed sender or query lease and borrow a new one. | + +### Ingestion errors -### Text literal (easy to use) +A local column or `at()` validation error discards the unfinished row. +A **server** rejection can arrive after `flush()` resolves: register +`ingressSession.onSenderError` to receive it, and branch on the error's +`appliedPolicy` and `category`. ```typescript -import { Sender } from "@questdb/nodejs-client"; +import { + connectQwpNodeClient, + QWP_SENDER_ERROR_CATEGORY, + QWP_SENDER_ERROR_POLICY, +} from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;", { + ingressSession: { + onSenderError: (error) => { + // Keep serverMessage out of external trackers: it may contain row values. + const details = { + category: error.category, + table: error.tableName, + fromFsn: error.fromFsn, + toFsn: error.toFsn, + }; + if (error.appliedPolicy === QWP_SENDER_ERROR_POLICY.TERMINAL) { + // The sender has stopped; see "Recovering from a terminal rejection". + console.error("QuestDB rejected a batch; ingestion stopped", details); + } else if (error.appliedPolicy === QWP_SENDER_ERROR_POLICY.ABANDONED) { + console.error("rows set aside in", error.quarantinedPath, details); + } else if (error.category === QWP_SENDER_ERROR_CATEGORY.WRITE_ERROR) { + console.warn("QuestDB could not write a batch; resending", details); + } else { + console.warn("QuestDB rejected a batch; resending", details); + } + }, + onError: (event) => { + // terminal: true means the sender has stopped; other events are warnings. + if (event.terminal) { + console.error("QuestDB ingestion stopped:", event.error.message); + } + }, + }, +}); +await db.close(); +``` -async function runDecimalsText() { - const sender = await Sender.fromConfig( - "tcp::addr=localhost:9009;protocol_version=3", - ); +Each error has `category`, `appliedPolicy`, `serverStatusByte`, +`serverMessage`, `messageSequence`, the rejected frame range `fromFsn` to +`toFsn`, `tableName` when the server reports one, `detectedAtMs`, and, when +abandoned store-and-forward data was preserved on disk, `quarantinedPath`. +Categories and policies are lowercase, hyphenated strings; compare them with +the `QWP_SENDER_ERROR_CATEGORY` and `QWP_SENDER_ERROR_POLICY` constants. Each +category has a fixed default policy, because Node.js does not apply the +`on_*_error` keys: + +| Category | Default policy | Meaning | +|---|---|---| +| `schema-mismatch` | `terminal` | The batch does not match the table schema | +| `parse-error` | `terminal` | QuestDB could not parse the batch | +| `security-error` | `terminal` | QuestDB denied the write, for example by ACL | +| `protocol-violation` | `terminal` | The client and server disagree on the protocol | +| `write-error` | `retriable` | The write failed, for example on a table that is not accepting writes | +| `internal-error` | `retriable` | An unexpected server-side failure | +| `dictionary-gap` | `retriable` | The connection lacks symbol dictionary entries; the client resends them | +| `not-writable` | `retriable-other` | The node cannot accept writes; the client tries another endpoint | +| `cancelled`, `limit-exceeded` | `retriable` | Current servers send these statuses only on query connections | +| `unknown` | `retriable` | A status this client does not know, for example from a newer server | +| `data-loss` | `abandoned` | Store-and-forward data was set aside; see [Quarantined journal slots](#quarantined-journal-slots) | + +The handler also runs for retriable rejections, which the client resends: only +`terminal` (the sender stops) and `abandoned` (journal data set aside) mean +the rows are not being delivered. Without a callback, the client logs +rejections. `waitForAcknowledged()` rejects with `QwpIngressNackError` for a +terminally rejected batch; its `senderError` property has the fields above. +Branch on category, not the unstable message +text, and redact messages before sending them to external trackers. +`ingressSession.onError` receives an event with `error`, `terminal`, +`timestampMs`, and, for a server rejection, `senderError`. It also reports +non-terminal problems, such as ACK timeouts; `terminal: true` means the sender +has stopped. + +#### Recovering from a terminal rejection + +A terminal batch rejection stops that sender, and its `close()` rejects with +`QwpReplayRejectedError`. Without `sf_dir`, unacknowledged rows on the failed +sender are lost: fix the row or schema before retrying. With `sf_dir`, the +rejected batch stays at the front of the journal and blocks **all** tables +using it, across restarts. A pooled sender is replaced after a failed +`close()`, but its replacement sees the same blocked journal. To unblock the +journal: + +1. Stop the process that owns the slot: `/-` for a + pooled sender, or `/` for a standalone one. The + `onSenderError` report, or the client's log line, gives the category and + the server message. If the slot stays locked after a crash, see + [Lock recovery](#sf-lock-recovery). +2. Either fix the cause on the server so that every journaled batch is + accepted, for example by changing a conflicting column with + `ALTER TABLE ... ALTER COLUMN ... TYPE`, or move the slot directory out of + `sf_dir` to discard its unacknowledged rows. Keep the moved directory for + inspection. +3. Start the client again with the same `sf_dir` and `sender_id`. After a + server-side fix, it replays the whole journal, including the rejected + batch. After a move, it starts with an empty slot. + +#### Quarantined journal slots + +Store-and-forward does not delete rows that it cannot deliver. It sets them +aside and reports them to `onSenderError` with category `data-loss` and +policy `abandoned`: + +- **Corrupt journal.** When a sender opens a slot whose journal is + structurally corrupt, the client renames the slot directory to + `.unreplayable-N`, adds a `.failed` file, and continues with an empty + slot. The error's `quarantinedPath` names the renamed directory. The client + never replays it; keep it for inspection. +- **Undeliverable slot.** When the background drainer (see + [Replaying the journal after a restart](#replaying-the-journal-after-a-restart)) + cannot deliver a slot, for example because QuestDB terminally rejects the + oldest batch or rejects authentication, it adds a `.failed` file to the + slot and stops retrying it. The rows stay in the slot. + +To replay an undeliverable slot, fix the cause, then remove its marker with +`retryQwpNodeOrphanSlot()`. A running client replays the slot on its next +scan, within 30 seconds: - await sender - .table("fx") - .symbol("pair", "EURUSD") - .decimalColumnText("mid", "1.234500") // keeps trailing zeros - .atNow(); +```typescript +import { retryQwpNodeOrphanSlot } from "@questdb/nodejs-client"; - await sender.flush(); - await sender.close(); +await retryQwpNodeOrphanSlot("/var/lib/myapp/qdb-sf/trades-1"); +``` + +### Query errors + +SQL errors reject query iteration and `completion` with +`QwpEgressQueryError` (`status`, `message`, `requestId`); the lease remains +usable. Compare `status` with `QWP_STATUS` constants instead of matching +server error text: + +```typescript +import { connectQwpNodeClient, QWP_STATUS, QwpEgressQueryError } from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;"); +try { + const lease = await db.borrowQuery(); + try { + const query = await lease.query("SELECT no_such_column FROM trades"); + for await (const batch of query) console.log(batch.rowCount); + await query.completion; + } catch (error) { + if (!(error instanceof QwpEgressQueryError)) throw error; + const invalidSql = error.status === QWP_STATUS.PARSE_ERROR; + console.error(invalidSql ? "invalid SQL:" : "query failed:", error.message); + } finally { + await lease.close(); + } +} finally { + await db.close(); +} +``` + +`requestId` numbers queries per connection; it is not a server-side +correlation ID. A server that shuts down interrupts its running queries. With +failover on (the default), the client treats this as a lost connection: it +runs the query again from its first row, or fails with +`QwpReconnectExhaustedError` once failover gives up; see +[Query failover](#query-failover). With `failover=off`, the query fails with +`status` `QWP_STATUS.CANCELLED`; retry it on a new lease. Your own +`query.cancel()` also produces `CANCELLED`, so do not retry a query you +cancelled. Other failures are separate classes, not `QwpEgressQueryError`: + +- `QwpEgressQueryTimeoutError`: `timeoutMs` expired. +- `QwpEgressQueryAbandonedError`: the loop ended early; see + [Cancellation and timeouts](#cancellation-and-timeouts). +- `QwpEgressQueryCancelTimeoutError`: a cancellation did not drain in time. +- `QwpEgressSessionClosedError`: the connection was lost with failover off. +- `QwpReconnectExhaustedError`: failover gave up. + +After a cancellation or a lost connection, return the lease before borrowing +another. + +### Connection-level errors + +The pool wraps a failure to open a connection in `QwpPoolResourceError`. +Follow its `cause` chain to the reason: + +- **Retried first connection.** If the first connection is retried, the next + cause is a `QwpReconnectExhaustedError` that wraps the last attempt's error. + Queries retry it with `failover=on`, any `failover_*` key, or a typed + `egressSession.reconnect`; senders retry it with `initial_connect_retry=on` + or any `reconnect_*` key (see [Startup and outage modes](#ingestion-modes)). +- **Several endpoints.** When every endpoint fails, for example because none is + reachable, the error is a `QwpFailoverError` whose `attempts` records each + endpoint's failure. +- **One endpoint.** The error is that endpoint's `QwpUpgradeError`. +- **Authentication.** A `401` or `403` stops the endpoint sweep and is not + retried, so the cause is the `QwpUpgradeError` itself, even with several + endpoints or a retried first connection. + +A `QwpUpgradeError` covers any failure while opening the WebSocket, so check +its `kind`: `authentication` for an HTTP `401` or `403`, `transport` or +`timeout` when QuestDB is unreachable, and others such as `role-rejected` and +`version-mismatch`. `QwpRoleMismatchError` and `QwpDurableAckUnavailableError` +extend `QwpUpgradeError`, so test for them first. A borrow at pool capacity +times out as `QwpPoolAcquireTimeoutError` after `acquire_timeout_ms` +(5 seconds by default). + +```typescript +import { + connectQwpNodeClient, + QwpFailoverError, + QwpPoolResourceError, + QwpReconnectExhaustedError, + QwpUpgradeError, +} from "@questdb/nodejs-client"; + +// Skip the pool and retry wrappers to reach the connection error itself. +function connectionError(error: unknown): unknown { + let cause = error; + while ( + (cause instanceof QwpPoolResourceError || + cause instanceof QwpReconnectExhaustedError) && + cause.cause !== undefined + ) { + cause = cause.cause; + } + return cause; +} + +try { + const db = await connectQwpNodeClient("ws::addr=localhost:9000;"); + console.log("connected to QuestDB"); + await db.close(); +} catch (error) { + const cause = connectionError(error); + if (cause instanceof QwpUpgradeError && cause.kind === "authentication") { + console.error("QuestDB rejected the credentials"); + } else if (cause instanceof QwpFailoverError) { + console.error("no endpoint accepted the connection:", cause.attempts); + } else { + console.error("cannot connect to QuestDB:", cause); + } + process.exitCode = 1; +} +``` + +An authentication rejection never moves the client to another endpoint. It is +terminal before a sender's first successful connection. After that, senders +with `sf_dir` or a background start retry it indefinitely, keep buffering, +and emit an `attempt-failed` [connection event](#connection-events) for each +failed attempt; other senders and query connections treat it as terminal. See +[Authentication is cluster-wide](/docs/high-availability/client-failover/concepts/#authentication-is-cluster-wide). + +#### Connection timeouts + +`connect_timeout` and `auth_timeout_ms` default to 15 seconds. They cover +connection setup and WebSocket upgrade; the query connection also waits for +the server's initial information frame. Set shorter timeouts if a request +needs a tighter deadline. + +## Failover and high availability + +Multi-host failover requires QuestDB Enterprise replication; reconnecting to +a single restarted server also works in open source. + +### Multiple endpoints + +```text +wss::addr=db-a.example.com:9000,db-b.example.com:9000;token=YOUR_TOKEN; +``` + +Ingestion needs the primary; queries can use any healthy node. `target` +filters the roles queries accept (`any`, `primary`, `replica`), while `zone` +prefers same-zone nodes. On Node.js, setting `target=replica` **in the shared +connect string also filters ingestion**, so use the typed query-only option +`{ egress: { target: "replica" } }` instead. `target=replica` is strict, not +"prefer replica and fall back to primary". If no replica is up at startup, +set `query_pool_min=0` to defer the query connection. + +For a query-only client with only replica endpoints, set `sender_pool_min=0`: +otherwise the default pool prewarms a sender and startup fails because no +primary can accept its write connection. Set the replica filter in the typed +query options so it applies only to queries. If you later borrow a sender, +include a primary endpoint in `addr`. + +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient( + "wss::addr=db-a.example.com:9000,db-b.example.com:9000;" + + "token=YOUR_TOKEN;sender_pool_min=0;", + { egress: { target: "replica" } }, +); +try { + const lease = await db.borrowQuery(); + try { + const query = await lease.query( + "SELECT timestamp, symbol FROM trades LIMIT 10", + ); + for await (const batch of query) { + for (const row of batch.rows()) console.log(row); + } + await query.completion; + } finally { + await lease.close(); + } +} finally { + await db.close(); } ``` -`decimalColumnText` accepts strings or numbers. String literals go through `validateDecimalText` and are written verbatim with the `d` suffix, so every digit (including trailing zeros or exponent form) is preserved. Passing a number is convenient, but JavaScript’s default formatting will drop insignificant zeros. +A query that fails over restarts from its first row; see +[Query failover](#query-failover) before accumulating results. + +### Ingestion reconnect + +Senders resend unacknowledged batches after a disconnect. Between attempts, +they wait a random delay below a ceiling that starts at +`reconnect_initial_backoff_millis` (100 ms by default) and doubles up to +`reconnect_max_backoff_millis` (5 seconds). A sender in default memory mode +gives up after `reconnect_max_duration_millis` (5 minutes by default): it +fails with `QwpReconnectExhaustedError` and its unacknowledged rows are lost. +Background memory mode and store-and-forward retry indefinitely, subject to +queue or journal capacity. Setting any `reconnect_*` key also makes senders +retry their first connection; see +[Startup and outage modes](#ingestion-modes). + +### Query failover + +Query failover is on by default. A lost query connection re-executes the +query **from its first row**, even if your loop has already consumed rows. The +re-executed query reads the data as it is then, so it can return fewer rows +than the first attempt, or no batches at all. For streaming results you cannot +retract, use `failover=off` and retry the whole operation when the query fails +with `QwpEgressSessionClosedError`. Otherwise buffer the result and reset the +buffer in `egressSession.onReplayReset`. Give that callback a client with one +query connection (`query_pool_max=1`), because request IDs are per connection, +not unique across pooled leases: + +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const rows: unknown[][] = []; +// One query connection, so every replay reset belongs to the query below. +const db = await connectQwpNodeClient( + "ws::addr=localhost:9000;query_pool_max=1;sender_pool_min=0;", + { egressSession: { onReplayReset: () => { rows.length = 0; } } }, +); +try { + const lease = await db.borrowQuery(); + try { + const query = await lease.query( + "SELECT timestamp, symbol, price FROM trades WHERE symbol = 'ETH-USD'", + { initialCredit: 1024 * 1024 }, + ); + for await (const batch of query) { + for (const row of batch.rows()) rows.push([...row]); + } + await query.completion; + console.log(`${rows.length} rows`); + } finally { + await lease.close(); + } +} finally { + await db.close(); +} +``` -### Binary form (high throughput) +`batch.batchSequence === 0n` detects a nonempty replay but not a replay +returning no batches. A single-row aggregate, such as `count()` or `avg()` +without `GROUP BY`, needs no reset: every execution returns exactly one row, +so keep the last row you receive. + +Between failover attempts, the client waits a random delay below a ceiling +that starts at `failover_backoff_initial_ms` (50 ms by default) and doubles up +to `failover_backoff_max_ms` (1 second). It makes at most +`failover_max_attempts` (8) attempts within `failover_max_duration_ms` +(30 seconds). Against refused connections, as while a server restarts, eight +attempts take only a few seconds, so the attempt limit ends failover long +before the time budget: raise `failover_max_attempts` together with +`failover_max_duration_ms` for longer outages. When failover gives up, the +query fails with `QwpReconnectExhaustedError`; close the lease and borrow a +new one. + +### Typed reconnect policy + +Typed `ingressSession.reconnect` and `egressSession.reconnect` objects +**replace** their connect-string policies: a field you omit takes the default +below, not the connect-string value, so set every limit you rely on in the +typed object. Durations are in milliseconds, and `maxDurationMs: 0` removes +the time limit. + +| Field | Ingestion key (default) | Query key (default) | +|---|---|---| +| `initialBackoffMs` | `reconnect_initial_backoff_millis` (100) | `failover_backoff_initial_ms` (50) | +| `maxBackoffMs` | `reconnect_max_backoff_millis` (5000) | `failover_backoff_max_ms` (1000) | +| `maxDurationMs` | `reconnect_max_duration_millis` (300000) | `failover_max_duration_ms` (30000) | +| `maxAttempts` | No key (0, unlimited) | `failover_max_attempts` (8) | +| `maxFrameRejections` | `max_frame_rejections` (4) | Not used | +| `poisonMinEscalationWindowMs` | `poison_min_escalation_window_millis` (300000) | Not used | +| `onEvent` | No key; see [Connection events](#connection-events) | No key | + +- **Ingestion.** `reconnect_*` keys trigger first-connection retry; a typed + ingestion `reconnect` object does not. +- **Queries.** The first query connection is retried only if one of these is + set: + - `failover=on`, explicitly; + - a `failover_*` key, without `failover=off`; + - a typed `egressSession.reconnect` object. +- **Failover off.** A typed `egressSession.reconnect` object turns query + failover back on even when the connect string says `failover=off`. Do not + add one, even just for `onEvent`, to a client that must not re-execute SQL. + +### Connection events + +Set `ingressSession.reconnect.onEvent` or `egressSession.reconnect.onEvent` to +observe connection events. Each event has a `kind`, an `attempt` number, +`timestampMs`, and, where relevant, `endpoint`, `previousEndpoint`, and +`cause`. The kinds are `connected`, `reconnecting`, `attempt-failed` (one per +failed connection attempt, with its `cause`), `reconnected`, `failed-over`, +and `durable-ack-unavailable`. The store-and-forward background drainer (see +[Replaying the journal after a restart](#replaying-the-journal-after-a-restart)) +sends its events to the same `ingressSession.reconnect.onEvent` and also +reports `primary-unavailable` and `durable-ack-persistent-failure`. +`QWP_RECONNECT_EVENT_KIND` lists them all. ```typescript -const sender = await Sender.fromConfig( - "tcp::addr=localhost:9009;protocol_version=3", +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;", { + ingressSession: { + // A typed reconnect object replaces the connect-string policy: set its limits too. + reconnect: { + maxDurationMs: 300_000, + onEvent: (event) => { + if (event.kind === "attempt-failed") { + console.warn("connection attempt failed", event.endpoint, event.cause); + } else { + console.info("connection event", event.kind, event.endpoint); + } + }, + }, + }, +}); +await db.close(); +``` + +No event marks a terminal failure: use +[`ingressSession.onError`](#ingestion-errors) with +`terminal: true` for that. An `egressSession.reconnect` object added for +`onEvent` re-enables query failover on a `failover=off` client; see +[Typed reconnect policy](#typed-reconnect-policy). Connect-string +`connection_listener_inbox_capacity` configures ingestion events only; for +query events, use typed `egressSession.connectionListenerInboxCapacity`. + +## Concurrency + +Share one `QwpClient`, but keep one sender per producer and one query lease per +concurrent query. Worker threads need their own clients and, with `sf_dir`, +distinct `sender_id` values. + +### Writing from request handlers + +In a Node.js service, every request handler that writes rows is a concurrent +producer, even though all handlers run on one thread. Two patterns work: + +- **Borrow per request.** `borrowSender()` hands out an idle pooled sender + without reconnecting, and `close()` flushes the request's rows and returns + the sender. At most `sender_pool_max` handlers (4 by default) hold a sender + at once; another borrow waits up to `acquire_timeout_ms` (5 seconds by + default), then fails with `QwpPoolAcquireTimeoutError`. Each request sends + its own batch. With `sf_dir`, each pooled sender journals into its own + `-` slot. +- **One shared sender.** Borrow one sender at startup and build each row in + one synchronous chain from `table()` to `at()` or `atNow()`, with no `await` + in between. Rows from concurrent handlers then never interleave, auto-flush + batches them together, and flushes are serialized. A handler that awaits + mid-row makes the next handler's `table()` throw. Auto-flush runs only when a + row is added, so also call `flush()` from a timer, or the last rows wait for + the next request. A terminal failure stops the sender for every handler, + and with `transaction=on` all handlers share one transaction. + +With a background start, `borrowSender()`, `at()`, `flush()`, and `close()` +return promptly while QuestDB is down, until the replay queue or journal is +full (see [Backpressure](#backpressure)); only `borrowQuery()` fails until +QuestDB is reachable. In default memory mode, `flush()`, `close()`, and an +auto-flushing `at()` wait for the reconnect instead; see +[Startup and outage modes](#ingestion-modes). + +```typescript +import { createServer } from "node:http"; +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +// A background start lets the service start, and record trades, while +// QuestDB is down. +const db = await connectQwpNodeClient( + "ws::addr=localhost:9000;lazy_connect=on;sf_max_segment_bytes=1m;", ); -const scale = 4; -const notional = 12345678901234567890n; // represents 1_234_567_890_123_456.7890 +async function recordTrade(price: number, amount: number): Promise { + const sender = await db.borrowSender(); + try { + await sender + .table("trades") + .symbol("symbol", "ETH-USD") + .symbol("side", "buy") + .doubleColumn("price", price) + .doubleColumn("amount", amount) + .at(Date.now(), "ms"); + } finally { + await sender.close(); // flushes the row and returns the sender + } +} -await sender - .table("positions") - .symbol("desk", "ny") - .decimalColumnUnscaled("notional", notional, scale) - .atNow(); +const server = createServer((req, res) => { + recordTrade(2615.54, 0.5).then( + () => res.end("recorded\n"), + (error) => { + console.error("could not record the trade:", error); + res.statusCode = 503; + res.end(); + }, + ); +}); +server.listen(8080); -await sender.flush(); -await sender.close(); +process.once("SIGTERM", () => { + // db.close() waits up to close_flush_timeout_millis for acknowledgements. + server.close(() => void db.close()); +}); ``` -`decimalColumnUnscaled` converts `BigInt` inputs into the ILP v3 binary payload. You can also pass an `Int8Array` if you already have a two’s-complement, big-endian byte -array. The scale must stay between 0 and 76, and payloads wider than 32 bytes are rejected up front. This binary path keeps rows compact, making it the preferred option for high-performance feeds. + + +## Configuration reference + +The [connect string reference](/docs/connect/clients/connect-string/) lists +shared keys and defaults; Node.js differences follow the typed options. -## Configuration options +### Programmatic options -The minimal configuration string needs to have the protocol, host, and port, as -in: +The second argument of `connectQwpNodeClient` accepts typed `sender`, +`ingressSession`, `egressSession`, `egress`, `webSocket`, `storeAndForward`, and +`pool` options. For example: +```typescript +import { connectQwpNodeClient } from "@questdb/nodejs-client"; + +const db = await connectQwpNodeClient("ws::addr=localhost:9000;", { + sender: { awaitServerAck: true }, + ingressSession: { + onSenderError: (error) => + console.error("rejected batch:", error.category, error.serverMessage), + }, + pool: { senderPoolMax: 2 }, +}); +await db.close(); ``` -http::addr=localhost:9000; + +Typed options override the corresponding connect-string settings. A typed +`reconnect` object replaces the whole reconnect policy from the string; see +[Typed reconnect policy](#typed-reconnect-policy). For the full typed API, see +the +[client reference](https://questdb.github.io/nodejs-questdb-client/modules/_questdb_nodejs-client.html). + +### Differences from other clients + +| Area | Node.js behavior | +|---|---| +| Startup/outage | `connectQwpNodeClient()` also opens a query connection, so it starts while QuestDB is down only with `query_pool_min=0`, which `lazy_connect=on` sets. A sender in default memory mode gives up after `reconnect_max_duration_millis` per outage. See [Startup and outage modes](#ingestion-modes). | +| Authentication | After a sender's first successful connection, `401`/`403` is retried indefinitely by senders with `sf_dir` or a background start; other senders and queries treat it as terminal. | +| Query startup | First-connect retry requires explicit `failover=on`, a `failover_*` key (without `failover=off`), or typed `egressSession.reconnect`. | +| `target`, `zone` | Also apply to ingestion when set in the connect string; use typed `egress.target` for queries only. | +| `sf_dir` | Node.js recursively creates missing parents and a slot; pooled senders use slots named `-`. Its `.lock.owner` directory can outlive a crashed process; see [Lock recovery](#sf-lock-recovery). | +| Background drainer | With `sf_dir`, a pooled client replays slots of its own `sender_id` that no running sender holds, even without `drain_orphans`; `drain_orphans=on` adds other `sender_id`s. See [Replaying the journal after a restart](#replaying-the-journal-after-a-restart). | +| SF-only keys | Explicit `sf_durability` (even `memory`), `sf_sync_interval_millis`, `drain_orphans`, `max_background_drainers`, and `catch_up_cap_gap_min_escalation_window_millis` require `sf_dir`. `sf_durability=append` is supported. | +| `sf_max_total_bytes` | With `sf_dir` it is a journal size **target**, not a disk quota; without `sf_dir` it caps the memory queue. | +| Durable ACK | Background-started senders, including with `sf_dir`, retry an unavailable capability from startup. Explicit `durable_ack_keepalive_interval_millis` also requests durable ACK even at `0`; negatives are rejected. | +| Timeouts | `connect_timeout` defaults to 15 seconds and also bounds DNS and the TLS handshake; `auth_timeout_ms` defaults to `connect_timeout`. `close_flush_timeout_millis` defaults to 5 seconds. When it expires, a standalone sender's `close()` rejects with `QwpSenderCloseTimeoutError`, while `db.close()` resolves and usually reports a non-terminal `QwpIngressAckTimeoutError` to `ingressSession.onError`. | +| `poison_min_escalation_window_millis` | Defaults to 300000 (5 minutes), not 5000. | +| Auto-flush | `auto_flush_interval` counts from the last flush (or sender creation), not from the first buffered row. `auto_flush_bytes` defaults to 0 (off). | +| `max_lifetime_ms` | Closes only idle connections above the pool minimum; the minimum connections are never recycled. | +| `connection_listener_inbox_capacity` | Configures only ingestion events, not query events. | +| Parsing | Use `0`, not `off`, for `auto_flush_rows` and `auto_flush_interval`. Size keys take single-letter suffixes (`k`, `m`, `g`, `t`), but `initial_credit` takes plain bytes. `compression_level` requires `compression=zstd` or `auto`. `tls_roots` must be PEM, and `tls_roots_password` is rejected; without `tls_roots`, certificates are checked against Node.js's bundled CA store. `init_buf_size` and `max_buf_size` are rejected on `ws` and `wss` strings. | +| Reconnect jitter | Ingestion uses full jitter, unlike the shared guide's equal-jitter ingestion schedule. | +| Error categories | `onSenderError` reports categories and policies as lowercase, hyphenated strings, such as `schema-mismatch` and `retriable-other`, and adds the `cancelled` and `limit-exceeded` categories. | +| `on_*_error` | Accepted by every entry point but not applied; observe rejections with `ingressSession.onSenderError`. | +| Typed SF options | A typed `storeAndForward` option passed with a connect string, as in `connectQwpNodeClient(conf, { storeAndForward })`, keeps the connect-string defaults: memory durability, 10 GiB, and wait-on-full. Without a connect string, as in `connectQwpNodeSender({ url, storeAndForward })`, the defaults are append durability, 1 GiB, and fail-on-full. | +| Standalone `Sender` | Ignores pool and query-only keys with a warning, but applies `client_id` and `lazy_connect`. | + +## Migration + +### From ILP to QWP + +The existing `Sender` row API can migrate from `http::` or `tcp::` to `ws::` +or `wss::`, followed by `await sender.connect()`. QWP adds querying, replay, +store-and-forward, and richer types. Unlike ILP HTTP, a QWP `flush()` can +resolve **before** the server accepts the rows, so wait for an ACK when +committing source offsets. Legacy ILP-only keys such as `retry_timeout`, +`init_buf_size`, and `tls_ca` are rejected on QWP strings. The +[Standalone Sender](#standalone-sender) example shows the QWP form of the row +API; [ILP transports](#ilp-transports-legacy) shows the ILP form. + +### Upgrading from 4.x + +Version 5.0.0 requires Node.js 20.18.1 or newer. Existing ILP senders now +omit columns passed `null` or `undefined` (instead of throwing for most +values); `decimalColumn()` rejects non-integer scales instead of silently +coercing them. The package adds `ws` for QWP and retains the old ILP +transports. + +## Full example: ingestion and querying with failover + +This program combines the production settings from the sections above: TLS +and a token, two endpoints, a background start with store-and-forward, +deduplicated replays, error callbacks, an acknowledgement wait, and a query +that is safe under failover. If QuestDB is unreachable, the program still +starts and journals the rows, and `borrowQuery()` fails once query failover +gives up. Create the table first, for example as a migration step, so that +replays are deduplicated even if an ingester starts while QuestDB is down: + +```questdb-sql +CREATE TABLE IF NOT EXISTS trades_sf ( + timestamp TIMESTAMP, + trade_id VARCHAR, + symbol SYMBOL, + price DOUBLE +) TIMESTAMP(timestamp) PARTITION BY DAY +DEDUP UPSERT KEYS(timestamp, trade_id); ``` -For all the extra options you can use, please check -[the client docs](https://questdb.github.io/nodejs-questdb-client/classes/SenderOptions.html) +```typescript +import { + connectQwpNodeClient, + QWP_SENDER_ERROR_POLICY, + QwpIngressAckTimeoutError, +} from "@questdb/nodejs-client"; + +const token = process.env.QDB_TOKEN; +if (!token) throw new Error("QDB_TOKEN is not set"); + +const db = await connectQwpNodeClient( + "wss::addr=db-a.example.com:9000,db-b.example.com:9000;" + + `token=${token};` + + // Start while QuestDB is down and journal rows across restarts. + "lazy_connect=on;sf_dir=/var/lib/myapp/qdb-sf;sender_id=trades;" + + "sf_max_segment_bytes=1m;" + + // Bound query buffering. Let query failover retry for up to a minute: + // the default cap of 8 attempts would otherwise end it within seconds. + "initial_credit=1048576;failover_max_attempts=1000;" + + "failover_max_duration_ms=60000;", + { + ingressSession: { + onSenderError: (error) => { + if (error.appliedPolicy === QWP_SENDER_ERROR_POLICY.TERMINAL) { + console.error("ingestion stopped:", error.category, error.tableName); + } + }, + onError: (event) => { + if (event.terminal) console.error("ingestion stopped:", event.error); + }, + }, + }, +); +try { + // Stable trade IDs and event timestamps let DEDUP absorb replays. + const now = Date.now(); + const fills = [ + { tradeId: "trade-1001", symbol: "ETH-USD", price: 2615.54, tsMs: now }, + { tradeId: "trade-1002", symbol: "ETH-USD", price: 2615.62, tsMs: now + 1 }, + ]; + const sender = await db.borrowSender(); + try { + for (const fill of fills) { + await sender + .table("trades_sf") + .stringColumn("trade_id", fill.tradeId) + .symbol("symbol", fill.symbol) + .doubleColumn("price", fill.price) + .at(fill.tsMs, "ms"); + } + await sender.flush(); // journaled locally + try { + await sender.waitForAcknowledged(sender.publishedSequence, 10_000); + } catch (error) { + if (!(error instanceof QwpIngressAckTimeoutError)) throw error; + // Not a rejection: the journal keeps the rows and replays them. + console.warn("not acknowledged yet; the rows stay in the journal"); + } + } finally { + await sender.close(); + } + + // A single-row aggregate is safe under query failover: a replay returns + // its own row, which replaces the first attempt's. + const lease = await db.borrowQuery(); + try { + const sinceMicros = BigInt(Date.now() - 3_600_000) * 1000n; + const query = await lease.query( + "SELECT count(), avg(price) FROM trades_sf " + + "WHERE symbol = $1 AND timestamp >= $2", + { + binds: (binds) => + binds.setVarchar(0, "ETH-USD").setTimestampMicros(1, sinceMicros), + timeoutMs: 30_000, + }, + ); + let summary: unknown[] = []; + for await (const batch of query) { + for (const row of batch.rows()) summary = [...row]; + } + await query.completion; + // Rows written above may not be visible yet; see Read-after-write. + console.log("ETH-USD, last hour [trades, average price]:", summary); + } finally { + await lease.close(); + } +} finally { + await db.close(); +} +``` -Alternatively, for a breakdown of Configuration string options available across -all clients, see the [Connect string](/docs/connect/clients/connect-string/) page. +Keep `target=replica` out of this connect string: on Node.js it also filters +ingestion; see [Multiple endpoints](#multiple-endpoints). To also wait for the +upload to object storage on QuestDB Enterprise, see +[Durable acknowledgement](#durable-acknowledgement). -## Next Steps +## ILP transports (legacy) -Please refer to the [ILP overview](/docs/connect/compatibility/ilp/overview) for details -about transactions, error control, delivery guarantees, health check, or table -and column auto-creation. +The standalone `Sender` still speaks InfluxDB Line Protocol (ILP) over HTTP +and TCP, for ingestion only. ILP over HTTP sends each `flush()` as an HTTP +request and has no `connect()` step; calling it throws: -Dive deeper into the Node.js client capabilities, including TypeScript and -Worker Threads examples, by exploring the -[GitHub repository](https://github.com/questdb/nodejs-questdb-client). +```typescript +import { Sender } from "@questdb/nodejs-client"; -To learn _The Way_ of QuestDB SQL, see the -[Query & SQL Overview](/docs/query/overview/). +const sender = await Sender.fromConfig("http::addr=localhost:9000;"); +try { + await sender + .table("trades") + .symbol("symbol", "ETH-USD") + .symbol("side", "sell") + .floatColumn("price", 2615.54) + .floatColumn("amount", 0.00044) + .at(Date.now(), "ms"); + await sender.flush(); // close() does not flush: unflushed rows are lost +} finally { + await sender.close(); +} +``` -Should you encounter any issues or have questions, the -[Community Forum](https://community.questdb.com/) is a vibrant platform for -discussions. +For authentication, add `username=...;password=...;` (or, on QuestDB +Enterprise, `token=...;`) to the connect string. For ILP over TCP, use +`tcp::addr=localhost:9009;` and call `await sender.connect()` before writing. +`Sender.fromEnv()` reads the connect string from the `QDB_CLIENT_CONF` +environment variable. ILP uses the same nine column methods as the standalone +QWP `Sender`; see [Column methods](#column-methods). See the +[ILP overview](/docs/connect/compatibility/ilp/overview/) for TCP +authentication, protocol versions, and transport configuration. + +## Next steps + +- [Delivery semantics](/docs/concepts/delivery-semantics/) for replay and deduplication. +- [Store-and-forward concepts](/docs/high-availability/store-and-forward/concepts/). +- [Connect string reference](/docs/connect/clients/connect-string/) and the + [Node.js API reference](https://questdb.github.io/nodejs-questdb-client/modules/_questdb_nodejs-client.html). +- [`@questdb/nodejs-client` on npm](https://www.npmjs.com/package/@questdb/nodejs-client) + and its [source on GitHub](https://github.com/questdb/nodejs-questdb-client). diff --git a/documentation/connect/compatibility/pgwire/nodejs.md b/documentation/connect/compatibility/pgwire/nodejs.md index 5815c4fae..891ff8a15 100644 --- a/documentation/connect/compatibility/pgwire/nodejs.md +++ b/documentation/connect/compatibility/pgwire/nodejs.md @@ -29,13 +29,23 @@ for performance. Our recommendation is to use the `pg` client for most use cases :::tip -For data ingestion, we recommend using QuestDB's first-party clients with -the [InfluxDB Line Protocol (ILP)](/docs/connect/overview/) instead of PGWire. PGWire should primarily be used for -querying data in QuestDB. QuestDB provides an official [JavaScript client](/docs/connect/clients/nodejs/) for data -ingestion using ILP. +For data ingestion, we recommend QuestDB's first-party clients instead of +PGWire. QuestDB provides an official +[Node.js client](/docs/connect/clients/nodejs/) with high-throughput ingestion +and streaming SQL queries over QWP. PGWire remains a good fit when you need a +standard PostgreSQL driver or ORM. ::: +:::note Example schema + +The PGWire examples below assume a pre-existing `trades` table with `ts` as +its designated timestamp and `symbol` and `price` columns. This differs from +the [QWP Node.js quick start](/docs/connect/clients/nodejs/#quick-start), which +auto-creates `trades.timestamp`. To query that table with these examples, +replace SQL `ts` and JavaScript `.ts` with `timestamp` and `.timestamp`. + +::: ## Connection Parameters @@ -747,7 +757,7 @@ async function latestByQuery() { // Get the latest values for each symbol const latest = await sql` SELECT * FROM trades - LATEST ON timestamp PARTITION BY symbol + LATEST ON ts PARTITION BY symbol ` console.log(`Latest prices for ${latest.length} symbols:`) @@ -780,9 +790,8 @@ latestByQuery() QuestDB's support for the PostgreSQL Wire Protocol allows you to use standard JavaScript PostgreSQL clients for querying time-series data. Both `pg` and `postgres` clients offer good performance and features for working with QuestDB. -We recommend the `pg` client for querying. -For data ingestion, consider QuestDB's first-party clients with the InfluxDB Line Protocol (ILP) for maximum -throughput. +Among PGWire drivers, we recommend the `pg` client for querying. For data ingestion, and for streaming query +results without a PostgreSQL driver, use the QuestDB [Node.js client](/docs/connect/clients/nodejs/), which speaks QWP. Remember that QuestDB is optimized for time-series data, so make the most of its specialized time-series functions like `SAMPLE BY` and `LATEST ON` for efficient queries. diff --git a/documentation/connect/overview.md b/documentation/connect/overview.md index 2aedc1ee5..dffe2b397 100644 --- a/documentation/connect/overview.md +++ b/documentation/connect/overview.md @@ -31,13 +31,13 @@ Pick the path that matches your environment. The first-party libraries for **Java, Python, Go, Rust, Node.js, C & C++, and .NET** are the recommended way to talk to QuestDB. They speak the **QuestDB Wire Protocol (QWP)** and unify ingest and query under one -configuration and one connection. +client configuration. Ingestion and queries run over separate WebSocket +connections, which the clients' pools manage for you. ### QWP support -QWP ships in the libraries below. The remaining language clients are being -updated — until they ship a QWP build, they continue to use ILP for ingestion -and PGWire for queries. +QWP ships in every library below. A library marked Beta may still change its +QWP API before it is declared stable. | Language | QWP support | | --------- | ----------- | @@ -45,25 +45,32 @@ and PGWire for queries. | C & C++ | ✓ Stable | | Rust | ✓ Stable | | Python | ✓ Stable | +| Node.js | ✓ Stable | | .NET | Beta | | Go | Beta | -| Node.js | Planned | Highlights: - **Binary on the wire** — roughly half the size of ILP or HTTP. - **Streaming both directions** — sustained 800 MiB/s ingress, up to 2.5 GiB/s egress on a single connection. -- **Automatic failover** — ingress and egress fail over without application - intervention. +- **Automatic failover** — ingress and egress reconnect and fail over without + application intervention. A query that fails over restarts from its first + row, so code that accumulates rows must reset them; see each client's page. - **Store-and-forward** — survives server outages, including full server destruction. Sub-200 ns offload latency. - **One configuration** — a single [connect string](/docs/connect/clients/connect-string/) drives every - option, portable across all languages. + option, with the same keys in every language. A few defaults and behaviors + differ per client, as the connect string reference notes. - **Schema-flexible** — automatic table creation and on-the-fly column additions. +The throughput and latency figures are peaks. Actual rates depend on the +client, the hardware, and the row shape: a Node.js process, for example, +encodes rows on a single CPU core. Measure with your own client and data +before sizing an ingestion tier. + Pick a language: @@ -107,8 +114,8 @@ covering the WebSocket variants for ingress and egress. Read these if you are embedding QuestDB connectivity into an existing framework. QWP also has a UDP transport for fire-and-forget metrics, supported by the -Java, Rust, C and C++ clients via the `udp` connect-string schema. It is -configured through the [`qwp.udp.*` server +Java, Python, Rust, C, C++ and Node.js clients via the `udp` connect-string +schema. It is configured through the [`qwp.udp.*` server settings](/docs/configuration/qwp/#udp-receiver) and is disabled by default; there is no separate byte-level specification page for it. diff --git a/documentation/connect/wire-protocols/qwp-client-behavior.md b/documentation/connect/wire-protocols/qwp-client-behavior.md index c4e73a3d5..7ee112984 100644 --- a/documentation/connect/wire-protocols/qwp-client-behavior.md +++ b/documentation/connect/wire-protocols/qwp-client-behavior.md @@ -76,7 +76,9 @@ Why each line matters: It bounds only the **blocking** initial connect (`initial_connect_retry=on` / `sync`). Once a sender is running, the reconnect loop never consults it and retries a transport outage forever. Setting a large value here does nothing for -a running producer. See [Reconnect and outage handling](#reconnect-and-outage-handling). +a running producer. The exception is a Node.js sender in default memory +mode, which applies it to every outage. See +[Reconnect and outage handling](#reconnect-and-outage-handling). ::: @@ -106,9 +108,10 @@ Why each line matters: - `target=replica` is required to avoid binding a primary/standalone server. The default `target=any` will accept any role. -- `failover=on` is the default. It does **not** affect startup; it only governs - reconnect+replay after a query connection that was already established later - fails during `execute()`. +- `failover=on` is the default. In the Java reference client it does **not** + affect startup; it only governs reconnect+replay after an established query + connection fails during `execute()`. Node.js also uses explicit failover + settings for initial query retries; see the [mental model](#mental-model). --- @@ -125,10 +128,23 @@ share a startup model. You must hold all three in mind: | Query client initial connect | (no mode; always synchronous) | always blocking | | Facade prewarm (how many of each connect at `build()`) | `sender_pool_min`, `query_pool_min` | eager if `min>0`, lazy if `min=0` | -`failover=on` (query default) is **not** a startup setting — it only affects -query execution after a connection exists. This naming trips people up +In Java, `failover=on` (query default) is **not** a startup setting: it only +affects query execution after a connection exists ([sharp edge #3](#known-sharp-edges)). +:::note Node.js query startup + +Node.js also retries initial query connections when `failover=on` is explicitly +set, a `failover_*` tuning key is supplied without `failover=off`, or an +`egressSession.reconnect` options object is supplied. Retries apply only to +retryable failures and stay within the failover budget. With no explicit policy, +failover is enabled for established query connections but initial connection +attempts are not retried. `failover=off` disables the reconnect wrapper; an +explicit `egressSession.reconnect` value overrides the connect-string policy. +See [Node.js connection events](/docs/connect/clients/nodejs/#connection-events). + +::: + ### Ingest initial-connect modes | `initial_connect_retry` | Mode | `build()` behavior on a down server | @@ -183,17 +199,17 @@ callers block up to `acquire_timeout_ms` then throw. | `sender_id` | `default` | | `sf_max_segment_bytes` (segment size) | `4 MiB` | | `sf_max_total_bytes` | `10 GiB` (SF mode) · `128 MiB` (memory mode) | -| `sf_durability` | `memory` (also supports `periodic`) | +| `sf_durability` | `memory` (also supports `periodic`; Node.js also `append`) | | `sf_sync_interval_millis` | `5000` (requires `sf_durability=periodic`) | | `sf_append_deadline_millis` | `30000` | -| `reconnect_max_duration_millis` | `300000` — bounds the **blocking initial connect only** | +| `reconnect_max_duration_millis` | `300000` — bounds the **blocking initial connect only** (Node.js default memory mode: every outage) | | `reconnect_initial_backoff_millis` | `100` | | `reconnect_max_backoff_millis` | `5000` | -| `close_flush_timeout_millis` | `60000` (Java/.NET) · `5000` (Rust/C/C++/Python) | -| `connect_timeout` | unset — per-endpoint TCP connect bound, must be `> 0` | +| `close_flush_timeout_millis` | `60000` (Java/.NET) · `5000` (Rust/C/C++/Python/Go/Node.js) | +| `connect_timeout` | unset (Node.js: `15000`, also bounding DNS and TLS) — per-endpoint TCP connect bound, must be `> 0` | | `auth_timeout_ms` | `15000` | | `max_frame_rejections` | `4` | -| `poison_min_escalation_window_millis` | `5000` | +| `poison_min_escalation_window_millis` | `5000` (Node.js: `300000`; Go: not supported, its window is `reconnect_max_duration_millis`) | ### Query client @@ -210,8 +226,8 @@ callers block up to `acquire_timeout_ms` then throw. There is no "retry forever" setting to look for on the reconnect keys — a running sender already does. `reconnect_max_duration_millis` applies only to a -blocking initial connect; see -[Reconnect and outage handling](#reconnect-and-outage-handling). +blocking initial connect, except on a Node.js sender in default memory mode; +see [Reconnect and outage handling](#reconnect-and-outage-handling). --- @@ -281,9 +297,9 @@ here. | --- | --- | --- | | 1 | `initial_connect_retry` is implicitly promoted to `SYNC` when any `reconnect_*` knob is set — a resilience knob silently makes startup block. | Candidate | | 2 | `reconnect_max_duration_millis` is named as if it governs reconnection, but a running sender never consults it — it bounds only the blocking initial connect. | Candidate (naming) | -| 3 | `failover` sounds like it covers startup but only affects post-connect query `execute()`. Queries have no async/lazy initial connect at all. | Candidate | +| 3 | In Java, `failover` sounds like it covers startup but only affects post-connect query `execute()`. Java queries have no async/lazy initial connect. For the Node.js exception, see the [mental model](#mental-model). | Candidate | | 4 | No first-class write-only facade: a write-only user must still supply a query config and remember `query_pool_min=0`, or use `lazy_connect=true`. | Candidate | -| 5 | A single endpoint returning `401`/`403` is treated as cluster-wide terminal and aborts the whole endpoint walk, even at startup, even if other endpoints would accept the credentials. | Intended (documented), revisit | +| 5 | A single endpoint returning `401`/`403` aborts the whole endpoint walk, even if other endpoints would accept the credentials. Some clients retry after a sender's first connection; see [Authentication is cluster-wide](/docs/high-availability/client-failover/concepts/#authentication-is-cluster-wide). | Intended (documented), revisit | | 6 | Query `serverInfoTimeoutMs` has no config key, so a facade query client cannot tune it. | Candidate | | 7 | The simplest API (`fromConfig` + async) has the worst error visibility — terminal async failures surface only on later producer calls or at `close()`. | Candidate | | 8 | `SenderProgressHandler` has no builder setter on either surface; it must be installed post-construction via `QwpWebSocketSender.setProgressHandler`. | Candidate | @@ -312,8 +328,8 @@ here. - Multiple independent senders sharing one `sf_dir` must use distinct `sender_id` values, else the second fails because the slot lock is held. - In pooled `QuestDB` usage, the pool derives per-slot IDs from the base so - pooled senders never collide. The minted name is client-specific: Java uses - `-0`, `-1`, …; the Rust, C and C++ pool uses + pooled senders never collide. The minted name is client-specific: Java and + Node.js use `-0`, `-1`, …; the Rust, C and C++ pool uses `-ingest-0`, `-ingest-1`, …. - On restart, the cursor engine opens existing segment files and replays unacknowledged frames; acknowledged/truncated frames are not replayed. @@ -344,20 +360,41 @@ With `initial_connect_retry=async`: - Terminal errors go to a configured `SenderErrorHandler`; without one they surface on later producer calls or at close-time. -A sender in async mode does not give up because time passed. What ends it is a -**terminal** condition — authentication rejection, a durable-ack capability -mismatch, or the poison-frame detector — or the producer hitting -`sf_max_total_bytes` and exhausting `sf_append_deadline_millis` on `append()`. +A sender in async mode does not give up because time passed. What stops it is +a terminal condition: a server rejection with a terminal policy (by default +`SCHEMA_MISMATCH`, `PARSE_ERROR`, `SECURITY_ERROR`, and `PROTOCOL_VIOLATION`; +see [Error frames](/docs/high-availability/store-and-forward/concepts/#error-frames)), +poison-frame escalation at any time, and an authentication rejection or +durable-ack capability mismatch before its first successful connection. After +that first connection, the Java reference client +retries authentication and durable-ack rejections instead, so a credential or +capability change on the cluster cannot stop the producer. The Rust, C, C++, +Python, Go, and .NET clients treat them as terminal. Producer calls can also +fail when the buffer reaches `sf_max_total_bytes` and exhausts +`sf_append_deadline_millis` on `append()`. + +The Node.js client retries authentication rejections after the first +connection only in senders with `sf_dir` or in background memory mode +(`initial_connect_retry=async` or `lazy_connect=on`); see +[Authentication is cluster-wide](/docs/high-availability/client-failover/concepts/#authentication-is-cluster-wide). +For unsupported durable acknowledgement, Node.js senders with a background +start (`initial_connect_retry=async` or `lazy_connect=on`) keep retrying from +startup and emit `durable-ack-unavailable`, even with `sf_dir`. With a +foreground start and `sf_dir`, the first connection fails but later mismatches +are retried after a successful connection. Monitor these +[connection events](/docs/connect/clients/nodejs/#connection-events) and buffer +usage; see [Node.js durable acknowledgement](/docs/connect/clients/nodejs/#durable-acknowledgement). ### Reconnect and outage handling -**A running sender retries a transport outage indefinitely.** There is no -wall-clock give-up and no budget-exhaustion event. Backoff grows from +**A running sender retries a transport outage indefinitely**, except the +Node.js senders described in the note below. There is no wall-clock give-up +and no budget-exhaustion event. Backoff grows from `reconnect_initial_backoff_millis` to `reconnect_max_backoff_millis` and stays there; the loop rotates through the endpoints in `addr` as it goes. -`reconnect_max_duration_millis` has exactly two consumers, neither of which is -the steady-state loop: +In the Java reference client, `reconnect_max_duration_millis` has exactly two +consumers, neither of which is the steady-state loop: 1. The **blocking sync initial connect** (`initial_connect_retry=on` / `sync`), which gives up and throws when the budget expires. @@ -374,7 +411,13 @@ has outlasted your configuration. :::note Alignment This is the behaviour of the Java reference client and the .NET client. Other -clients are aligned to it. If you are implementing a new client, the contract +clients are aligned to it, except a Node.js sender in default memory mode, +without `sf_dir`, `initial_connect_retry=async`, or `lazy_connect=on`, which +gives up after `reconnect_max_duration_millis`; see the +[Node.js client](/docs/connect/clients/nodejs/#ingestion-reconnect) and its +[other differences](/docs/connect/clients/nodejs/#differences-from-other-clients). +If you +are implementing a new client, the contract is: retry transport failures forever, surface only genuine terminal conditions, and apply back-pressure to the producer rather than dropping data. @@ -389,8 +432,8 @@ and apply back-pressure to the producer rather than dropping data. | TLS session/certificate failure | transport error; try next endpoint | | HTTP upgrade timeout / non-auth transport error | try next endpoint | | `421` with `X-QuestDB-Role: REPLICA` | role reject; try next endpoint | -| `401` / `403` auth failure | **terminal**; do not try later endpoints ⚠ | -| durable-ack requested but unsupported | terminal mismatch | +| `401` / `403` auth failure | never try later endpoints; **terminal** before the first successful connection, then client-specific: [Java and some Node.js senders retry](/docs/high-availability/client-failover/concepts/#authentication-is-cluster-wide) ⚠ | +| durable-ack requested but unsupported | terminal mismatch, except that Java senders retry after their first successful connection. Node.js senders retry from startup with a background start (with or without `sf_dir`); with foreground startup and `sf_dir`, they retry only after a first successful connection ([details](/docs/connect/clients/nodejs/#durable-acknowledgement)) | | successful write upgrade | bind this endpoint | | all endpoints fail transport | throw / retry per initial/reconnect mode | | all endpoints role-reject as replicas | `QwpRoleMismatchException` | @@ -555,5 +598,12 @@ source states the contract directly: > converts it into the durable-ack capability-gap budget. Neither bounds this > loop's steady-state reconnect. -`QwpAuthFailedException` and `WebSocketUpgradeException` raised inside the loop -are terminal across all endpoints. Everything else is retried. +`QwpAuthFailedException`, `WebSocketUpgradeException`, and +`QwpDurableAckMismatchException` raised inside the loop are terminal across +all endpoints before the sender's first successful connection, and in an +orphan drainer. After a first successful connection, the loop reports each one +to the error handler as `RETRIABLE` (`SECURITY_ERROR` for an authentication or +upgrade rejection, `PROTOCOL_VIOLATION` for a durable-ack mismatch) and keeps +retrying; see +[Authentication is cluster-wide](/docs/high-availability/client-failover/concepts/#authentication-is-cluster-wide). +Everything else is retried. diff --git a/documentation/connect/wire-protocols/qwp-egress-websocket.md b/documentation/connect/wire-protocols/qwp-egress-websocket.md index f981bd4f7..6a91d1a44 100644 --- a/documentation/connect/wire-protocols/qwp-egress-websocket.md +++ b/documentation/connect/wire-protocols/qwp-egress-websocket.md @@ -33,8 +33,8 @@ For data ingestion, see If your language already has a QuestDB client, use it — the [language client guides](/docs/query/overview) list what's available. The rest of this section is for implementers writing a new one (e.g., to bring -QWP query support to JavaScript, Rust, .NET, or runtimes that the existing -clients don't cover). +QWP query support to Ruby, PHP, or runtimes that the existing clients don't +cover). Compared with the row-oriented HTTP `/exec` JSON endpoint, QWP egress trades a denser binary encoding for higher throughput and lower CPU on both ends: diff --git a/documentation/connect/wire-protocols/qwp-ingress-websocket.md b/documentation/connect/wire-protocols/qwp-ingress-websocket.md index e27c7ea5e..297a2cb2e 100644 --- a/documentation/connect/wire-protocols/qwp-ingress-websocket.md +++ b/documentation/connect/wire-protocols/qwp-ingress-websocket.md @@ -31,8 +31,8 @@ clients, see [QWP egress (WebSocket)](/docs/connect/wire-protocols/qwp-egress-we If your language already has a QuestDB client, use it — the [language client guides](/docs/connect/overview) list what's available. The rest of this section is for implementers writing a new one (e.g., to bring -QWP to JavaScript, Rust, Ruby, .NET, or an embedded runtime that the existing -clients don't cover). +QWP to Ruby, PHP, or an embedded runtime that the existing clients don't +cover). Compared with the line-oriented ILP protocols (`http`, `https`, `tcp`), QWP trades a denser binary encoding for higher throughput and lower CPU on @@ -1127,11 +1127,15 @@ The client uses double-buffered microbatches: | Byte size | disabled | | Time since first row | 100 ms | +The Node.js client measures the interval from its last flush, or from sender +creation, instead of from the first buffered row. + ### Failover and high availability Ingress senders use a reconnect loop regardless of whether store-and-forward -is configured. The two storage modes share identical failover semantics; they -differ only in where unacknowledged data lives: +is configured. The two storage modes share the same failover semantics, apart +from the Node.js default-memory-mode budget in the table below; they differ +only in where unacknowledged data lives: - **`sf_dir` set** (store-and-forward): segments are memory-mapped files under `sf_dir`. Unacknowledged data survives sender restarts and is replayed by @@ -1147,7 +1151,7 @@ section of the connect string reference: | Key | Default | Description | |----------------------------------|-----------|-------------------------------------------| -| `reconnect_max_duration_millis` | `300000` | Budget for the blocking sync initial connect only; the running loop retries indefinitely. | +| `reconnect_max_duration_millis` | `300000` | Budget for the blocking sync initial connect only; the running loop retries indefinitely, except on a Node.js sender in default memory mode, without `sf_dir`, `initial_connect_retry=async`, or `lazy_connect=on` ([details](/docs/connect/clients/nodejs/#differences-from-other-clients)). | | `reconnect_initial_backoff_millis` | `100` | First post-failure sleep. | | `reconnect_max_backoff_millis` | `5000` | Cap on per-attempt sleep. | | `initial_connect_retry` | `off` | Retry on first connect (`on`, `sync`, `async`). | @@ -1158,8 +1162,15 @@ Key behaviors: every host's zone tier is equivalent and selection is based on health state only. The `zone=` connect-string key is accepted but silently ignored, so a connect string shared with egress clients works unchanged on ingress. -- **Authentication errors are terminal** at any host (`401`/`403`). The - reconnect loop does not continue past them. + The Node.js client is the exception: it applies `zone=` and `target=` to + ingress too, so `target=replica` in a shared connect string stops its + ingestion. +- **Authentication rejection (`401`/`403`) never moves to another host.** It + is terminal before a sender's first successful connection. After that, the + Java client retries it indefinitely, the Node.js client does so for senders + with `sf_dir` or in background memory mode (`initial_connect_retry=async` or + `lazy_connect=on`), and other clients stop; see + [Authentication is cluster-wide](/docs/high-availability/client-failover/concepts/#authentication-is-cluster-wide). - **`421 + X-QuestDB-Role`** is a role reject: transient if the role is `PRIMARY_CATCHUP`, topology-level otherwise. - **All other upgrade errors are transient** and feed into the reconnect loop, diff --git a/documentation/getting-started/quick-start.mdx b/documentation/getting-started/quick-start.mdx index 6a9bbee51..9608b63d3 100644 --- a/documentation/getting-started/quick-start.mdx +++ b/documentation/getting-started/quick-start.mdx @@ -269,7 +269,7 @@ Now... Time to really blast-off. 🚀 Next up: Bring your data - the _life blood_ of any database. -Choose from one of our premium ingest-only language clients: +Choose a first-party client library for ingestion and queries: diff --git a/documentation/high-availability/client-failover/concepts.md b/documentation/high-availability/client-failover/concepts.md index be95b64a9..93b154c19 100644 --- a/documentation/high-availability/client-failover/concepts.md +++ b/documentation/high-availability/client-failover/concepts.md @@ -57,7 +57,7 @@ host. | `Unknown` | The host has not been tried in this round, or its classification was reset. | | `TransientReject` | The server returned `421` with `X-QuestDB-Role: PRIMARY_CATCHUP` — it is a primary that is still catching up after promotion. Expected to recover. | | `TransportError` | TCP/TLS handshake failed, an HTTP upgrade returned a transient error code, or an established connection broke mid-stream. | -| `TopologyReject` | The server returned `421` with any role other than `PRIMARY_CATCHUP` (`PRIMARY`, `REPLICA`, `STANDALONE`, or an unrecognised token), or — on egress — a successfully-upgraded host whose `SERVER_INFO` role does not satisfy the requested `target=` filter. The host will not become usable without a topology change. | +| `TopologyReject` | The server returned `421` with any role other than `PRIMARY_CATCHUP` (`PRIMARY`, `REPLICA`, `STANDALONE`, or an unrecognised token), or a successfully-upgraded host whose `SERVER_INFO` role does not satisfy the requested `target=` filter (egress only; the Node.js client also applies it to ingress). The host will not become usable without a topology change. | A lower state in the table above is preferred when the client picks the next host to try. @@ -80,7 +80,10 @@ the host re-advertises a different zone. `target=primary` collapses every host's zone tier to `Same` — writers must follow the primary regardless of geography. Ingress is currently zone-blind in both storage modes, so the `zone=` key is silently accepted on ingress -connections and only takes effect on egress. +connections and only takes effect on egress. The Node.js client is the +exception: it applies `zone=` and `target=` to ingress too. Its other +deviations are listed under +[Differences from other clients](/docs/connect/clients/nodejs/#differences-from-other-clients). ### Selection priority @@ -120,7 +123,8 @@ The `target=` key controls which server role the client is willing to bind to: up to its predecessor's WAL — the client treats it as transient and retries the same host (with a fresh round, no exponential backoff) until it becomes a full `PRIMARY`. On an ingress sender this retry has no deadline; the producer -is bounded by buffer capacity rather than by elapsed time. +is bounded by buffer capacity rather than by elapsed time, except for the +Node.js sender described under [Ingress (writes)](#ingress-writes). A `421 Misdirected Request` response **without** an `X-QuestDB-Role` header is treated as a generic transport error, not a role reject — the client walks @@ -138,16 +142,23 @@ very different goals. The ingress reconnect loop sits inside the store-and-forward I/O thread. It runs continuously in the background, retrying through outages while the -producer keeps appending to the local buffer. There is no wall-clock give-up: -the loop retries an outage of any length, and what bounds your tolerance is -buffer capacity (`sf_max_total_bytes` and disk), not a timer. +producer keeps appending to the local buffer. There is no wall-clock give-up, +except in one Node.js mode described below: the loop retries an outage of any +length, and what bounds your tolerance is buffer capacity +(`sf_max_total_bytes` and disk), not a timer. - Initial backoff: `100 ms` - Maximum backoff: `5 s` - Per-outage budget: **none**. `reconnect_max_duration_millis` bounds only the blocking sync initial connect, and the running loop never consults it. + The exception is a Node.js sender in default memory mode, without `sf_dir`, + `initial_connect_retry=async`, or `lazy_connect=on`, which gives up after + `reconnect_max_duration_millis` and fails with + `QwpReconnectExhaustedError`; see the + [Node.js client](/docs/connect/clients/nodejs/#ingestion-reconnect). - Jitter: **equal-jitter** `[base, 2·base)` — non-zero lower bound damps - reconnect storms when many producers share a cluster + reconnect storms when many producers share a cluster. The Node.js client + uses full jitter, `[0, base)`, instead - Inter-host pause within a round: **none** — the client walks the full address list as fast as `auth_timeout_ms` allows, paying one backoff sleep at round exhaustion @@ -187,8 +198,8 @@ will not help. | Condition | Why terminal | |---|---| -| HTTP `401` / `403` on upgrade | Credentials are cluster-wide; retrying floods server logs without recovery. | -| Server-status reject (SF) | Application-layer reject; replay reproduces the same response. | +| HTTP `401` / `403` on upgrade | Credentials are assumed to be cluster-wide. Some clients retry after a sender's first successful connection; see [Authentication is cluster-wide](#authentication-is-cluster-wide). | +| Server rejection with a terminal policy: by default `SCHEMA_MISMATCH`, `PARSE_ERROR`, `SECURITY_ERROR`, and `PROTOCOL_VIOLATION`, or a batch that keeps being rejected | Replaying the same bytes reproduces the rejection. Retriable categories, such as `WRITE_ERROR`, are resent instead; see [Error frames](/docs/high-availability/store-and-forward/concepts/#error-frames). | ### Topology — handled inside the round @@ -197,7 +208,8 @@ within the same round. No exponential backoff is consumed. - `421` + `X-QuestDB-Role: PRIMARY_CATCHUP` → `TransientReject` - `421` + any other non-empty role, including unrecognised tokens → `TopologyReject` -- `SERVER_INFO.Role` does not match the requested `target=` (egress only) +- `SERVER_INFO.Role` does not match the requested `target=` (egress only; the + Node.js client also applies it to ingress) If every host in a round role-rejects, ingress pays one fixed backoff sleep (reset to `InitialBackoff`, no doubling) and starts a fresh round; egress @@ -213,7 +225,9 @@ and walks to the next host. When a round exhausts with transient errors, the client sleeps for the backoff interval and starts the next round. On the ingress sender the rounds -continue indefinitely; on the egress query client they are bounded by +continue indefinitely, apart from the Node.js exception described under +[Ingress (writes)](#ingress-writes); on the egress query client they are +bounded by `failover_max_attempts` and `failover_max_duration_ms`, which apply per `execute()`. @@ -232,14 +246,30 @@ when at least one peer is healthy." ## Authentication is cluster-wide -A `401` or `403` on the HTTP upgrade is terminal — the client does not retry -other hosts. The assumption is that auth credentials are configured -identically across the cluster, so a credential failure against one node is -a credential failure against all of them. Retrying would spam every peer's -audit log without recovering. - -If your deployment has per-host credentials, that is unsupported and outside -the failover model — split the workload into one connect string per credential. +A `401` or `403` on the HTTP upgrade does not send the client to another host: +credentials are assumed to be configured identically across the cluster, so +another node would repeat the rejection. Whether the rejection is terminal +depends on the client and on when it arrives: + +| Client | Before a sender's first successful connection | After it | +|---|---|---| +| Java | Terminal | Retried indefinitely | +| Node.js | Terminal | Retried indefinitely by senders with `sf_dir` or in background memory mode (`initial_connect_retry=async` or `lazy_connect=on`). Terminal for other senders | +| Rust, C, C++, Python, Go, .NET | Terminal | Terminal | + +Query connections and orphan drainers treat the rejection as terminal in every +client, except that a Java orphan drainer whose token comes from a token +provider retries for a bounded time before quarantining the slot. A sender +that retries keeps buffering until authentication succeeds again, bounded by +its buffer capacity, so a credential change on the cluster does not stop the +producer. Monitor such a sender: the Java client reports each rejection to the +sender's error handler as a retriable `SECURITY_ERROR`, and the Node.js client +emits an `attempt-failed` connection event for each failed attempt. See +[Node.js connection errors](/docs/connect/clients/nodejs/#connection-level-errors) +for the Node.js rules. + +Per-host credentials are outside the failover model. Use a separate connect +string for each credential. ## Next steps diff --git a/documentation/high-availability/client-failover/configuration.md b/documentation/high-availability/client-failover/configuration.md index 1e708fc69..51596d8cc 100644 --- a/documentation/high-availability/client-failover/configuration.md +++ b/documentation/high-availability/client-failover/configuration.md @@ -15,8 +15,11 @@ first. ## Common keys `addr` and `auth_timeout_ms` apply to every WS / WSS / HTTP / HTTPS client. -`zone` is accepted everywhere but only takes effect on egress; `target` is an -egress-only key and is rejected as an unknown key on an ingress connect string. +`zone` and `target` are accepted everywhere but only take effect on egress: +ingress parsers accept and ignore them, so one connect string can serve both +directions. The Node.js client is the exception: it applies both keys to +ingress too; its other deviations are listed under +[Differences from other clients](/docs/connect/clients/nodejs/#differences-from-other-clients). They are documented in full on the [connect-string reference](/docs/connect/clients/connect-string#failover-keys); the table below summarises the failover-relevant subset. @@ -24,9 +27,9 @@ the table below summarises the failover-relevant subset. | Key | Type | Default | Notes | |---|---|---|---| | `addr` | `host:port[,host:port…]` | required | Comma-separated peer list. The two syntactic forms (`addr=h1,h2` and repeated `addr=h1;addr=h2`) accumulate. Empty entries are rejected. | -| `zone` | string | unset | Client's zone identifier (opaque, case-insensitive — `eu-west-1a`, `dc-amsterdam`, etc.). Egress prefers same-zone peers when `target` is `any` or `replica`. Silently accepted but ignored on ingress. | -| `target` | `any` \| `primary` \| `replica` | `any` | **Egress only.** Which server role the query client accepts. Rejected as an unknown key on an ingress connect string. See [Role filter](/docs/high-availability/client-failover/concepts/#role-filter-target) for the role table. | -| `auth_timeout_ms` | int (ms) | `15000` | Upper bound on the HTTP-upgrade response read per host. Does **not** cover the TCP connect or TLS handshake — those use the OS default. Set lower if you have well-known network paths and want faster failover; set higher only if upgrade is genuinely slow. | +| `zone` | string | unset | Client's zone identifier (opaque, case-insensitive — `eu-west-1a`, `dc-amsterdam`, etc.). Egress prefers same-zone peers when `target` is `any` or `replica`. Silently accepted but ignored on ingress, except by the [Node.js client](/docs/connect/clients/nodejs/#multiple-endpoints), which also ranks ingress endpoints by zone. | +| `target` | `any` \| `primary` \| `replica` | `any` | **Egress only** (the Node.js client also applies it to ingress). Which server role the query client accepts. Other clients accept and ignore it on an ingress connect string. On the [Node.js client](/docs/connect/clients/nodejs/#multiple-endpoints), set the query-side role through the typed `egress` option instead. See [Role filter](/docs/high-availability/client-failover/concepts/#role-filter-target) for the role table. | +| `auth_timeout_ms` | int (ms) | `15000` | Upper bound on the HTTP-upgrade response read per host. Does **not** cover TCP connect or TLS handshake. `connect_timeout` bounds the TCP connect separately: most clients leave it unset by default and then use the OS timeout, while Node.js defaults it to 15 s and also bounds DNS and TLS with it. On Node.js, `auth_timeout_ms` defaults to `connect_timeout` when only that key is set. Lower `auth_timeout_ms` for faster upgrade failure detection; tune the connect timeout separately. | `addr` syntax — both of these are equivalent and produce the same three-peer list: @@ -47,9 +50,9 @@ for the full list. The failover-relevant keys are: | Key | Type | Default | Notes | |---|---|---|---| -| `reconnect_max_duration_millis` | int (ms) | `300000` (5 min) | Bounds the blocking sync initial connect only (`initial_connect_retry=on`/`sync`). A running sender's reconnect loop never consults it and retries indefinitely, so raising this does nothing for failover windows. | +| `reconnect_max_duration_millis` | int (ms) | `300000` (5 min) | Bounds the blocking sync initial connect only (`initial_connect_retry=on`/`sync`). A running sender's reconnect loop never consults it and retries indefinitely, so raising this does nothing for failover windows. Exception: a Node.js sender in default memory mode, without `sf_dir`, `initial_connect_retry=async`, or `lazy_connect=on`, applies it to every outage; see [the Node.js client](/docs/connect/clients/nodejs/#ingestion-reconnect). | | `reconnect_initial_backoff_millis` | int (ms) | `100` | Starting backoff sleep at round exhaustion. Doubles up to `reconnect_max_backoff_millis`. | -| `reconnect_max_backoff_millis` | int (ms) | `5000` | Cap on the exponential backoff. With equal-jitter, the actual sleep lands in `[max, 2·max)` once the base saturates. | +| `reconnect_max_backoff_millis` | int (ms) | `5000` | Cap on the exponential backoff. With equal-jitter, the actual sleep lands in `[max, 2·max)` once the base saturates. The Node.js client uses full jitter, so its sleep lands in `[0, max)`. | | `initial_connect_retry` | `off` \| `on` \| `async` | `off` | Whether to apply the same retry loop to the very first connect attempt. See below. | ### `initial_connect_retry` @@ -61,8 +64,8 @@ network), and retrying for five minutes only hides it. | Value | Behaviour | |---|---| | `off` (default; alias `false`) | First-connect failure is terminal. The producer's call to build the sender throws immediately. | -| `on` (aliases `sync`, `true`) | First-connect failures are retried on the caller's thread. The constructor blocks until it connects or `reconnect_max_duration_millis` expires — this is the **only** place that key applies. Once the sender is running, reconnection is unbounded. | -| `async` | The constructor returns immediately; the background I/O thread drives the reconnect loop. The producer experiences backpressure if it tries to publish before the connection comes up. Intended for unattended producers where the SF directory may already carry segments from a prior process and the server may come up later. | +| `on` (aliases `sync`, `true`) | First-connect failures are retried on the caller's thread. The constructor blocks until it connects or `reconnect_max_duration_millis` expires — this is the **only** place that key applies. Once the sender is running, reconnection is unbounded, except for the Node.js senders described in the `reconnect_max_duration_millis` row above. | +| `async` | The constructor returns immediately; the background I/O thread drives the reconnect loop. The producer experiences backpressure if it tries to publish before the connection comes up. Intended for unattended producers where the SF directory may already carry segments from a prior process and the server may come up later. On Node.js, `connectQwpNodeClient()` also opens a query connection at startup, so it returns while the server is down only with `query_pool_min=0`, which `lazy_connect=on` sets; see [startup and outage modes](/docs/connect/clients/nodejs/#ingestion-modes). | ## Egress (query) diff --git a/documentation/high-availability/store-and-forward/concepts.md b/documentation/high-availability/store-and-forward/concepts.md index 6acafca19..e98647f69 100644 --- a/documentation/high-availability/store-and-forward/concepts.md +++ b/documentation/high-availability/store-and-forward/concepts.md @@ -19,7 +19,9 @@ arrive asynchronously. A network outage or a server restart leaves your producer code unaffected — the I/O thread quietly reconnects and replays what remains. In SF mode, even a crash of the sender process itself loses no unacked data: the next sender on the slot recovers it from disk and -replays it. +replays it. The one exception is a Node.js sender in default memory mode, +whose `flush()` waits for the reconnect during an outage; see +[Reconnect and replay](#reconnect-and-replay). ## Two modes @@ -33,12 +35,18 @@ SF runs in either of two modes selected by the connect string: | Unacked data if the sender crashes | Lost | Recovered and replayed on restart | | Unacked data if the sender's host reboots | Lost | Recovered, if the disk persists | | Tolerates transient network blips | Yes | Yes | -| Tolerates multi-minute server outages | Bounded by RAM cap | Bounded by disk cap | +| Tolerates multi-minute server outages | Bounded by RAM cap (Node.js default memory mode: also by `reconnect_max_duration_millis`) | Bounded by disk cap | | Recovers another sender's stale slot | n/a | Opt-in via `drain_orphans=on` | Both modes share the same reconnect loop, the same backoff and retry budgets, and the same on-the-wire behaviour. The only difference is -where unacked data lives. +where unacked data lives. The Node.js client is the exception: it splits +memory mode in two. A sender in default memory mode gives up after +`reconnect_max_duration_millis` (5 minutes by default), while a sender in +background memory mode (`initial_connect_retry=async` or `lazy_connect=on`) +retries indefinitely, as SF mode does. See the +[Node.js ingestion modes](/docs/connect/clients/nodejs/#ingestion-modes) and +[Differences from other clients](/docs/connect/clients/nodejs/#differences-from-other-clients). ## What "frame" means here @@ -62,8 +70,9 @@ Two distinct counters track frame identity: - **FSN** (frame-sequence-number) — a monotonic counter assigned when a frame is appended to the substrate. FSN survives reconnects and (in SF mode) restarts. It is the substrate's permanent identifier for a frame. -- **wireSeq** — the per-connection counter the server uses for - deduplication, reset to `0` on every successful WebSocket upgrade. +- **wireSeq** is the per-connection counter used to correlate acknowledgements + with sent frames. It resets to `0` on every successful WebSocket upgrade + and is not a row-deduplication key. On every (re)connect the relationship is pinned: @@ -82,9 +91,12 @@ Two consequences: - Frames **must** be sent in strict order. The wire format does not serialise `wireSeq` — the server assigns it implicitly from receive order. Reordering breaks the FSN mapping. -- After a reconnect, the server sees the **same payloads** at new - `wireSeq` values. Server-side dedup keys off `messageSequence` inside - the payload, not `wireSeq`, so replay does not produce double-writes. +- After a reconnect, the server may see the **same payloads** at new + `wireSeq` values. An acknowledgement can be lost after a batch was + committed, so replay is **at least once** and can insert duplicate rows. + `wireSeq` is transport bookkeeping, not a deduplication key. For + idempotent ingestion, use stable source IDs and timestamps with table-level + `DEDUP UPSERT KEYS`; see [Deduplication](/docs/concepts/deduplication/). ## Trim: how unacked data is reclaimed @@ -123,10 +135,18 @@ object store** (S3, Azure Blob, GCS, or NFS). watermarks. The client matches the head of the OK queue against these watermarks; each fully-covered head entry pops, and `ackedFsn` advances to the highest covered wireSeq. -- The client opt-in is mandatory — the connect fails loudly if the server - does not echo `X-QWP-Durable-Ack: enabled` on the upgrade response. - This avoids the silent failure mode where the producer waits forever - for ack frames that will never arrive. +- The client requires an `X-QWP-Durable-Ack: enabled` echo on the upgrade + response and rejects a connection without it, rather than waiting for ack + frames it cannot receive. In most clients the rejection is terminal. The + Java client retries it after a sender's first successful connection, so a + capability change on the cluster cannot stop the producer. Node.js senders + with a background start (`initial_connect_retry=async` or + `lazy_connect=on`) retry from startup and emit `durable-ack-unavailable`, + with or without `sf_dir`. A Node.js sender with `sf_dir` and a foreground + start fails on the first connection but retries after a successful + connection; see + [Node.js durable acknowledgement](/docs/connect/clients/nodejs/#durable-acknowledgement). + A retrying sender keeps buffering, so monitor it. Durable-ack mode is the right choice when "data is in the object store" is the durability bar, but it has two costs: a longer time-to-trim (so @@ -143,7 +163,13 @@ When the wire connection breaks — for any reason — the I/O thread enters the reconnect loop documented in [Client failover concepts](/docs/high-availability/client-failover/concepts/). The producer is **not notified**: it keeps publishing into the substrate, -bounded by `sf_max_total_bytes` (see backpressure below). +subject to available capacity (see [Backpressure](#backpressure)). + +On Node.js, only a sender in default memory mode waits for the reconnect in +`flush()`, up to `reconnect_max_duration_millis`. Background memory mode, +enabled by `initial_connect_retry=async` or `lazy_connect=on`, keeps accepting +batches into the memory replay queue until capacity is exhausted and retries +indefinitely. See the [three Node.js ingestion modes](/docs/connect/clients/nodejs/#ingestion-modes). On every successful (re)connect: @@ -151,8 +177,11 @@ On every successful (re)connect: 2. `wireSeq` resets to `0`. 3. The read cursor rewinds to the first un-acked frame on disk (or in memory). -4. Frames stream to the wire in FSN order. The server's dedup window - absorbs any frames that landed before the disconnect. +4. Frames stream to the wire in FSN order. A frame committed before the + disconnect but not acknowledged can be inserted again: replay is at least + once. Prevent duplicate rows with table-level `DEDUP UPSERT KEYS` covering + the designated timestamp and stable source identity, preserving both on + retries. See [Deduplication](/docs/concepts/deduplication/). 5. New frames appended by the producer during replay are picked up automatically — the I/O loop watches a volatile `publishedFsn` cursor. @@ -162,12 +191,18 @@ in the `getTotalFramesReplayed` observability counter. ## Backpressure -The substrate enforces `sf_max_total_bytes` as a hard cap on resident -storage. When the cap is hit, the producer's `appendBlocking` call +The substrate uses `sf_max_total_bytes` to apply backpressure on resident +storage. When capacity is exhausted, the producer's `appendBlocking` call busy-spins (with cooperative yield) up to `sf_append_deadline_millis` waiting for ACK-driven trim to free space. If the deadline fires, the call throws a typed exception. +On Node.js with `sf_dir`, this value is a journal size target rather than a +hard disk limit. Transaction-closing batches and retained symbol dictionaries +can exceed it, and other metadata needs additional space. Provision headroom +for each sender; see [Node.js journal capacity](/docs/connect/clients/nodejs/#sf-capacity). +Without `sf_dir`, the key caps the in-memory replay queue. + The exception message distinguishes the two scenarios: - **Backpressure while the wire is publishing** — the server is acking @@ -184,13 +219,19 @@ The exception message distinguishes the two scenarios: `close()` waits up to `close_flush_timeout_millis` for `ackedFsn` to reach `publishedFsn` — i.e. for the server to acknowledge everything the producer has handed in. The default differs by client: 60 s on Java and .NET, 5 s on Rust, -C, C++ and Python. If the wait succeeds, all data is acked. If the timeout -fires, a `WARN` is logged and: +C, C++, Python, Go and Node.js. If the wait succeeds, all data is acked. If the +timeout fires, a `WARN` is logged and: - in **SF mode**, the un-acked tail is left on disk and recovered by the next sender on the same slot; - in **memory mode**, the un-acked tail is lost. +On the Node.js client, a standalone sender's `close()` rejects with +`QwpSenderCloseTimeoutError` instead of logging. The pooled client's +`db.close()` resolves, and usually reports the timeout to +`ingressSession.onError` as a non-terminal `QwpIngressAckTimeoutError`; that +report is best-effort. + Setting `close_flush_timeout_millis=0` (or `-1`) skips the drain wait entirely — useful for fast shutdown paths where you do not want to block. Even in this branch, the slot lock is released and segments are unmapped @@ -297,27 +338,55 @@ until an operator intervenes. The orphan flow is opt-in because in a multi-tenant deployment with shared `sf_dir`, blindly draining unknown slots may be surprising. +:::caution Node.js client + +A Node.js drainer adopts a slot only if it can take the slot's `.lock.owner` +lock. It can reclaim a crashed owner's lock only on the same host, and only +when the recorded process ID is no longer in use, so it skips a slot whose +owner ran in a replaced container or in a container restarted in place. Those +rows stay on disk until the stale lock is removed; see +[Node.js lock recovery](/docs/high-availability/store-and-forward/operating-and-tuning/#nodejs-lock-recovery). +A pooled Node.js client also replays slots of its own `sender_id` without +`drain_orphans=on`; the key adds the other `sender_id`s. + +::: + ## Error frames -Not every server response is an OK. Server errors fall into six -categories, each with a default policy: +Not every server response is an OK. A rejected batch is **not** silently +dropped and trimmed: the client either retries it or reports a terminal error. +There is no drop policy. The table lists the built-in default policy of each +category in the Java reference client and the Node.js client; see the +[connect-string error policies](/docs/connect/clients/connect-string/#error-handling) +for the shared vocabulary and which clients let you override the defaults. -| Category | Default | Meaning | +| Category | Default policy | Meaning | |---|---|---| -| `SCHEMA_MISMATCH` | `DROP_AND_CONTINUE` | The batch's schema doesn't match the server. Replay won't help — the substrate logs and advances trim past the rejected span. | -| `WRITE_ERROR` | `DROP_AND_CONTINUE` | Per-batch write failure (e.g. table is not currently accepting writes). | -| `PARSE_ERROR` | `HALT` | Almost certainly a client bug. The substrate preserves on-disk frames for postmortem. | -| `INTERNAL_ERROR` | `HALT` | Catch-all server fault. | -| `SECURITY_ERROR` | `HALT` | Cluster-wide auth / authorization failure. | -| `PROTOCOL_VIOLATION` | `HALT` (forced) | Connection is gone after a terminal WebSocket close code; no choice. | - -Errors are also delivered to an **error inbox** — a bounded queue -consumed by a daemon dispatcher that invokes your registered handler. -Overflow drops the oldest entry rather than the newest (watermarks are -monotonic; the latest entry is the most informative). The default -handler logs every received error: silence is forbidden by the contract, -because a buggy or no-op handler would hide data loss -indistinguishably from a healthy connection. +| `SCHEMA_MISMATCH` | `terminal` | The schema does not match. The sender stops; in SF mode, the rejected bytes remain in the journal for inspection. | +| `PARSE_ERROR` | `terminal` | Malformed payload; replaying identical bytes cannot help. | +| `SECURITY_ERROR` | `terminal` | Authorization failed, for example an ACL denial on a writable node. | +| `PROTOCOL_VIOLATION` | `terminal` (forced) | Protocol failure; stop and report it. | +| `WRITE_ERROR` | `retriable` | A write failed, for example under temporary storage pressure; reconnect and replay. | +| `INTERNAL_ERROR` | `retriable` | Retry after an unexpected server-side failure. | +| `DICTIONARY_GAP` | `retriable` | The connection is missing symbol dictionary entries; resend them and replay. | +| `NOT_WRITABLE` | `retriable_other` | The node cannot accept writes, for example a replica; replay on another endpoint. Reserved: current servers close the connection instead, and the client reconnects. | +| `UNKNOWN` | `retriable` (forced) | A status the client does not know, for example from a newer server; retry rather than stop. | + +A batch that keeps being rejected without progress escalates to a terminal +error through the poison-frame detector (`max_frame_rejections`). The Java and +Node.js clients also report a client-side `DATA_LOSS` category, with the +`abandoned` policy, when they set aside a corrupt store-and-forward journal. +The Node.js client also defines `cancelled` and `limit-exceeded` categories, +both retriable; current servers send those statuses only to query +connections. + +Rejections are delivered asynchronously through a bounded error inbox +(`error_inbox_capacity`, default `256`) that drops the oldest notification on +overflow, to the application's error handler, such as `onSenderError` on +Node.js. The default handler logs every rejection, because a silent handler +would hide data loss. The Node.js client reports categories and policies as +lowercase, hyphenated strings, such as `schema-mismatch` and +`retriable-other`, and accepts the `on_*_error` keys without applying them. ## Next steps diff --git a/documentation/high-availability/store-and-forward/configuration.md b/documentation/high-availability/store-and-forward/configuration.md index 6d11a1330..8c58e865b 100644 --- a/documentation/high-availability/store-and-forward/configuration.md +++ b/documentation/high-availability/store-and-forward/configuration.md @@ -27,8 +27,8 @@ mode. | `sf_dir` | path | unset | Group root directory. When set, the slot lives at `//` and unacked data is durable across process restarts. When unset, the substrate runs in memory mode. | | `sender_id` | string | `default` | Slot subdirectory name. Two senders sharing the same `sender_id` and `sf_dir` will collide on the slot lock. Must not contain path separators or be empty. | | `sf_max_segment_bytes` | size | `4M` | Per-segment file size; rotation threshold. | -| `sf_max_total_bytes` | size | `128M` (memory) / `10G` (SF) | Hard cap on resident SF storage. Triggers producer backpressure when full. | -| `sf_durability` | enum | `memory` | `memory` (page-cache durable) and `periodic` (background checkpoint to stable storage) both ship. `periodic` requires `sf_dir`. `flush` and `append` parse but are rejected at build time. The .NET client accepts `memory` only. | +| `sf_max_total_bytes` | size | `128M` (memory) / `10G` (SF) | Capacity for producer backpressure. Node.js disk journals can exceed this target for transaction completion and symbol dictionaries; provision [additional disk headroom](/docs/connect/clients/nodejs/#sf-capacity). Without `sf_dir`, this caps the memory replay queue. | +| `sf_durability` | enum | `memory` | `memory` relies on the page cache; `periodic` checkpoints in the background and requires `sf_dir`. Node.js also supports `append`, which makes each journal append durable before `flush()` resolves. Go and .NET accept only `memory`; other clients reject `append` at build time. `flush` is not supported. | | `sf_sync_interval_millis` | int (ms) | `5000` | Checkpoint cadence for `sf_durability=periodic`; rejected without it. A floor, not a guarantee: scheduler and storage latency add to it. | | `sf_append_deadline_millis` | int (ms) | `30000` | How long a producer `appendBlocking` call waits for ACK-driven trim to free space before throwing. | | `drain_orphans` | bool | `off` | Scan `/*` at startup and spawn drainers for sibling slots that contain unacked data. See [orphan adoption](/docs/high-availability/store-and-forward/concepts/#orphan-adoption). | @@ -48,11 +48,11 @@ and host-walk semantics are documented in | Key | Type | Default | Description | |---|---|---|---| -| `reconnect_max_duration_millis` | int (ms) | `300000` (5 min) | Bounds the blocking sync initial connect only (`initial_connect_retry=on`/`sync`). A running sender's reconnect loop never consults it and retries indefinitely. | +| `reconnect_max_duration_millis` | int (ms) | `300000` (5 min) | Bounds the blocking sync initial connect only (`initial_connect_retry=on`/`sync`). A running sender's reconnect loop never consults it and retries indefinitely. Exception: a Node.js sender in default memory mode, without `sf_dir`, `initial_connect_retry=async`, or `lazy_connect=on`, applies it to every outage; see [the Node.js client](/docs/connect/clients/nodejs/#ingestion-reconnect). | | `reconnect_initial_backoff_millis` | int (ms) | `100` | Initial backoff sleep at round exhaustion. | -| `reconnect_max_backoff_millis` | int (ms) | `5000` | Cap on the exponential backoff. With equal-jitter the actual sleep lands in `[max, 2·max)`. | +| `reconnect_max_backoff_millis` | int (ms) | `5000` | Cap on the exponential backoff. With equal-jitter the actual sleep lands in `[max, 2·max)`; the Node.js client uses full jitter, `[0, max)`. | | `initial_connect_retry` | enum | `off` | `off` (alias `false`): first-connect failure is terminal. `on` (aliases `sync`, `true`): same retry loop as reconnect, blocking the constructor. `async`: same retry loop in the I/O thread, non-blocking. | -| `close_flush_timeout_millis` | int (ms) | `60000` on Java and .NET; `5000` on Rust, C, C++ and Python | `close()` blocks up to this long waiting for `ackedFsn ≥ publishedFsn`. `0` or `-1` skips the drain wait. The safety-net `checkError()` still runs. | +| `close_flush_timeout_millis` | int (ms) | `60000` on Java and .NET; `5000` on Rust, C, C++, Python, Go and Node.js | `close()` blocks up to this long waiting for `ackedFsn ≥ publishedFsn`. `0` or `-1` skips the drain wait. The safety-net `checkError()` still runs. | Cross-reference: [connect-string #reconnect-keys](/docs/connect/clients/connect-string#reconnect-keys). @@ -64,19 +64,22 @@ Opt in to object-store-durable trim. See | Key | Type | Default | Description | |---|---|---|---| -| `request_durable_ack` | bool | `off` | Opt-in via the upgrade header `X-QWP-Request-Durable-Ack: true`. Trim is then driven by `STATUS_DURABLE_ACK` frames only; OK frames no longer advance the trim watermark. Connect fails loudly if the server does not echo `X-QWP-Durable-Ack: enabled`. WebSocket transports only. | -| `durable_ack_keepalive_interval_millis` | int (ms) | `200` | Cadence of WebSocket PING the I/O loop sends while there are pending durable confirmations and the producer is idle. `0` or negative disables. | +| `request_durable_ack` | bool | `off` | Opt-in via the upgrade header `X-QWP-Request-Durable-Ack: true`. Trim is then driven by `STATUS_DURABLE_ACK` frames only; OK frames no longer advance the trim watermark. A missing `X-QWP-Durable-Ack: enabled` echo is terminal in most clients; Java and some Node.js senders keep retrying (see [Concepts](/docs/high-availability/store-and-forward/concepts/#trim-how-unacked-data-is-reclaimed)). WebSocket transports only. | +| `durable_ack_keepalive_interval_millis` | int (ms) | `200` | Cadence of WebSocket PING while durable confirmations are pending and the producer is idle. `0` disables the PING; some clients also accept negative values. Node.js rejects negative values, and explicitly setting even `0` requests durable ACK, so an unsupported server may reject the connection. See [Node.js durable acknowledgement](/docs/connect/clients/nodejs/#durable-acknowledgement). | ## Error-handling keys | Key | Type | Default | Description | |---|---|---|---| | `error_inbox_capacity` | int (≥16) | `256` | Bounded SPSC queue capacity for async error notifications. Overflow drops the oldest entry and increments `getDroppedErrorNotifications`. | -| `on_server_error`, `on_schema_error`, `on_parse_error`, `on_internal_error`, `on_security_error`, `on_write_error` | enum | per category | Override the default policy (`HALT` or `DROP_AND_CONTINUE`) for a category. Reserved in the spec but not yet recognised by the connect-string parser. | +| `on_server_error`, `on_schema_error`, `on_parse_error`, `on_internal_error`, `on_security_error`, `on_write_error` | enum | per category | All clients accept these keys. Go and .NET apply them; Java, Node.js, Rust, C, C++, and Python currently ignore them. The two that apply them accept different values: .NET takes only `halt`/`terminal` and `retry`/`retriable` and rejects `auto` and `retriable_other`. There is no `DROP_AND_CONTINUE` policy. See [Error handling](/docs/connect/clients/connect-string/#error-handling). | The per-category defaults are documented in [Concepts § Error frames](/docs/high-availability/store-and-forward/concepts/#error-frames). -`PROTOCOL_VIOLATION` and `UNKNOWN` are forced `HALT` and not user-overridable. +`PROTOCOL_VIOLATION` is always terminal and `UNKNOWN` defaults to retriable, +so a status from a newer server leads to a retry rather than a silently +dropped batch. The `on_*_error` keys cannot change either; only the .NET +client's programmatic resolver can change `UNKNOWN`. ## Other relevant keys @@ -90,14 +93,14 @@ canonical entries. | `username` / `password` | string | unset | HTTP Basic auth on the upgrade request. | | `token` | string | unset | Bearer token on the upgrade request. | | `tls_verify` | enum | `on` | `on` or `unsafe_off`. Applies to `wss::` / TLS connections. | -| `tls_roots` | path | system trust | Custom CA trust store. | +| `tls_roots` | path | system trust (Node.js: bundled CAs) | Custom CA trust store. | | `tls_roots_password` | string | unset | Trust store password. | | `auto_flush` | bool | `on` | Global on/off for auto-flush triggers. | -| `auto_flush_rows` | int / `off` | `1000` | Row-count flush trigger. | +| `auto_flush_rows` | int / `off` | `1000` | Row-count flush trigger. Node.js: use `0` to disable; `off` is rejected. | | `auto_flush_bytes` | int / `off` | `0` (off) | Byte-size flush trigger. | -| `auto_flush_interval` | int (ms) / `off` | `100` | Time-since-first-row flush trigger. | -| `init_buf_size` | size | `64K` | Initial encode buffer capacity. | -| `max_buf_size` | size | `100M` | Max encode buffer capacity. | +| `auto_flush_interval` | int (ms) / `off` | `100` | Time-since-first-row flush trigger (Node.js: since last flush or sender creation). Node.js: use `0` to disable; `off` is rejected. | +| `init_buf_size` | size | `64K` | Initial encode buffer capacity; not supported by the Node.js QWP `ws`/`wss` client. | +| `max_buf_size` | size | `100M` | Max encode buffer capacity; not supported by the Node.js QWP `ws`/`wss` client. | | `max_name_len` | int | `127` | Local validation cap for table / column names. | ## Validation @@ -106,8 +109,9 @@ The parser rejects: - Unknown keys (forward compatibility is via the spec, not silent acceptance). -- `sf_durability` values other than `memory`, `flush`, `append`. `flush` - and `append` parse but are rejected at build time today. +- Unsupported `sf_durability` values. Go and .NET accept only `memory`; + Node.js also accepts `periodic` and `append`. Other clients accept + `periodic`, but reject `flush` and `append` at build time. - `sender_id` containing path separators or empty. - `request_durable_ack=on` on non-WebSocket transports. diff --git a/documentation/high-availability/store-and-forward/operating-and-tuning.md b/documentation/high-availability/store-and-forward/operating-and-tuning.md index 9b69e434c..d4a08b3b2 100644 --- a/documentation/high-availability/store-and-forward/operating-and-tuning.md +++ b/documentation/high-availability/store-and-forward/operating-and-tuning.md @@ -20,8 +20,9 @@ In SF mode every sender owns one **slot directory**: ``` // -├── .lock # advisory exclusive lock (kernel-released on process exit) +├── .lock # OS lock; Node.js retains it for compatibility only ├── .lock.pid # UTF-8 text: holder PID + '\n' (diagnostic only) +├── .lock.owner/ # Node.js ownership directory; may survive a crash ├── .failed # optional drainer-failure sentinel (UTF-8 reason text) ├── .ack-watermark # optional 16-byte durable-ack high-water mark ├── sf-0000000000000001.sfa @@ -36,10 +37,10 @@ the host. ### `.lock` and `.lock.pid` -The `.lock` file is held under an advisory exclusive lock for the engine's -lifetime — POSIX clients use `flock` / `fcntl`, Windows uses -`LockFileEx`. The lock is released automatically when the file descriptor -closes, including on hard process exit (kernel cleanup). +For clients other than Node.js, the `.lock` file is held under an advisory +exclusive lock for the engine's lifetime — POSIX clients use `flock` / +`fcntl`, Windows uses `LockFileEx`. Their lock is released automatically when +the file descriptor closes, including on hard process exit (kernel cleanup). A second sender pointing at the same slot directory will fail to start with an error that names the holder's PID, read from `.lock.pid`. The @@ -53,6 +54,25 @@ files are harmless — the next acquirer silently overwrites them. **not** share a slot on a network filesystem. Their lock primitives are incompatible. +:::caution Node.js client + +The Node.js client does not use an OS lock. It locks a slot with a +`.lock.owner` directory that records the owner's host name and process ID, and +keeps `.lock` and `.lock.pid` only for compatibility. After a crash, a new +Node.js sender takes the slot over automatically only on the same host, once +the recorded process ID is no longer in use. Containers usually defeat that +check: a replacement container has a new host name, and a container restarted +in place typically gives the restarted process its previous process ID, which +is often 1. The new sender then reports `QwpReplayStoreLockedError` on every +start. +[Node.js lock recovery](#nodejs-lock-recovery) describes how to remove a stale +lock safely. + +Node.js and other clients do not see each other's locks, so never let them use +the same `sf_dir` at the same time. + +::: + ### `.failed` Present iff a previous drainer attempt gave up on the slot — reconnect @@ -89,8 +109,11 @@ and the second start fails loudly. A common cause is a redeploy where the old process hasn't fully exited when the new one comes up. Solutions: -- Wait for the old process to release the lock (the kernel releases on - exit; `kill -9` is sufficient). +- Stop the previous process. For clients using OS locks, the kernel releases + the lock on exit (even after `kill -9`). A killed Node.js sender can leave + a stale `.lock.owner` directory behind: verify that the old owner is gone + before removing it, as described under + [Node.js lock recovery](#nodejs-lock-recovery). - Use a deployment unit that orders shutdown before startup. - For containerised deployments, set `sender_id` from a per-pod stable identity so two pods with the same template name don't collide. @@ -98,6 +121,38 @@ when the new one comes up. Solutions: `drain_orphans=on` does **not** override the lock — a busy orphan slot is skipped, not stolen. +### Node.js lock recovery {#nodejs-lock-recovery} + +A Node.js sender can leave its `.lock.owner` directory behind when it crashes. +On the same host, the next sender reclaims the lock automatically once the +recorded process ID is no longer in use. It cannot reclaim a lock recorded on +another host, such as a container replaced under a new host name, or one whose +process ID is in use again, including by the restarted process itself in a +container restarted in place; a new sender on the slot then fails with +`QwpReplayStoreLockedError`. To recover: + +1. Verify that the previous owner has exited and that no process is using the + slot. The `.lock.owner` directory records the owner's host name and process + ID. +2. Remove the stale `//.lock.owner` directory, where `` is + ``, or `-` for a pooled sender, and start the + client again. +3. If startup still reports `QwpReplayStoreLockedError`, also inspect + `/.slot-locks/.lock.owner`. This short-lived guard can + survive a crash during lock acquisition or quarantine. Remove that specific + owner directory only after verifying that its owner has exited. + +Never delete the shared `.slot-locks` directory or another slot's locks. + +Automate this cleanup only where the deployment guarantees that the previous +owner has exited before a new one starts, for example a single replica that +uses the Kubernetes `Recreate` update strategy and a `ReadWriteOnce` volume, +or a container restarted in place whose volume no other process uses. A +startup step can then remove the stale owner directories of the client's own +slots before it creates the client. Anywhere two processes can overlap, +recover manually. See also the +[Node.js client](/docs/connect/clients/nodejs/#sf-lock-recovery). + ## Sizing capacity Two limits matter: @@ -116,10 +171,22 @@ disk usage staying high under slow ack cadence. ### `sf_max_total_bytes` — slot capacity (default `128 MiB` memory / `10 GiB` SF) -This is the **hard cap** on resident SF storage — sealed segments plus -the active segment. When this fills, producer `appendBlocking` calls -block (with cooperative yield) for up to `sf_append_deadline_millis` -waiting for ACK-driven trim to free space; on timeout the call throws. +This controls resident SF storage: sealed segments plus the active segment. +When capacity is exhausted, producer `appendBlocking` calls block (with +cooperative yield) for up to `sf_append_deadline_millis` waiting for ACK-driven +trim to free space; on timeout the call throws. + +:::caution Node.js disk-journal capacity + +With `sf_dir`, Node.js treats `sf_max_total_bytes` as a target, not a hard disk +limit. Transaction-closing batches can reserve extra segments to make the +commit possible when the journal is full. Segment reservations can reach +roughly twice the target, depending on segment rounding, with retained symbol +dictionaries and other metadata requiring additional space. Provision +headroom per sender and monitor actual disk usage; do not use the target as a +filesystem quota. See [Node.js journal capacity](/docs/connect/clients/nodejs/#sf-capacity). + +::: Size this against your **worst expected outage** times your ingest rate: @@ -158,7 +225,8 @@ records every producer thread that hit the cap. When an SF-mode sender opens, it runs this sequence: -1. Acquire `//.lock`. Fail loudly on contention. +1. Acquire the slot lock, `//.lock` (the Node.js client + uses a `.lock.owner` directory instead). Fail loudly on contention. 2. Scan every `*.sfa` file: - Validate magic, version, header. - Walk frames forward verifying each CRC32C-Castagnoli. @@ -182,9 +250,9 @@ fresh start: no segments, no replay. | Symptom | Likely cause | Operator action | |---|---|---| -| "Slot held by PID ``" | Two processes claiming the same `sender_id`. | Stop the duplicate. The lock releases on its exit. | +| "Slot held by PID ``" or `QwpReplayStoreLockedError` (Node.js) | Another process holds the slot, or a Node.js `.lock.owner` is stale after a crash. | Stop the duplicate. OS locks release on exit; for Node.js verify the owner is gone before removing `.lock.owner` (see [Node.js lock recovery](#nodejs-lock-recovery)). | | "Gap between segments" | Corruption — a segment was deleted out of band. | Restore from backup or accept data loss; the substrate refuses to start. | -| "Watermark exceeds publishedFsn" | `.ack-watermark` is corrupt; the engine falls back to the no-watermark seed. | Logged as `WARN`. Replay will re-send the lowest segment's frames; rely on server deduplication. | +| "Watermark exceeds publishedFsn" | `.ack-watermark` is corrupt; the engine falls back to the no-watermark seed. | Logged as `WARN`. Replay will re-send the lowest segment's frames, which inserts duplicate rows unless the table has `DEDUP UPSERT KEYS`. | | Torn tail count > 0 | The previous process crashed mid-frame-write. | Informational; the CRC + zero-fill design discards the partial frame. | ## Close and shutdown @@ -193,10 +261,13 @@ fresh start: no segments, no replay. | Value | Behaviour | |---|---| -| `5000` (default) | Block up to 5 s waiting for `ackedFsn ≥ publishedFsn`. Log `WARN` on timeout; un-acked tail stays on disk (SF) or is lost (memory). | +| client default: `60000` on Java and .NET, `5000` on Rust, C, C++, Python, Go, and Node.js | Block up to that long waiting for `ackedFsn ≥ publishedFsn`. Log `WARN` on timeout; un-acked tail stays on disk (SF) or is lost (memory). | | `0` or `-1` | Skip the drain wait. Pending data persists on disk (SF) for the next sender, or is lost (memory). | | any other positive value | That timeout in milliseconds. | +On the Node.js client, a standalone sender's `close()` rejects with +`QwpSenderCloseTimeoutError` on timeout instead of logging a `WARN`. + In every branch `close()`: - Performs a non-blocking safety-net check that rethrows any latched @@ -275,8 +346,9 @@ A conformant client exposes at minimum: by category. Background `SCHEMA_MISMATCH` is usually a schema-drift symptom worth alerting on. -The default error handler logs every received `SenderError` — -`ERROR`-level for HALT, `WARN`-level for DROP. Replace it only if you +The default error handler logs every received `SenderError`: +`ERROR`-level for terminal and abandoned errors, `WARN`-level for retriable +ones. Replace it only if you are also routing the errors somewhere else (Sentry, structured logs): silence is forbidden by the contract. @@ -289,7 +361,9 @@ When several senders share a host and a `sf_dir`: sender. - Consider `drain_orphans=on` if dynamic sender identities mean dead instances can leave permanent orphans. -- Size `sf_max_total_bytes × number_of_senders` against available disk. +- Size `sf_max_total_bytes × number_of_senders` against available disk. For + Node.js, also reserve each sender's transaction and metadata headroom (see + [Sizing capacity](#sizing-capacity)). - Plan for the worst-case lock-collision recovery: a misconfigured fleet that all share `sender_id=default` will leave only one sender alive on each host. That is the design — fail loudly rather than diff --git a/documentation/high-availability/store-and-forward/when-to-use.md b/documentation/high-availability/store-and-forward/when-to-use.md index 8897ed6a9..fa9a40ce3 100644 --- a/documentation/high-availability/store-and-forward/when-to-use.md +++ b/documentation/high-availability/store-and-forward/when-to-use.md @@ -57,7 +57,10 @@ Unacked frames are written to mmap'd files under Both modes share the same wire behaviour, the same failover loop, and the same connect-string keys for everything other than storage. You can switch between them without changing application code — only the connect -string. +string. On the Node.js client, a sender in default memory mode, without +`sf_dir`, `initial_connect_retry=async`, or `lazy_connect=on`, also gives up +after `reconnect_max_duration_millis` of outage; see +[Differences from other clients](/docs/connect/clients/nodejs/#differences-from-other-clients). ## Comparison at a glance @@ -100,17 +103,26 @@ GCS, or NFS). - WAL-local durability on the primary is sufficient. - You want minimum steady-state disk usage. - You are running OSS or a build that does not support durable-ack. - (The handshake fails loudly if you opt in but the server cannot - deliver — see below.) + Opting in makes those connection attempts fail; see [Caveats](#caveats) for + the clients that keep retrying. ### Caveats - **Server support is required.** The client sends `X-QWP-Request-Durable-Ack: true` on the upgrade. The server must echo - back `X-QWP-Durable-Ack: enabled`. If it does not — OSS build, - uninitialised primary, missing registry, hitting a replica — the - connect **fails loudly**, by design. Silently waiting for ack frames - that never arrive would let the SF disk fill up. + back `X-QWP-Durable-Ack: enabled`. Without it, for example on an OSS build + or an uninitialised primary, the connection attempt is rejected. In most + clients this is terminal. The exceptions follow. +- **Senders that keep retrying.** The Java client retries after a sender's + first successful connection. Node.js senders with a background start + (`initial_connect_retry=async` or `lazy_connect=on`) retry from startup, + even with `sf_dir`. With `sf_dir` and a foreground start, the first + connection fails but later mismatches are retried after a successful + connection. Retrying Node.js senders emit `durable-ack-unavailable` + [connection events](/docs/connect/clients/nodejs/#connection-events). + Monitor retrying senders and their buffer usage: continued buffering can + fill the journal or memory queue even though startup succeeded. See + [Node.js durable acknowledgement](/docs/connect/clients/nodejs/#durable-acknowledgement). - **Idle keepalive.** The OSS server only flushes pending durable-ack frames during inbound recv events. The client sends a WebSocket PING every `durable_ack_keepalive_interval_millis` (default 200 ms) when @@ -142,6 +154,11 @@ spawn background drainers to clear them. - You prefer "automatic eventual delivery" over "operator manually reattaches the slot." +On the Node.js client, both a restarted process and a drainer recover a +crashed sender's slot only if they can reclaim its lock, which fails in a +replaced container or in a container restarted in place. Clear such locks +with [Node.js lock recovery](/docs/high-availability/store-and-forward/operating-and-tuning/#nodejs-lock-recovery). + ### Leave it off when - Each `sender_id` is statically pinned to a specific process — there @@ -170,7 +187,7 @@ If you are currently using HTTP or TCP ILP ingest, the comparison is: | Server outage tolerance | Best-effort retry | None | Reconnect loop with multi-minute budget | | Multi-host failover | Yes (HTTP only) | No | Yes | | Cross-region durability ack | No | No | Yes (`request_durable_ack=on`) | -| Cluster-wide ordering | Best-effort | Best-effort | FSN-driven, server-deduplicated | +| Cluster-wide ordering | Best-effort | Best-effort | FSN-ordered; replay is at least once, so use `DEDUP UPSERT KEYS` | The transition is application-transparent — `Sender.fromConfig` accepts a `ws::` or `wss::` connect string and the public builder API is the diff --git a/documentation/high-availability/tuning.md b/documentation/high-availability/tuning.md index 73b93a1a9..0b34977b2 100644 --- a/documentation/high-availability/tuning.md +++ b/documentation/high-availability/tuning.md @@ -22,7 +22,7 @@ restart. | Setting | Node | Default | What it does | |---------|------|---------|-------------| -| `replication.primary.throttle.window.duration` | Primary | `10000` (10s) | Maximum time before an incomplete WAL segment is flushed | +| `replication.primary.throttle.window.duration` | Primary | `1000` (1s) | Maximum time before an incomplete WAL segment is flushed | | `replication.replica.poll.interval` | Replica | `1000` (1s) | How often the replica checks for new data | | `cairo.wal.segment.rollover.size` | Primary | `2097152` (2 MiB) | Max WAL segment size before rollover | @@ -59,7 +59,7 @@ replication.replica.poll.interval=100 No configuration needed. The defaults are: -- `replication.primary.throttle.window.duration=10000` (10s) +- `replication.primary.throttle.window.duration=1000` (1s) - `replication.replica.poll.interval=1000` (1s) - `cairo.wal.segment.rollover.size=2097152` (2 MiB) @@ -124,8 +124,8 @@ write ops typically cost ~\$5/million and read ops ~\$0.40/million. |---|---| | 50ms / 50ms | ~$280 | | 100ms / 100ms | ~$140 | -| 1s / 1s | ~$14 | -| 10s / 1s (default) | ~$2 | +| 1s / 1s (default) | ~$14 | +| 10s / 1s | ~$2 | Multiply by the number of tables being actively written to. With 10 tables at 100ms intervals, that's ~$1,400/month in API charges alone. With NFS, that same @@ -212,7 +212,7 @@ Tiering requires files over 128 KiB. ### Throttle window ```ini -replication.primary.throttle.window.duration=10000 # 10 seconds (default) +replication.primary.throttle.window.duration=1000 # 1 second (default) ``` Maximum time before uploading an incomplete segment. If a segment hasn't reached @@ -223,8 +223,8 @@ segments fill up before upload, reducing redundant uploads (write amplification) |-------|----------| | `50` (50ms) | Ultra-low latency. Best with NFS transport. | | `100` (100ms) | Low latency. Good balance for NFS transport. | -| `1000` (1s) | Low latency for object store transport. | -| `10000` (10s) | Default. Balanced. | +| `1000` (1s) | Default. Low latency for object store transport. | +| `10000` (10s) | 10 second delay OK. Fewer uploads. | | `60000` (60s) | 1 minute delay OK. Fewer uploads. | | `300000` (5 min) | Cost-sensitive. Batches more data. | diff --git a/documentation/partials/_sf-dedup-warning.partial.mdx b/documentation/partials/_sf-dedup-warning.partial.mdx index d01c79f1f..35d419391 100644 --- a/documentation/partials/_sf-dedup-warning.partial.mdx +++ b/documentation/partials/_sf-dedup-warning.partial.mdx @@ -1,11 +1,11 @@ -:::caution Replay is at-least-once — enable DEDUP +:::caution Replay is at-least-once — use DEDUP for exactly-once outcomes After a reconnect or a sender restart, the client replays frames the server may have accepted but not yet acknowledged. Without -[DEDUP](/docs/concepts/deduplication/) on the target table, replay produces -duplicate rows. Tables ingested over a reconnecting or multi-host connection -**must** declare `DEDUP UPSERT KEYS(...)` covering row identity. See -[Delivery semantics](/docs/concepts/delivery-semantics/) for the full -at-least-once / exactly-once model. +[DEDUP](/docs/concepts/deduplication/) on the target table, replay can produce +duplicate rows. If your application requires exactly-once outcomes, declare +`DEDUP UPSERT KEYS(...)` covering row identity. Applications that tolerate +occasional duplicates can skip DEDUP; see +[Delivery semantics](/docs/concepts/delivery-semantics/) for the full model. ::: diff --git a/documentation/query/overview.md b/documentation/query/overview.md index e39f89004..ee3723ef1 100644 --- a/documentation/query/overview.md +++ b/documentation/query/overview.md @@ -97,19 +97,31 @@ against the demo instance. ## QuestDB client libraries The official client libraries speak the QuestDB Wire Protocol (QWP), a binary -protocol that carries both ingestion and query traffic over one connection and -one configuration string. This is the fastest way to get data out of QuestDB -from an application. +protocol for both ingestion and querying, configured with one connection +string. Ingestion and queries run over separate WebSocket connections, which +the clients' pools manage for you. Results stream rather than arriving in one block. The server sends batches as -it produces them, so an application starts processing the head of a result -while the tail is still being computed, and a result larger than memory never -has to be materialized at all. - -Connections recover on their own. When a connection drops and replicas are -available, the client reconnects and retries against another one without the -application intervening. A query that fails over restarts from the beginning, -which is transparent if you materialize the whole result. +it produces them, so an application can process the head of a result while +the tail is still being computed without materializing the whole result. +Clients may buffer batches ahead of the consumer: for large Node.js results, +set a [byte-credit window](/docs/connect/clients/nodejs/#flow-control) to +bound client-side buffering (the default is unbounded). + +Connections can recover from transport failures. With failover enabled, a +client may reconnect and re-execute an in-flight query, including on the same +host, and the result then restarts from its first row. Helpers that collect a +whole result, such as Python's `to_pandas()`, discard the partial result for +you. If you process batches yourself, reset any accumulated state when the +result restarts, or handle the client's error. Re-execution can also repeat +SQL writes such as `INSERT`. Each client describes its failover behavior: +[Java](/docs/connect/clients/java/#query-failover), +[Python](/docs/connect/clients/python/#reader-failover), +[Go](/docs/connect/clients/go/#query-failover), +[Rust](/docs/connect/clients/rust/#failover-and-errors), +[C and C++](/docs/connect/clients/c-and-cpp/#querying-data), +[.NET](/docs/connect/clients/dotnet/#failover), and +[Node.js](/docs/connect/clients/nodejs/#query-failover). The Rust, C++, and Python clients hand back results as Arrow record batches. That is the native memory layout of @@ -117,8 +129,15 @@ That is the native memory layout of [Polars](/docs/integrations/data-processing/polars/), and DuckDB, so a query becomes a DataFrame with no row-by-row conversion in between. -See the [Connect overview](/docs/connect/overview/) for the per-language -guides and which clients ship QWP today. +To get started, see the querying section of your client: +[Java](/docs/connect/clients/java/#querying-with-query-and-completion), +[Python](/docs/connect/clients/python/#querying), +[Go](/docs/connect/clients/go/#querying-and-sql-execution), +[Rust](/docs/connect/clients/rust/#querying), +[C and C++](/docs/connect/clients/c-and-cpp/#querying-data), +[.NET](/docs/connect/clients/dotnet/#querying-and-sql-execution), or +[Node.js](/docs/connect/clients/nodejs/#querying). The +[Connect overview](/docs/connect/overview/) compares the clients. ## PostgreSQL diff --git a/package.json b/package.json index b9d5f8749..82d1eff57 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,8 @@ "build": "cross-env NO_UPDATE_NOTIFIER=true USE_SIMPLE_CSS_MINIFIER=true PWA_SW_CUSTOM= docusaurus build", "deploy": "docusaurus deploy", "serve": "docusaurus serve", - "swizzle": "docusaurus swizzle" + "swizzle": "docusaurus swizzle", + "test": "node --test \"plugins/**/*.test.js\"" }, "dependencies": { "@docusaurus/faster": "^3.8.1", diff --git a/plugins/raw-markdown/convert-components.js b/plugins/raw-markdown/convert-components.js index fc576b441..172b065c9 100644 --- a/plugins/raw-markdown/convert-components.js +++ b/plugins/raw-markdown/convert-components.js @@ -605,18 +605,47 @@ function bumpHeadings(markdown, bumpBy = 1) { } /** - * Removes import statements from processed markdown - * Handles both single-line and multi-line imports + * Removes MDX import statements without touching imports in fenced examples. + * Handles both single-line and multi-line imports outside code fences. * @param {string} content - The markdown content - * @returns {string} Content with imports removed + * @returns {string} Content with MDX imports removed */ function removeImports(content) { - let processed = content - // First handle single-line imports - processed = processed.replace(/^import\s+.+\s+from\s+['"].+['"];?\s*$/gm, '') - // Then handle multi-line imports (where line breaks exist) - processed = processed.replace(/^import\s+[\s\S]*?\s+from\s*\n?\s*['"].+['"];?\s*$/gm, '') - return processed + const lines = content.split('\n') + const output = [] + let segmentStart = 0 + let fenceChar = '' + let fenceLen = 0 + + function appendOutside(end) { + if (segmentStart === end) return + let segment = lines.slice(segmentStart, end).join('\n') + segment = segment.replace(/^import\s+.+\s+from\s+['"].+['"];?\s*$/gm, '') + segment = segment.replace(/^import\s+[\s\S]*?\s+from\s*\n?\s*['"].+['"];?\s*$/gm, '') + output.push(...segment.split('\n')) + } + + for (let i = 0; i < lines.length; i++) { + const line = lines[i] + // Ignore a CRLF line ending, so fences in CRLF files are still recognized. + const fence = line.replace(/\r$/, '').match(/^ {0,3}(`{3,}|~{3,})(.*)$/) + if (!fenceChar) { + // CommonMark: a backtick fence's info string cannot contain a backtick. + if (!fence || (fence[1][0] === '`' && fence[2].includes('`'))) continue + appendOutside(i) + fenceChar = fence[1][0] + fenceLen = fence[1].length + } else if (fence && fence[1][0] === fenceChar && + fence[1].length >= fenceLen && /^\s*$/.test(fence[2])) { + fenceChar = '' + fenceLen = 0 + segmentStart = i + 1 + } + output.push(line) + } + + if (!fenceChar) appendOutside(lines.length) + return output.join('\n') } /** diff --git a/plugins/raw-markdown/convert-components.test.js b/plugins/raw-markdown/convert-components.test.js new file mode 100644 index 000000000..70ccd63aa --- /dev/null +++ b/plugins/raw-markdown/convert-components.test.js @@ -0,0 +1,64 @@ +const assert = require('node:assert/strict') +const fs = require('node:fs') +const path = require('node:path') +const test = require('node:test') +const matter = require('gray-matter') +const { removeImports } = require('./convert-components') + +test('removes MDX imports but keeps TypeScript imports in fenced examples', () => { + const markdown = [ + 'import Widget from "@site/src/components/Widget"', + 'import {', + ' Tabs,', + ' TabItem,', + '} from "@theme/Tabs"', + '', + '```typescript', + 'import {', + ' connectQwpNodeClient,', + ' QwpEgressQueryError,', + '} from "@questdb/nodejs-client";', + '```', + '', + '~~~ts', + 'import { client } from "@questdb/nodejs-client";', + '~~~', + ].join('\n') + + const result = removeImports(markdown) + assert.doesNotMatch(result, /@site\/src\/components\/Widget|@theme\/Tabs/) + assert.match(result, /```typescript\nimport \{\n connectQwpNodeClient,\n QwpEgressQueryError,\n\} from "@questdb\/nodejs-client";\n```/) + assert.match(result, /~~~ts\nimport \{ client \} from "@questdb\/nodejs-client";\n~~~/) +}) + +test('recognizes fences in files with CRLF line endings', () => { + const markdown = [ + 'import Widget from "@site/src/components/Widget"', + '```typescript', + 'import { Sender } from "@questdb/nodejs-client";', + '```', + ].join('\r\n') + + const result = removeImports(markdown) + assert.doesNotMatch(result, /@site\/src\/components\/Widget/) + assert.match(result, /import \{ Sender \} from "@questdb\/nodejs-client";/) +}) + +test('does not open a fence on backticks whose info string has a backtick', () => { + const markdown = [ + '```not `a fence`', + 'import Widget from "@site/src/components/Widget"', + ].join('\n') + + const result = removeImports(markdown) + assert.doesNotMatch(result, /@site\/src\/components\/Widget/) +}) + +test('preserves imports from the Node.js Quick start while removing its MDX import', () => { + const file = path.join(__dirname, '../../documentation/connect/clients/nodejs.md') + const { content } = matter(fs.readFileSync(file, 'utf8')) + const result = removeImports(content) + + assert.doesNotMatch(result, /^import SfDedupWarning from /m) + assert.match(result, /```typescript\nimport \{ connectQwpNodeClient \} from "@questdb\/nodejs-client";/) +}) diff --git a/shared/clients.json b/shared/clients.json index 649cbf198..393030b9d 100644 --- a/shared/clients.json +++ b/shared/clients.json @@ -31,6 +31,14 @@ "logo": "/images/logos/python.svg", "protocol": "QWP" }, + { + "href": "/docs/connect/clients/nodejs", + "name": "Node.js", + "description": + "Pooled QWP ingestion and streaming SQL for Node.js and TypeScript.", + "logo": "/images/logos/nodejs-light.svg", + "protocol": "QWP" + }, { "href": "/docs/connect/clients/dotnet", "name": ".NET", @@ -104,10 +112,10 @@ "protocol": "PGWire" }, { - "href": "/docs/connect/clients/nodejs", + "href": "/docs/connect/clients/nodejs#ilp-transports-legacy", "name": "Node.js", "description": - "JavaScript runtime client. Ingests over ILP; QWP support is planned.", + "Node.js client for ILP ingestion over HTTP and TCP, alongside QWP.", "logo": "/images/logos/nodejs-light.svg", "protocol": "ILP" },