[jovfer]

Tandem: questdb/questdb#7979, questdb/questdb-enterprise#1308

Why

Applications that borrow a pooled Sender per message paid one QWP frame and one server-side WAL commit per message, because the pooled close() flushed. A customer benchmark of that shape took 54 s for 1M rows against 6.5 s for the batched HTTP sender, and the server's commit rate was the ceiling. The quick start taught exactly that shape.

What changes

Close releases, flush ships, drain waits. close() on a borrowed sender now returns the slot and ships the slot's pending rows once the oldest is older than auto_flush_interval or another auto-flush threshold trips. Rows left in the slot go out when an auto-flush threshold trips on a later append or release, or when the pool's housekeeper settles the idle slot on the auto_flush_interval cadence (100 ms by default). A lease that holds the slot without appending or closing holds earlier leases' rows back until it does. Borrowing per message therefore batches exactly like a sender held for the whole run. flush() on a lease still ships now, drain() still waits for the server.

• flush_on_close=on (builder flushOnClose(true)) restores the behaviour before 1.3.12: one frame and one commit per close. A directly built SenderPool keeps flushing on close; only the QuestDB facade defaults to off.
• The housekeeper gains a settle step on its own cadence; reaping stays on housekeeper_interval_ms. A slot nobody borrows again no longer keeps its last rows until the handle closes.
• The free list is LIFO, so per-message borrows concentrate on hot slots and batches fill.
• A partial row at close() is cancelled and the slot kept (the close still throws); a dead delegate is retired by an explicit probe instead of a throwing flush.
• Close-time errors are wrapped: a LineSenderServerException keeps its type, other errors arrive as a LineSenderException whose cause is the delegate's error. A lease body that already threw keeps its exception as the primary one (previously try-with-resources failed with "Self-suppression not permitted").
• Sender gains ten default methods used by the pool: checkError, hasRowInProgress, pendingRowCount, rowsAppended, flushDeferred, hasDeferredFrames, markSavepoint, resetToSavepoint, autoFlushIfDue, tryFlush.

Commit window. With flush_on_close=on, commit_rows and commit_interval_ms (builder commitRows, commitIntervalMillis; setting one defaults the other to 1000 rows / 100 ms) make each close ship a frame flagged defer-commit and the pool commit once per window: the release that fills the window ships the committing frame, and the settle commits an idle open group. Rows are in the store-and-forward ring the moment close() returns, and the server commits once per window instead of once per close. Acks arrive at the commit, so a drain() should follow a flush(). A rejection rolls back the whole open group, every lease since the last commit. The builder injects transaction=on for this mode and rejects a window with flush_on_close=off (designed, gated until measured) or with transaction=off.

Lease-scoped reset(). borrow() marks a savepoint on the delegate and the pooled reset() rolls back to it, so a borrower following the BatchTooLargeForCapException remedy discards only its own rows, not those earlier leases left in the slot. The WebSocket sender rolls every table buffer back to the mark, restores the pending counters and the batch symbol watermark, reclaims the dictionary ids the dropped rows registered (delta mode), and drops column definitions the server has not seen. A pooled sender ignores a borrower's markSavepoint(); its resetToSavepoint() is the lease-scoped reset().

Byte trigger bounded by the server cap. With auto_flush_bytes unset, the WebSocket sender flushes at 90 percent of the batch cap the server advertises; an explicit off keeps byte flushing off, and a node that advertises no cap gets no byte trigger. This changes the standalone default too: a batch the server would reject ships earlier instead. Rows appended before the first handshake (async connect) carry no byte trigger; the first append after it flushes the backlog.

Over-cap batches. The sender remembers a cap rejection until reset(), a rollback or a later flush that passes the pre-flight check, and reports it through checkError(). When the cap rejects the auto-flush that follows a row and other rows are pending, the sender ships the rows before that row and keeps the row buffered. A held row that alone reaches 90 percent of the cap flushes at once, also with auto_flush_bytes=off, so a row whose own frame exceeds the cap fails on the lease that wrote it. At release the pool rolls the lease back to its borrow mark and retries the flush: when the rows earlier leases left ship, the slot stays in service and the borrower gets a wrapper naming its own dropped rows. Only an inherited batch the cap still rejects retires the slot, and the settle step retires such a slot directly. Both retirements log at ERROR with the dropped row count.

Housekeeper and shutdown. Housekeeper deadlines use a monotonic clock and saturate, so housekeeper_interval_ms=Long.MAX_VALUE disables reaping and nothing else. The settle step flushes one idle slot at a time, a failed startup recovery step is retried once a second, and an interrupted appendBlocking parks instead of spinning. The reaper skips idle slots that still hold rows or an open commit group. SenderPool.close() logs ERROR with the row count when an idle slot's delegate close fails. The settle publishes through Sender.tryFlush() and never waits for a full store-and-forward ring: a busy ring returns the slot at once and the rows wait for the next settle; a settle holds a slot past QuestDB.close() only while frames after its first wait for ring room. checkError() also reports a failed symbol-dictionary recycle.

Docs. The README, Javadoc and examples say that by default one frame carries several leases' rows, name flush() before close() and flush_on_close=on as the isolation tools, and no longer promise replay after a rejection. Comments that said close() flushes are corrected.

Wording trap. In this client "flush" means published to the store-and-forward ring, not acknowledged by the server. flush_on_close=on does not mean committed when close() returns.

Design decisions and trade-offs

• Shared frames by default. With the default, a terminal rejection of one lease's row discards the other leases' rows in the same frame, after their close() calls returned, and the error surfaces on a later call on that slot, possibly to a lease that wrote none of the rejected rows. This is the HTTP sender's failure span moved across threads. The alternative, flush_on_close=on as the default, keeps one frame and one commit per message, which is the regression this PR removes.
• Hold-back of the crossing row. The byte trigger fires after the row that crosses it is committed, so a cap rejection holds that row back rather than failing a lease's valid rows because of rows earlier leases left. Two effects remain:
  • Columns first used by a held row ride in the prefix frame, all-null, so the server creates them even if the lease then calls reset().
  • When the rows before the crossing row still exceed the cap (only with auto_flush_bytes=off or after failed flushes), atNow() throws the standalone remedy text, which tells a pooled lease to call reset(). On a lease reset() keeps the inherited rows, so the next append is rejected again until close() rolls the lease back or retires the slot.
• Savepoint walk per borrow. The savepoint mark walks every table buffer of the delegate. It runs after the pool lock is released, at parity with the pre-PR pool's per-close walk, but still grows with the table count: about 17 us per borrow at 1,000 tables and 72-74 us at 5,000 on one thread. At 4 threads borrows run at 232-239k/s with 1,000 tables and 53-54k/s with 5,000, against 151-233k/s and 37-54k/s for the pre-PR pool. An O(1) mark is a follow-up.
• The reaper keeps slots that hold rows. A slot whose settle keeps failing is not reaped, by idle timeout or max_lifetime_ms, until a settle succeeds; reaping it would drop the rows silently. Rows in an idle slot whose delegate close fails at shutdown are lost, as before this PR, where the lease's close() threw instead; they are now logged at ERROR.
• A direct pool keeps flushing on close. SenderPool defaults flushOnClose to true, because nothing settles a direct pool's slots; the facade sets it from flush_on_close. flush_on_close in a direct pool's config string is accepted and ignored, like sender_pool_max, idle_timeout_ms and the other pool keys, which ConfigSchema marks as applied by the facade only.
• Small advertised caps in tests. One existing test with a 150-byte advertised cap needed auto_flush_bytes=off to keep its explicit-flush control flow. Downstream tests that call setAdvertisedMaxBatchSize with small values may need the same.
• First frame only. tryFlush() refuses to wait only for the first frame of a flush; once a frame is on the ring the flush completes as before, so a split batch or dictionary chunks never leave a partial group behind.

Testing

Wire tests cover each behaviour above, and each new test fails when its target line is mutated. The PR's test classes pass on the branch after merging main (365 tests in 16 classes). CI runs the full suite, including the JDK 8 job for the Java 8 floor.

Benchmark

QwpPoolBorrowPerRowBenchmark (questdb/questdb#7979, benchmarks/) ran against a standalone server on the same machine (macOS on Apple Silicon, local SSD, other load running): a WAL table, one thread, sender_pool_max=1. Each time runs until the server has applied every row; ranges cover two runs.

Default cairo.commit.mode (nosync), 1M rows:

configuration	time to rows applied	WAL commits
held lease, default	69-72 ms	1,000
borrow per row, default	95-97 ms	1,000
borrow per row, flush_on_close=on (previous behaviour)	4,732-4,752 ms	1,000,000
borrow per row, flush_on_close=on + window 1000/100	3,333-3,409 ms	1,000

cairo.commit.mode=sync, 100k rows:

configuration	time to rows applied	WAL commits
held lease, default	36-37 ms	100
borrow per row, default	35-36 ms	100
borrow per row, flush_on_close=on (previous behaviour)	29,780-31,300 ms	100,000
borrow per row, flush_on_close=on + window 1000/100	358-363 ms	100

The default ships one frame and one commit per auto_flush_rows batch. The window removes the per-message commits but keeps one frame per close(), so its gain depends on what a commit costs on the server: compared with flush_on_close=on alone, it cuts the time by about 30% with nosync and by about 85x with sync. macOS fsync does not flush the drive cache, so a sync commit typically costs more on Linux.

Release note (1.3.12)

close() on a sender borrowed from QuestDB no longer flushes; it ships the slot's pending rows only when the oldest is older than auto_flush_interval or another auto-flush threshold trips. Set flush_on_close=on to restore flushing. A directly built SenderPool still flushes on close. Rows left in a slot ship within auto_flush_interval once the slot is idle, or at the next append or close on it after that interval. Only LineSenderServerException keeps its type at a pooled close(); other close-time errors arrive as a LineSenderException with the original as cause. New keys commit_rows and commit_interval_ms. reset() on a pooled sender discards only the rows appended through that lease. With auto_flush_bytes unset, the WebSocket sender flushes at 90 percent of the server's advertised batch cap. The free list is LIFO, so an idle pool shrinks to its working set; set sender_pool_min equal to sender_pool_max to keep every connection warm.

Follow-ups

• Settle-failure WARN flag clears only on a settle-side success.
• Server-side QWP commit interval and per-connection schema caching (OSS), and the fourth combination gate, per the spec.
• Documentation repository: quick start, pool and migration sections.
• A slot retired with an over-cap batch, or an idle slot whose close fails at shutdown, logs at ERROR but does not reach SenderErrorHandler; routing client-side drops there needs a client-side error category.
• O(1) savepoint: mark each table buffer on its first touch after borrow() instead of walking every buffer.
• Settling a slot that is leased but idle. It needs a guard between the lease's row boundaries and the settle flush. Today a lease that never appends or closes still holds earlier rows.
• A non-cap failure partway through the hold-back retry's split publish leaves the restored over-cap batch with no retained rejection until the next flush attempt.
• A pooled-specific remedy in the rejection that atNow() throws when the rows before a crossing row still exceed the cap (see the hold-back trade-off).
• Encoding only the columns that existed before a held row, so a rolled-back held row leaves no empty column on the server.
• Known narrow gaps:
  • In full-dictionary mode a symbol value larger than the cap stays in the dictionary after the rollback (id reclaim is delta-mode only), so the release path keeps the slot and later leases that add a new symbol fail with the "dropped N row(s)" wrapper. Retiring the slot when the rejection came from the dictionary would restart the id space.
  • In the full-dictionary chunk fallback, a non-cap failure can leave an older cap rejection latched.
  • No test drives the release-time rollback and retry through close's own flush with flush_on_close=on, with or without a commit window.
  • A node that advertises no cap gets no byte bound on the config-string path, while the programmatic path keeps 8 MiB.
  • The housekeeper wakes on the settle cadence, 100 ms by default, for the life of the handle, retries the startup-recovery step once a second after it completes, and a settle drop logs two lines.

🤖 Generated with Claude Code

https://claude.ai/code/session_011PpCzTucC66DiMgFiSuduj

[jovfer]

PR #108 tandem review, round 4 (level 3)

Reviewed at level 3, tandem mode, as one logical change across three PRs:

PR	Head	Base
#108	47603c71	70a7f853 (round 3 ended at b2de8e4f)
questdb/questdb#7979	1f84849722	69796aab30
questdb/questdb-enterprise#1308	3087dda29	73dd5c863

Verdicts:

• Client feat(qwp)!: stop flushing pooled senders on close, add a commit window and a lease-scoped reset #108: approve with comments. Please address Moderate 1.
• OSS #7979: approve with comments. Please fix Moderate 2 in this PR; otherwise OSS master gets a flaky test.
• Ent #1308: approve.

Every behavioural finding below was executed on both the head and the base revision.

• PR metadata:
  • All three titles follow the convention and pass the title check.
  • Labels are set: enhancement, tandem, QWP on feat(qwp)!: stop flushing pooled senders on close, add a commit window and a lease-scoped reset #108; tandem, non-breaking-tandem, QWP on #7979; tandem, QWP on #1308.
  • The OSS body's claim holds. The base version of testTwoSendersPerTableWhileReaderThreadStreamsTwoQueries fails against the new client with txn timed out [table=alpha, expectedTxn=2, writerTxn=1], and the PR version passes.
  • The Enterprise PR body's claims about its tests hold.
• Submodules:
  • Enterprise questdb, e0b540f913 -> 1f84849722: OFF-DEFAULT, in scope. The move also carries 69796aab30 (#7640), which is already on OSS master and out of scope.
  • OSS java-questdb-client, c41a4172 -> 47603c71: OFF-DEFAULT, in scope. The move also carries 21e32683 and 70a7f853 (test(qwp): close TestWebSocketServer connections gracefully so scripted drops deliver their acks #95), which are already on client main and out of scope.
• CI:
  • Client and Enterprise CI are green.
  • The OSS Azure test pipeline did not run on draft #7979, so no OSS test has run against the new client in CI.
  • A local run of the OSS QWP ingress, QWP e2e, WebSocket and facade test classes against the head client passed: 594 tests, 0 failures, 1 skipped.

Critical

None.

Moderate

1. [client] QuestDB.close() returns while the settle step still holds a slot blocked on a full ring

• Problem: a settle flush blocked on a full store-and-forward ring outlives QuestDB.close().
• Net impact: pool users who close the handle while the SF ring is full. The slot's lock and connection stay held for up to about 111 s after close() returns.
• Evidence: probe at 47603c71 and 70a7f853, all timeouts at their defaults, ring capped at 128 KiB, server never acks:
  • Head: the lease's close() returned in 0 ms. QuestDB.close() returned after 9,009 ms. An immediate QuestDB.connect on the same sf_dir failed with SlotLockContentionException: sf slot already in use by another process [... default-0, holder=pid=<own pid>]. The lock was released 110,981 ms after close() returned.
  • Base: the lease's close() blocked 120,001 ms and threw cursor SF append failed. QuestDB.close() then returned in 0 ms, and the immediate re-open succeeded.
  • The same result holds after a real outage (server closed after connect), and in two other probes with shorter deadlines.

Trigger:

• Facade default flush_on_close=off, with sf_dir set.
• The server is down or not acking long enough to fill sf_max_total_bytes (10 GiB by default with sf_dir, 128 MiB in memory).
• A lease closes with rows still in the slot. The housekeeper's settle takes the slot and blocks in CursorSendEngine.appendBlocking for up to sf_append_deadline_millis (30 s).
• The application calls QuestDB.close(), then reconnects in the same process.

Path:

• SenderPool.settleIdle() (SenderPool.java:1769-1816) takes the slot off the free list and flushes it outside the lock. appendBlocking deliberately does not stop on an interrupt.
• QuestDBImpl.close() (QuestDBImpl.java:237) first runs PoolHousekeeper.stop(): a 2 s join, an interrupt, then another 2 s join.
• SenderPool.close() then waits min(acquire_timeout_ms, 5 s) for the internal lease (SenderPool.java:1626). It logs "1 borrowed sender lease(s) still outstanding ... or leaks if it never is", and returns.
• Later, on the housekeeper, the append fails. requeueSettled sees closed (:1997) and calls retireLease, so delegate.close() appends again (another 30 s) and then drains for up to close_flush_timeout_millis (60 s).

What the application sees on head:

• close() returns normally, yet a re-open on the same sf_dir fails until the housekeeper finishes.
• The close-time WARN describes the internal settle lease as a borrowed lease that may leak, although the application closed every lease.
• The dropped rows are reported only by an ERROR on the housekeeper about 100 s after close() returned ("slot 0 retired with 1 unshipped row(s); delegate close failed"). If the JVM exits first, that line is never written. Base threw on the lease's close().
• In memory mode there is no slot lock, but the delegate's I/O thread and native buffers still outlive close() (4.5 s close, delegate still open, in a separate probe).

Base comparison:

• For the same steps, base returns from QuestDB.close() with the slot released, and re-open works.
• Base can show the same symptom through other triggers:
  • An idle reap that is closing a delegate on the housekeeper when close() runs. This was reproduced on both revisions: close() took 4,015 ms, the re-open failed, and the lock was released about 10.8 s later.
  • A startup-recovery drain.
• What this PR adds is a deterministic path. A single thread that runs lease.close(); db.close(); and then reconnects hits it every time once the ring is full.

Why Moderate, not Critical:

• The rows are lost on both revisions.
• The trigger needs a full ring.
• The kernel releases the lock at process exit.
• The window is bounded at about two minutes with defaults.

Suggested fix:

• Make the settle flush non-blocking: one append attempt, and on backpressure requeue the slot and retry on the next tick. This is the "non-blocking settle" follow-up the PR already lists, and it also removes the borrower wait behind a settle. A settle then never holds a slot across close(), and close()'s idle loop closes the delegate on the caller's thread, as on base.
• PoolHousekeeper.runLoop sets nextSettleAt from a now read before settleIdle() runs (PoolHousekeeper.java:246-251). After a 30 s settle the next one therefore starts with no wait. Read the clock after the settle.
• At minimum, report the internal lease separately from borrowed leases in the close-time WARN. Also state in the QuestDB.close() Javadoc that a settle in progress can outlive it.

2. [OSS, cross-repo] testElasticPoolGrowsUnderBurstAndShrinksWhenIdle is flaky with the new client

• Problem: the test assumes one WAL transaction per pooled close().
• Net impact: OSS CI. Under CPU load about one run in three fails once #7979 pins the new client.
• Evidence: the unmodified test, at OSS 1f84849722:
  • In a 40-iteration loop under CPU load (14-core Mac, 16 busy processes):
    • Head client 47603c71: 13 of 40 failed with txn timed out [table=el, expectedTxn=4, writerTxn=3] (10 runs) or writerTxn=2 (3 runs).
    • Base client 70a7f853: 0 of 40 failed.
  • Without load, in a warm-JVM loop:
    • Head: 6 of 100 iterations reused a slot, and all 6 ended at 3 transactions.
    • Base: 2 of 100 reused a slot, and both still reached 4.
  • Forced reuse (threads 1-3 wait until thread 0's lease has closed): head failed, base passed.

Details:

• The test is at core/src/test/java/io/questdb/test/client/QuestDBFacadeE2ETest.java:838-910; #7979 does not touch it.
• Four latch-released threads each borrow a sender, write one row and close it. The comment at :884 says "Each of the 4 burst senders flushes once on close -> 4 WAL txns", and :886 then waits for awaitTxn("el", 4).
• With the new client, a release ships nothing until a threshold trips or the settle runs (PooledSender.java:595-599). borrow() hands out the most recently returned slot (SenderPool.java:1252).
• A burst thread that borrows after another thread's lease has come home therefore appends to that slot. The settle ships both rows in one frame (server log: seqTxn=2, rowLo=0, rowHi=2), the table reaches 3 transactions, and awaitTxn throws after 15 s.
• On base every close flushed, so a reused slot still produced 4 frames.
• This test never ran in CI against the new client, because OSS Azure did not run on #7979.

Suggested fix:

• Call s.flush() inside the burst lease before it closes, the same remedy #7979 applied to testTwoSendersPerTableWhileReaderThreadStreamsTwoQueries.
• Correct the comment at :884.
• Waiting on the row count instead of the transaction count would also work.

Minor

• [client] Stale PooledSender Javadoc. The class Javadoc (PooledSender.java:50-52, "by default, ships nothing") and the close() Javadoc (:150-154, "By default nothing is shipped") still say that a release ships nothing.
  • The ship() Javadoc, corrected in round 3 (:576-578), says a release ships every pending row once the oldest is older than auto_flush_interval or another threshold trips.
  • A directly built SenderPool flushes on close by default (SenderPool.java:258-260).
  • Suggested: reuse the ship() wording and name the direct-pool default.
• [OSS] Stale bytes=0 in a test comment. QuestDBFacadeE2ETest.java:700 still reasons with "rows=1000 (> both row counts) and bytes=0". With the new client, an unset auto_flush_bytes means 90 percent of the advertised cap, not 0.
  • The conclusion (exactly two flushes per table) still holds, because the batches are a few KB.
  • #7979 edited the lines on either side of this one.

Coverage map

The test gate passes. There are no admitted coverage gaps.

• Round-3 tests: mutation runs at 47603c71 turn at least one test red for each of these mutations:
  • removing the reapIdle guard;
  • removing the SenderPool.close() ERROR log;
  • setting the direct-pool default flushOnClose to false;
  • removing QuestDBImpl's setFlushOnClose(...);
  • changing the pendingRowCount < 2 gate to < 1;
  • making the hold-back retry non-transactional.
• OSS facade fix: a working regression test. The base version of the test is red against the new client and the PR version is green.

Summary

• Verdicts:
  • Client feat(qwp)!: stop flushing pooled senders on close, add a commit window and a lease-scoped reset #108: approve with comments (Moderate 1).
  • OSS #7979: approve with comments (Moderate 2, plus the Minor OSS comment).
  • Ent #1308: approve.
• Severity: 0 Critical, 2 Moderate, 2 Minor.
• Where the findings sit:
  • Moderate 1 is in-diff: the new settle step meets the unchanged close budgets.
  • Moderate 2 is out-of-diff breakage: an unchanged OSS test is broken by the client contract that #7979 pins.
  • The Minor items are in-diff: Javadoc from this PR, and an OSS comment block that #7979 edited.
• Trade-off: Moderate 1 is the cost of moving the close-time flush off the borrower's thread. With a full ring, base blocked the borrower's close() for up to two minutes and then threw.
• Store-and-forward contract:
  • The pool's new calls on borrower and housekeeper threads (checkError(), autoFlushIfDue(), the settle flush()) surface only latched terminal errors and cap rejections, the same set base surfaced through the close-time flush().
  • A settle that fails transiently requeues the slot instead of retiring it.
  • No reconnect budget or new terminal path was added.
• Submodule provenance:
  • Enterprise questdb: OFF-DEFAULT, in scope, except the upstream commit 69796aab30.
  • OSS java-questdb-client: OFF-DEFAULT, in scope, except the upstream commits 21e32683 and 70a7f853.
• Callsite analysis covered:
  • every OSS and Enterprise consumer of the client:
    • the OSS facade E2E tests;
    • the OSS QWP e2e tests, including the tight-cap and oversize-row tests, under the new 90 percent byte trigger;
    • the Enterprise consumers in the tandem PR;
  • the QuestDBImpl constructor and direct SenderPool callers (none outside the client).

Artifacts

Probe sources and run logs for every finding, on head and base, are kept locally and are available on request.

[mtopolnik]

[PR Coverage check]

😍 pass : 609 / 646 (94.27%)

file detail

	path	covered line	new line	coverage
🔵	io/questdb/client/Sender.java	8	12	66.67%
🔵	io/questdb/client/impl/PoolHousekeeper.java	32	40	80.00%
🔵	io/questdb/client/impl/PooledSender.java	87	96	90.62%
🔵	io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java	153	160	95.62%
🔵	io/questdb/client/impl/SenderPool.java	133	139	95.68%
🔵	io/questdb/client/QuestDBBuilder.java	51	53	96.23%
🔵	io/questdb/client/cutlass/qwp/protocol/QwpTableBuffer.java	94	95	98.95%
🔵	io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java	19	19	100.00%
🔵	io/questdb/client/impl/ConfigSchema.java	3	3	100.00%
🔵	io/questdb/client/impl/SenderSlot.java	13	13	100.00%
🔵	io/questdb/client/cutlass/qwp/client/BatchTooLargeForCapException.java	5	5	100.00%
🔵	io/questdb/client/cutlass/qwp/client/QwpColumnWriter.java	7	7	100.00%
🔵	io/questdb/client/impl/QuestDBImpl.java	4	4	100.00%