storage-cassandra: cdrs-tokio drops its only connection under sustained load and never reconnects

What happened

Running the full Cassandra pytest suite (devtools/run-cassandra-tests -- --pytest --comprehensive --parallel, 2 workers) against a single-node cassandra:4.1 with a 60,000-item backup-restore fixture in the mix, the server lost its connection to Cassandra about 25 minutes in and never recovered. Every request after that failed with No nodes available in query plan! (surfaced to clients as InternalServerError, including Internal error during authentication), until the process was restarted. 90 tests failed and 314 errored in that run.

Server log at the moment it happened:

ERROR extenddb_engine::create_table: storage internal error internal_error=Internal error: Failed to drop data table: General error: Connection closed while waiting for response!
ERROR cdrs_tokio::transport: Transport error! error=General error: Unmatched stream id: 7914
WARN  cdrs_tokio::cluster::connection_pool: Connection down. Starting reconnection. broadcast_rpc_address=Some(240.10.0.6:7000)
ERROR extenddb_storage_cassandra::cassandra_util: account_exists query failed: General error: No nodes available in query plan!
... (every request from here on)

Reading of the driver (cdrs-tokio 9.0.2, `src/transport.rs`)

ResponseHandlerMap::next_stream_id hands out ids 1..=i16::MAX and wraps to 1 without checking that the id's previous request has been answered. A slow request (here, a DROP TABLE under load) can still be outstanding when its id is reused; when the late response arrives the handler is already gone, send_response returns Unmatched stream id, the ? in start_reading_normal_frames ends the read loop, and the connection is closed. The reconnection then targets broadcast_rpc_address (240.10.0.6:7000 in this container), not the contact point the server was configured with, and never succeeds, so the pool stays empty.

This is from reading the code and the log, not a reproduction in isolation.

Why it matters

A busy production server with one Cassandra node (or one connection per node) can hit this after ~32k requests with any one slow statement in flight, and then serves nothing until restarted.

Options

Open more than one connection per node (cdrs-tokio supports a per-node pool size), which divides the id space.
Make next_stream_id skip ids that still have a handler.
Reconnect to the configured contact point rather than broadcast_rpc_address, or make reconnection retry.
Upstream fix or driver change.

Workaround in #384

devtools/run-cassandra-tests now sets EXTENDDB_TEST_BACKUP_ITEMS=10000 unless it is already set, which keeps the test suite under the threshold. Nothing changes for production.

#457

While verifying #527 on a single cassandra:4.1 node: after a docker restart of the node mid-restore, the server's cdrs-tokio session did not come back once the node accepted CQL again (about 50 s later). Every request for the rest of a 240 s window answered InternalServerError (Internal error during authentication, No nodes available in query plan!), and the restore's own cleanup (abandon_restore_target) and the in-process restore sweep could not run, so the failed target stayed CREATING until the server was restarted, whose startup sweep then removed it in 4 s. Same session, same function on main; noted here since the reconnect behaviour is what this issue and #457 are about. Measurement details are in the #527 description under Recovery.

I traced the restart case from my last comment while reviewing #457. It's in cdrs-tokio's metadata refresh, and it happens on main as well as on #457.

After the node comes back, the control connection reconnects and refreshes the cluster metadata. On an incremental refresh the driver rebuilds each node with Node::clone_with_node_info (metadata_builder.rs:78-81 in 9.0.2), which marks the address as changed when the old gossip address differs from the new client address (node.rs:280-289). On Cassandra 4.x the gossip address is broadcast_address plus broadcast_port from system.local, here <container ip>:7000, and the client address is the contact point, 127.0.0.1:<port>. They never match, so the node is rebuilt as Unknown with no distance and an empty pool. The load balancer skips it from then on, and on a single node nothing moves it back because the node sends no status event about itself. Every query after that gets No nodes available in query plan!.

A full refresh doesn't have this problem, it copies the contact point and keeps it Up (metadata_builder.rs:20-28). The incremental path runs when the control connection reconnects while the node is still in the plan. Behind a published Docker port that's the usual case. Docker's proxy accepts the TCP connection while the node's native transport is down and then closes it, so a failed dial leaves a stale error in the control connection's one-slot error channel (transport.rs:394-405, control_connection.rs:43). When the control connection gets back and does a full refresh, it reads that stale error, drops the good connection, reconnects, and the second refresh is incremental:

23:01:21.095 connection_pool: All connections reestablished.
23:01:21.658 control_connection: Established new control connection.
23:01:21.662 metadata_builder: Copying contact point.
23:01:21.663 control_connection: Error in control connection! Trying to reconnect. error=IO error: Connection reset by peer (os error 104)
23:01:21.664 control_connection: Established new control connection.
23:01:21.667 metadata_builder: Updating old node.

Queries succeed from 21.095 and fail from 21.667 to the end of the run. There's a second trigger that needs no stale error. If the pool reconnects first and marks the node Up, the control connection's next refresh is incremental too.

Measured on one cassandra:4.1 node with nodetool disablebinary, 4s, then enablebinary, counting a run as recovered when queries then succeed for 30s without a failure:

Behind a published port, which is how CI and devtools/run-cassandra-tests expose the node, it recovered in 1 of 10 runs on fix(storage-cassandra): apply max_connections and override CDRS default reconnect policy #457's head and 1 of 10 on main. In most of the other runs the session ended on No nodes available in query plan! and didn't recover within 3 minutes. With docker restart instead it was 2 of 10 and 3 of 10.
With the contact point set to the container's own address, head recovered in 7 of 7 runs, in 0.1 to 4.8s, and main in 2 of 3.

The reconnection policy only changes how far apart the attempts are, so #457's 5s cap doesn't change this. 9.0.2 is the latest release and master has the same comparison, so it needs a driver patch or a workaround on our side. Comparing the old client address with the new one, instead of the gossip address, looks like the fix in the driver, but I haven't tried it.

I haven't reproduced the sustained-load case from the original report. Any reconnect of the control connection while the node is in the plan goes down the same path, so I expect it's the same cause. From the code, the comparison also fails without NAT, since the gossip port is 7000 and the client port is 9042, but I've only run it against a single node in Docker.

fix(storage-cassandra): apply max_connections and cap the reconnect d…

@LeeroyHannigan

Following up on @LeeroyHannigan's trace above. The root cause is in cdrs-tokio 9.0.2, Node::clone_with_node_info (cluster/topology/node.rs:280-282). It compares the old node's broadcast_address (the gossip address, <container ip>:7000) with the new node's broadcast_rpc_address (the client address, 127.0.0.1:9042). Those are two different addresses of the same node, so they never match. The incremental metadata refresh then rebuilds the node as Unknown, with an empty pool. On a single node, nothing marks it Up again, so every query after that fails with No nodes available in query plan!.

I re-ran Lee's measurement on #457's head: a single cassandra:4.1 node behind a published port, nodetool disablebinary, 4s, enablebinary. A run counts as recovered only after 30s with no failed query. The first query came back in 0.72 to 1.01s, but 0 of 4 runs recovered. Every run ended on No nodes available in query plan!.

The real fix is in the driver: compare the old broadcast_rpc_address with the new one. Until that lands, we have two options on our side.

Option 1: change how Docker exposes the node

The common trigger is Docker's userland port proxy. It accepts a TCP connection while the node's native transport is down, then closes it. That leaves a stale error in the control connection, which forces the incremental refresh. Two ways to avoid the proxy:

Change	What it does	Limits
Contact point = the container's own IP	the driver talks to the node directly, with no proxy	works only from the same host. Lee saw 7 of 7 runs recover this way.
`"userland-proxy": false` in `/etc/docker/daemon.json`, or `--network host`	a dead port refuses the connection instead of accepting it, so no stale error	`userland-proxy` is daemon-wide, so it affects every container on the host and needs a `dockerd` restart. `--network host` is Linux only. Not measured yet.

These avoid the common trigger only. Lee found a second trigger that needs no proxy: when the pool reconnects first and marks the node Up, the control connection's next refresh is incremental too. So option 1 makes the hang rare in dev and CI, but it doesn't remove it.

Option 2: rebuild the session on `No nodes available in query plan!`

This covers both triggers, and real deployments as well as Docker. The downsides:

Sharing. The session is an Arc held in about 56 places in storage-cassandra, including background workers and migration code. A rebuild helps only if every holder picks up the new session. That needs a swap point (for example arc-swap) instead of plain Arc clones. Most of the work is here.
Failed requests. At least one request fails before the rebuild runs, unless the failing call is retried after the rebuild.
Wrong trigger. The same error also means the cluster really is unreachable. A rebuild then fails again, so it needs a rate limit (for example one rebuild per 5 to 10s) to avoid a rebuild loop.
Cost. A new session reconnects, re-reads cluster metadata, and opens max_connections connections per node. The old session must be dropped cleanly, or its sockets and tasks leak.
In-flight work. Requests already running on the old session can fail during the swap.
It hides the bug. Each rebuild should be logged, so we can see how often it fires and tell when a driver fix makes it unnecessary.

ExtendDB doesn't cache prepared statements in this crate, so a rebuild has nothing to re-prepare.

Neither option has been built or measured yet, apart from Lee's container-IP runs.

krojew/cdrs-tokio#231 - Issue opened to cdrs-tokio regarding broadcast address comparison.

fix(storage-cassandra): apply max_connections and cap the reconnect d…

Navigation Menu

Repository navigation

storage-cassandra: cdrs-tokio drops its only connection under sustained load and never reconnects #403

What happened

Reading of the driver (cdrs-tokio 9.0.2, `src/transport.rs`)

Why it matters

Options

Workaround in #384

Option 1: change how Docker exposes the node

Option 2: rebuild the session on `No nodes available in query plan!`

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

storage-cassandra: cdrs-tokio drops its only connection under sustained load and never reconnects #403

Description

What happened

Reading of the driver (cdrs-tokio 9.0.2, src/transport.rs)

Why it matters

Options

Workaround in #384

Activity

jcshepherd commented on Oct 23, 2026

LeeroyHannigan commented on Oct 24, 2026

LeeroyHannigan commented on Oct 28, 2026

jcshepherd commented on Nov 3, 2026

Option 1: change how Docker exposes the node

Option 2: rebuild the session on No nodes available in query plan!

jcshepherd commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Reading of the driver (cdrs-tokio 9.0.2, `src/transport.rs`)

Option 2: rebuild the session on `No nodes available in query plan!`