Skip to content

fix(NODE-7894): reject pending socket writes when the connection closes - #5077

Open
philkunz wants to merge 1 commit into
mongodb:mainfrom
philkunz:fix/connection-pending-write-on-close
Open

philkunz wants to merge 1 commit into
mongodb:mainfrom
philkunz:fix/connection-pending-write-on-close

Conversation

@philkunz

Copy link
Copy Markdown

Description

Summary of Changes

A command written to a socket that is closing could wait for drain forever. This change makes that write settle with the connection's error:

  • Connection.cleanup() now rejects a write that is waiting for drain, with the same error it already passes to a pending read.
  • writeCommand() checks the connection's error after the command is serialized and before it writes. If the connection closed during serialization, the command fails instead of writing to a destroyed socket.
  • Two unit tests use a plain net server that ends every connection it accepts. Each test pins one of the two changes, and both fail on main without this change.
Notes for Reviewers

GitHub issues are disabled for this repository, and we could not open a NODE Jira ticket. This PR therefore includes the full reproduction below. Please associate a NODE ticket and add its scope to the title if required.

Why the write never settles (checked on Node.js v25.9.0):

  1. The peer ends the connection. With allowHalfOpen false, Node ends the client socket and replaces socket.write with an override for writes after FIN.
  2. The socket is destroyed before close is observed. After finish, the socket is auto-destroyed in the same turn. The close event follows only in a later event-loop iteration, after the handle's close callback. In that window connection.closed is still false, so the pool can still hand the connection out, and client.close() can still send endSessions or abortTransaction on it.
  3. The write waits for an event that never comes. A write in that window returns false. The EPIPE goes only to the write callback, which the driver does not pass, and destroy(err) is a no-op on a destroyed socket. Neither drain nor error is emitted, so once(socket, 'drain') never settles.
  4. Nothing else settles it. When close finally arrives, cleanup() settles only dataEvents. The read loop has not started yet because writeCommand() is still awaiting drain. socketTimeoutMS cannot fire on a destroyed socket. timeoutForSocketWrite bounds the wait only when timeoutMS is set.

rejectPendingWrite is a single field, like dataEvents, because a connection has at most one command in flight. The finally block clears it, and Promise.race already handles a rejection that arrives after drain won.

No dependencies, package metadata, build scripts, generated files or vendored specification fixtures are changed.

What is the motivation for this change?

When a server closes pooled connections (for example, on shutdown or failover), a command sent in the window between the socket's finish and close events hangs forever. This includes the commands MongoClient.close() sends, so client.close() can hang too. The window is short. It is easy to hit on macOS, where kqueue can deliver the peer's FIN and the application's next step in the same event-loop turn, and it can be reproduced deterministically on any platform.

For bug fixes

Current (incorrect) behavior:

A command written after the socket's finish event, but before its close event, never resolves or rejects. When close arrives, the connection is marked closed, but the command stays pending. MongoClient.close() hangs if it sends a command on such a connection.

Expected behavior:

The command rejects with a MongoNetworkError once the connection closes, and MongoClient.close() settles.

How to reproduce:

Self-contained CommonJS script. It needs no MongoDB server, only a plain net server that ends every connection:

const { once } = require('node:events');
const net = require('node:net');
const { Connection } = require('mongodb/lib/cmap/connection');
const { ns } = require('mongodb/lib/utils');

async function main() {
  // a server that closes every connection it accepts
  const server = net.createServer(socket => socket.end());
  server.listen(0, '127.0.0.1');
  await once(server, 'listening');

  const socket = net.createConnection(server.address().port, '127.0.0.1');
  const connection = new Connection(socket, {});
  const outcome = await new Promise(resolve => {
    socket.once('finish', () => {
      // the socket has ended and is being destroyed, but 'close' has not been emitted yet
      connection.command(ns('admin.$cmd'), { ping: 1 }, {}).then(
        () => resolve('resolved'),
        error => resolve(`rejected with ${error.name}: ${error.message}`)
      );
    });
    setTimeout(() => resolve('still pending after 5 seconds'), 5000);
  });

  console.log(outcome);
  server.close();
}

main();

Results:

  • mongodb@7.5.0 and mongodb@7.7.0 print still pending after 5 seconds.
  • With this change, the script prints rejected with MongoNetworkError: connection ... closed.

The same window is reachable through the public API. In a test that triggers client.close() from the pooled socket's finish event while the server shuts down, client.close() stays pending until the process exits. With this change it settles.

The new unit tests in test/unit/cmap/connection.test.ts cover both paths:

  • after the socket finished: the command is written after the socket's finish event but before its close event was observed.
  • destroyed before the write: the connection is destroyed while the command is still being serialized.

Affected versions:

  • Reproduced with mongodb@7.5.0 and mongodb@7.7.0, and on main at 16136bb, with Node.js v25.9.0 on Linux.
  • First observed on macOS.
  • Server version and topology do not matter, because the reproduction needs only a TCP peer that closes the connection.

Double check the following

  • Lint is passing (npm run check:lint): its steps build:dts, check:dts, check:eslint and check:tsd pass, and the unit suite passes
  • Self-review completed using the steps outlined here
  • PR title follows the correct format: type(NODE-xxxx)[!]: description (no NODE ticket could be opened, see above)
  • Changes are covered by tests
  • New TODOs have a related JIRA ticket (no new TODOs)

A command written after the socket's finish event, but before its close
event, could wait for drain forever. The peer had already ended the
socket, so the write returned false and the socket was destroyed without
emitting drain or error. Connection cleanup settled only pending reads,
and socketTimeoutMS cannot fire on a destroyed socket.

Connection cleanup now rejects a write that is waiting for drain with the
connection error, and writeCommand checks the connection error before it
writes, so a connection closed during serialization fails the command.
@philkunz
philkunz requested a review from a team as a code owner October 10, 2026 10:07
@PavelSafronov PavelSafronov changed the title fix: reject pending socket writes when the connection closes fix(NODE-7894): reject pending socket writes when the connection closes Oct 12, 2026
@PavelSafronov PavelSafronov added tracked-in-jira Ticket filed in MongoDB's Jira system External Submission PR submitted from outside the team labels Oct 12, 2026
@PavelSafronov

Copy link
Copy Markdown
Contributor

@philkunz , thanks for (yet another!) detailed write-up and repro! I’ve linked this to NODE-7894.
We're going to triage this work at the next opportunity and will keep you updated.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

External Submission PR submitted from outside the team tracked-in-jira Ticket filed in MongoDB's Jira system

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants