HTTP Server closed connections keep being in CLOSED_WAIT state

[jgaalen]

Expected behavior

If the server closes an HTTP connection, jmeter should handle the closed connection immediately and not keeping it in CLOSE_WAIT state until the next request, which could also lead to "Non HTTP response" errors

Actual behavior

Currently, when JMeter does an HTTP request and is in keep-alive state and server waits for the next request. If the server closes the connection after X-seconds and sends a FIN/ACK packet, JMeter does nothing. If the next request is done within 5 seconds after the FIN/ACK packet, it sets a "Non HTTP response message: : failed to respond" error immediately, because the socket is already in CLOSED state and can't pick up new packets. This results in this error.

If it tries to send a request later than 5 seconds, somehow it gracefully sends a FIN/ACK back and creates a new connection and does the request.

Steps to reproduce the problem

Set keep-alive timeout to 2 seconds on the server, so it shuts down the connection after 2 seconds.

Then create a jmeter script to send request 1, then wait 3 seconds, then send request 2. It will end up with the Non HTTP response error, because it tries to send the request on a closed socket.
Now wait 6 seconds rather than 3, and somehow it does work without an error.

JMeter Version

5.6.3

Java Version

17

OS Version

Mac + Ubuntu

[jgaalen]

I've done some more testing on this. It doesn't turn into a non-HTTP error against every server so I have no idea why that happens with one server I'm using, but the problem remains. On many servers somehow, after the FIN_ACK is sent and jmeter leaves it in CLOSE_WAIT state, when JMeter wants to do a new request, it somehow first sends the FIN_ACK back which cleans the connection and then does the request. Against some servers it doesn't send the FIN_ACK when it wants to do a new request, but just sends it over the connection and the server sends a RST because the connection is either cleaned up or it's in CLOSE_WAIT state.

It is a serious issue, because once the server tries to disconnect the connection (after its keep-alive timeout), the connection remains in CLOSE_WAIT state on both the webserver and jmeter client and still taking/tracking a socket which should not be there. Browsers clean up the socket immediately, so it's not there anymore on both the client and the server which is the behaviour we need to see in JMeter

[jgaalen]

I've made a setup with basic nginx, keep-alive timeout changed to 2 seconds. Then a script that does a request (with keep-alive), then 4 seconds later, an new requests. The 2nd requests fails due to this RST and connection in CLOSE_WAIT state.

It only happens with the HttpClient4 implementation. The Java implementation has no issues

[jgaalen]

So I've found the root cause of the issue and it's due to this config httpclient4.validate_after_inactivity.

This defaults to 4900 (ms), meaning that if a connection trying to be disconnected by the server within 4900ms, and a new request is done between the requested disconnect and 4900ms, it just sends the request packet over the connection in CLOSE_WAIT state, resulting in a connection reset error.

I think this is a bad implementation in the HttpClient4 library. It should simply remove a connection once the server requests it.

A work-around is to set httpclient4.validate_after_inactivity low (900). This only resolves the unexpected "Non HTTP response" errors, but doesn't resolve the high CLOSE_WAIT state taking unrealistic higher connections on jmeter clients and left FIN_WAIT_2 state on webservers

[Lonzak]

I experienced the same issue - thanks for posting here!
Should a ticket be added for this? Decreasing the validation period helps but as you mentioned doesn't resolve the high CLOSE_WAIT states etc.

[jgaalen]

I experienced the same issue - thanks for posting here! Should a ticket be added for this? Decreasing the validation period helps but as you mentioned doesn't resolve the high CLOSE_WAIT states etc.

Well I created a ticket with this right? But indeed, it should actually be resolved and behave like a browser, or any http client. So it should respond to the FIN packet and close the socket on the jmeter side and send a FIN back, so the server can remove the half closed state as well

[Lonzak]

Ah you are right - I thought it is still the bugzilla tracker... But since 2022 it was migrated here. So everything is fine :-)

[asenecal2keys]

Great find, I've been suffering from this issue as well. From the above it sounds like an issue was also filed with Apache? Would you mind providing the link, I've been searching in the Apache Jira without success .

[jgaalen]

Great find, I've been suffering from this issue as well. From the above it sounds like an issue was also filed with Apache? Would you mind providing the link, I've been searching in the Apache Jira without success .

No, the only issue, is this issue here. But as it's a httpclient bug I think it should be reported there I guess

[kapiljain786]

any update on this ticket ?

[jgaalen]

any update on this ticket ?

Sorry, after analysing and putting everything on a plate, nothing happens. Not sure how big the jmeter team is, but I've made some interesting proposals and even PRs, but nothing is happening. Last JMeter update was more than year ago, so it feels like a dead project lately.

[vlsi]

If the server closes an HTTP connection, jmeter should handle the closed connection immediately and not keeping it in CLOSE_WAIT state until the next request, which could also lead to "Non HTTP response" errors

Well, could you please clarify why would you expect exactly this behavior?
As far as I understand, It is pretty much fine for the "server" to close a connection while client could still send some data.

In other words, it means "server is done sendings its bytes", however, it is fine for a TCP connection to be in half-closed state, and the client can still send data and the server could receive it.

Currently, when JMeter does an HTTP request and is in keep-alive state and server waits for the next request. If the server closes the connection after X-seconds and sends a FIN/ACK packet, JMeter does nothing. If the next request is done within 5 seconds after the FIN/ACK packet, it sets a "Non HTTP response message: : failed to respond" error immediately, because the socket is already in CLOSED state and can't pick up new packets. This results in this error.

Well, this boils down to the expected outcome of your test.

In practice, people use JMeter to simulate actual applications. So you should configure JMeter exactly the same as the application/microservice/browser you try to impersonate.

For instance, if the application/microservice/browser does not expect the server to close a connection shortly, then the application would fall into the same issue of trying to send data over a broken connection. In other words, "Non HTTP response message: : failed to respond" surfaces a configuration error (assuming you've configured JMeter the same as your app).

If the application performs connection re-validation, you should configure JMeter to do so.

Does that make sense?

but doesn't resolve the high CLOSE_WAIT state taking unrealistic higher connections on jmeter clients and left FIN_WAIT_2 state on webservers

If your webserver closes connections immediately while JMeter trying to keep them alive, I expect it might be the following:
a) It might be a true configuration bug discovered by JMeter. In other words, you will have "many CLOSE_WAIT connections" in production if clients attempt to use keepalive while the server closes the connections early
b) If the clients do not use keepalive in production, you should disable keepalives in JMeter as well

WDYT?

[asenecal2keys]

This appears as a critical bug for my use case as well.
The same test script running against the same application under test will run great with JMeter 5.5 but will run grind to halt on JMeter 5.6.3 on account of timeouts due to these CLOSED_WAIT.
This, in concert with the transaction controller memory leak issue, has forced us to downgrade to JMeter 5.5.

[jgaalen]

This appears as a critical bug for my use case as well. The same test script running against the same application under test will run great with JMeter 5.5 but will run grind to halt on JMeter 5.6.3 on account of timeouts due to these CLOSED_WAIT. This, in concert with the transaction controller memory leak issue, has forced us to downgrade to JMeter 5.5.

For the transaction controller, the fix is either: don't enable 'Generate parent sample', or make a build of this PR: #6386

Regarding close_wait, it doesn't fix the server side, but it helps on the errors, to lower httpclient4.validate_after_inactivity to 900ms for example.