[Gargi-jais11]

What changes were proposed in this pull request?

When accept() itself fails in XceiverServerDomainSocket.run, the loop logs a WARN and calls accept() again right away. For example, accept(2) fails with "Too many open files" when the datanode is out of file descriptors. On Linux, the queued connection keeps failing, so the accept thread spins on a CPU core and logs thousands of WARN stack traces per second until descriptors are free again.

Changes done:
This change adds a 1-second pause before retrying when accept() itself fails, which is when no peer has been created yet. Rejections of accepted connections, such as at the xceiver limit, and idle accept timeouts behave as before. Shutdown isn't delayed, because stop() interrupts the pause.

KEPT in draft to save CI resources, will mark it ready to review once approved

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17436

How was this patch tested?

added testAcceptFailureIsRetriedAfterPause to TestXceiverServerDomainSocket. Without the fix it fails with about 14,600 accept() calls in 500 ms; with the fix it passes.
Without fix: (added few lines from test runs)

2026-10-27 11:02:22,680 [627313eb-83af-4e95-97d1-b2a5308e15c2-XceiverServerDomainSocket] WARN  server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null
java.net.SocketException: accept(2) error: Too many open files
	at org.apache.hadoop.hdds.TestXceiverServerDomainSocket.lambda$testAcceptFailureIsRetriedAfterPause$1(TestXceiverServerDomainSocket.java:647)
	at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:42)
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:103)
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134)
	at org.apache.hadoop.net.unix.DomainSocket.accept(DomainSocket.java:234)
	at org.apache.hadoop.ozone.container.common.transport.server.XceiverServerDomainSocket.run(XceiverServerDomainSocket.java:210)
	at java.base/java.lang.Thread.run(Thread.java:1583)
2026-10-27 11:02:22,680 [627313eb-83af-4e95-97d1-b2a5308e15c2-XceiverServerDomainSocket] WARN  server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null
java.net.SocketException: accept(2) error: Too many open files
	at org.apache.hadoop.hdds.TestXceiverServerDomainSocket.lambda$testAcceptFailureIsRetriedAfterPause$1(TestXceiverServerDomainSocket.java:647)
	at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:42)
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:103)
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134)
	at org.apache.hadoop.net.unix.DomainSocket.accept(DomainSocket.java:234)
	at org.apache.hadoop.ozone.container.common.transport.server.XceiverServerDomainSocket.run(XceiverServerDomainSocket.java:210)
	at java.base/java.lang.Thread.run(Thread.java:1583)
2026-10-27 11:02:22,680 [627313eb-83af-4e95-97d1-b2a5308e15c2-XceiverServerDomainSocket] WARN  server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null
java.net.SocketException: accept(2) error: Too many open files
.
.
.
org.opentest4j.AssertionFailedError: accept() was called 19061 times ==> 
Expected :true
Actual   :false

After fix:

2026-10-27 10:51:26,895 [b03e6da0-e68c-47b9-ae32-6ce2bda71c99-XceiverServerDomainSocket] WARN  server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null
java.net.SocketException: accept(2) error: Too many open files
	at org.apache.hadoop.hdds.TestXceiverServerDomainSocket.lambda$testAcceptFailureIsRetriedAfterPause$1(TestXceiverServerDomainSocket.java:647)
	at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:42)
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:103)
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134)
	at org.apache.hadoop.net.unix.DomainSocket.accept(DomainSocket.java:234)
	at org.apache.hadoop.ozone.container.common.transport.server.XceiverServerDomainSocket.run(XceiverServerDomainSocket.java:210)
	at java.base/java.lang.Thread.run(Thread.java:1583)

[Gargi-jais11]

@ChenSammi this is also ready for review. Once you approve I will mark it ready to review so that CI run just once.

[yandrey321]

should it be configurable? 1 second is a long wait...

[Gargi-jais11]

@yandrey321 thes pause only happens when accept() itself throws, suppose when the datanode is out of file descriptors. Normal accepts, idle timeouts, and rejected connections are not delayed. In that state a retry fails again almost at once: without the pause the test saw about 14,600 accept() calls in 500 ms. One second matches the fixed 30-second pause the same loop already uses after an OutOfMemoryError, and Hadoop's DataXceiverServer uses a similar fixed backoff.

@ChenSammi, what do you prefer? I can keep the fixed 1 second, 2.) use a shorter fixed value such as 100 ms, or add a datanode config key for it. A config key seems like more than this case needs, but I'm happy to add one if you think it's useful.