Monday, March 19, 2018

Jboss / Wildfly errors when accepting socket java.io.IOException: Bad file descriptor problem

Jboss/Wildfly random crashes under heavy load:

Symptoms:

Jboss/Wildfly crashes randomly with  java.io.IOException: Bad file descriptor. Undertow HTTp server does not restart but admin console and everything else is working and nothing in log files.

How to debug it:

Put this in your configuration file (standalone.xml for Wildfly)
        <logger category="org.xnio.nio">
        <level name="DEBUG"/>
        </logger>
The error:
2018-03-03 17:19:15,273 DEBUG [org.xnio.nio.tcp.server] (default Accept) Exception accepting request, closing server channel TCP server (NIO) <69a407dd>: java.io.IOException: Bad file descriptor
    at sun.nio.ch.IOUtil.configureBlocking(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:264)
    at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:477)
    at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

What is the reason for that:

The reason is that you are probably running a BSD and the bug is in Java implementation of KQueueSelectorImpl.java in latest OpenJDK for BSD (openjdk-1.8.162 at the moment). This was tested on NetBSD 7.1.

Solution:

There is no solution at the moment. You need to move your Jboss/Wildfly to Linux. You can try changing default selector with java command line arguments (example is solution for Solaris):

java -Dxnio.nio.selector.provider=sun.nio.ch.PollSelectorProvider .......

At the moment the only solution to this is not to run it under BSD. Under Linux with EPollSelectorProvider everything works just fine.