Jboss/Wildfly random crashes under heavy load:
Symptoms:
Jboss/Wildfly crashes randomly with java.io.IOException: Bad file descriptor. Undertow HTTp server does not restart but admin console and everything else is working and nothing in log files.
How to debug it:
Put this in your configuration file (standalone.xml for Wildfly)
<logger category="org.xnio.nio">
<level name="DEBUG"/>
</logger>
The error:
2018-03-03 17:19:15,273 DEBUG [org.xnio.nio.tcp.server] (default Accept) Exception accepting request, closing server channel TCP server (NIO) <69a407dd>: java.io.IOException: Bad file descriptor
at sun.nio.ch.IOUtil.configureBlocking(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:264)
at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:477)
at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
What is the reason for that:
The reason is that you are probably running a BSD and the bug is in Java implementation of KQueueSelectorImpl.java in latest OpenJDK for BSD (openjdk-1.8.162 at the moment). This was tested on NetBSD 7.1.
Solution:
There is no solution at the moment. You need to move your Jboss/Wildfly to Linux. You can try changing default selector with java command line arguments (example is
solution for Solaris):
java -Dxnio.nio.selector.provider=sun.nio.ch.PollSelectorProvider .......
At the moment the only solution to this is not to run it under BSD. Under Linux with EPollSelectorProvider everything works just fine.