Jboss / Wildfly errors when accepting socket java.io.IOException: Bad file descriptor problem

Jboss/Wildfly random crashes under heavy load:

Symptoms:

Jboss/Wildfly crashes randomly with  java.io.IOException: Bad file descriptor. Undertow HTTp server does not restart but admin console and everything else is working and nothing in log files.

How to debug it:

Put this in your configuration file (standalone.xml for Wildfly)

        <logger category="org.xnio.nio">
        <level name="DEBUG"/>
        </logger>

The error:

2018-03-03 17:19:15,273 DEBUG [org.xnio.nio.tcp.server] (default Accept) Exception accepting request, closing server channel TCP server (NIO) <69a407dd>: java.io.IOException: Bad file descriptor
    at sun.nio.ch.IOUtil.configureBlocking(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:264)
    at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:477)
    at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

What is the reason for that:

The reason is that you are probably running a BSD and the bug is in Java implementation of KQueueSelectorImpl.java in latest OpenJDK for BSD (openjdk-1.8.162 at the moment). This was tested on NetBSD 7.1.

Solution:

There is no solution at the moment. You need to move your Jboss/Wildfly to Linux. You can try changing default selector with java command line arguments (example is solution for Solaris):

java -Dxnio.nio.selector.provider=sun.nio.ch.PollSelectorProvider .......

At the moment the only solution to this is not to run it under BSD. Under Linux with EPollSelectorProvider everything works just fine.

Running JBOSS with Daemontools / Runit

JBOSS is installed in /usr/local/jboss
Jboss user is created for that purpose with home directory = /usr/local/jboss

daemontools/runit run script for jboss server:

#!/bin/sh

# clean the working dirs
/bin/rm -fr /usr/local/jboss/server/app1/work
/bin/rm -fr /usr/local/jboss/server/app1/tmp

JBOSS_HOST="10.10.0.25"

# next lines are fix for cyrillic letters inserted in database
# with AMERICAN_AMERICA.WE8ISO8859P1
NLS_LANG="AMERICAN_AMERICA.WE8ISO8859P1"
export LNS_LANG
LANG="bg_BG"
export LANG

exec /usr/bin/chpst -u jboss:jboss -U jboss:jboss /usr/lib/jvm/java-1.7.0-openjdk-i386/bin/java -Xms512m -Xmx1536m -XX:PermSize=256m -XX:MaxPermSize=512 -XX:+CMSClassUnloadingEnabled -XX:StackShadowPages=20 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=30 -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/usr/local/jboss/lib/endorsed -classpath /usr/local/jboss/bin/run.jar:/usr/lib/jvm/java-1.7.0-openjdk-i386/lib/tools.jar org.jboss.Main -c app1 -b $JBOSS_HOST

daemontools/runit script for logging:

#!/bin/sh
exec setuidgid jboss /usr/bin/svlogd -tt n35 s511048576 /var/log/jboss-app1