Jboss / Wildfly errors when accepting socket java.io.IOException: Bad file descriptor problem

Jboss/Wildfly random crashes under heavy load:

Symptoms:

Jboss/Wildfly crashes randomly with  java.io.IOException: Bad file descriptor. Undertow HTTp server does not restart but admin console and everything else is working and nothing in log files.

How to debug it:

Put this in your configuration file (standalone.xml for Wildfly)

        <logger category="org.xnio.nio">
        <level name="DEBUG"/>
        </logger>

The error:

2018-03-03 17:19:15,273 DEBUG [org.xnio.nio.tcp.server] (default Accept) Exception accepting request, closing server channel TCP server (NIO) <69a407dd>: java.io.IOException: Bad file descriptor
    at sun.nio.ch.IOUtil.configureBlocking(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:264)
    at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:477)
    at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

What is the reason for that:

The reason is that you are probably running a BSD and the bug is in Java implementation of KQueueSelectorImpl.java in latest OpenJDK for BSD (openjdk-1.8.162 at the moment). This was tested on NetBSD 7.1.

Solution:

There is no solution at the moment. You need to move your Jboss/Wildfly to Linux. You can try changing default selector with java command line arguments (example is solution for Solaris):

java -Dxnio.nio.selector.provider=sun.nio.ch.PollSelectorProvider .......

At the moment the only solution to this is not to run it under BSD. Under Linux with EPollSelectorProvider everything works just fine.

How to filter 99.99% of ssh brute force attacks

Recently I've decided to experiment with ssh ciphers / key exchange algorithms to raise the security of my servers. This is the /etc/ssh/sshd_config I've got:

HostKey /etc/ssh/ssh_host_ed25519_key
Ciphers chacha20-poly1305@openssh.com
KexAlgorithms curve25519-sha256@libssh.org

If you don't have HostKey for Ed25519 generate it:

# ssh-keygen -t ed25519

You need to use recent version of ssh / pyTTY to be able to login to this server.
It seems that using only this Cipher/Kex filters all brute force scanners probably because they do not support it. I see only this kind of messages:

Feb 13 14:41:39 horizon9 sshd[22849]: SSH: Server;Ltype: Version;Remote: xxx.x.xx.xxx-53810;Protocol: 2.0;Client: libssh2_1.7.0
Feb 13 14:41:39 horizon9 sshd[22849]: fatal: ssh_dispatch_run_fatal: no matching cipher found [preauth]

More information about ciphers/algorithms read here:

https://cr.yp.to/ecdh.html#curve25519-paper
https://en.wikipedia.org/wiki/Salsa20
https://en.wikipedia.org/wiki/Poly1305

Multiboot NetBSD and Debian GNU/Linux with GRUB2

Debian GNU/Linux 8 Jessie Beta2 and NetBSD 7_BETA on the same pc.
 
Linux is on /dev/sda1
NetBSD partition is on /dev/sda4
Grub is installed on master boot record (/dev/sda).

Edit /etc/grub.d/40_custom and add the following lines:

menuentry "NetBSD 7 on sda4" {
        insmod ufs2
        insmod bsd
        set root=(hd0,4)
        chainloader (hd0,4)+1
}

Don't forget to do 'update-grub' before rebooting.

Qmail starting error "alert: cannot start: unable to open mutex"

If you are getting this error message when you try to start qmail-send - most probably you are moving qmail from old installation to a new server.

The problem comes from missing diretories (and files) which are on most distributions links and when you archive it the real directories will not be trasfered.

Directory is /var/qmail/queue and as you can see from the example bellow it is just a symlink to another directory (example is from NetBSD qmail package):

 # ls -la /var/qmail      
drwxr-xr-x   2 root  wheel  512 Sep 18 15:25 .
drwxr-xr-x  26 root  wheel  512 Aug 24 22:11 ..
lrwxr-xr-x   1 root  wheel   24 Jul 16 04:36 alias -> /usr/pkg/etc/qmail/alias
lrwxr-xr-x   1 root  wheel   12 Jul 16 04:36 bin -> /usr/pkg/bin
lrwxr-xr-x   1 root  wheel   34 Jul 16 04:36 boot -> /usr/pkg/share/examples/qmail/boot
lrwxr-xr-x   1 root  wheel   26 Jul 16 04:36 control -> /usr/pkg/etc/qmail/control
lrwxr-xr-x   1 root  wheel   24 Jul 16 04:36 doc -> /usr/pkg/share/doc/qmail
lrwxr-xr-x   1 root  wheel   12 Jul 16 04:36 man -> /usr/pkg/man
lrwxr-xr-x   1 root  wheel   16 Jul 16 04:36 queue -> /var/spool/qmail
lrwxr-xr-x   1 root  wheel   24 Jul 16 04:36 users -> /usr/pkg/etc/qmail/users

So the real directory is "queue -> /var/spool/qmail" and here is the listing of subdirs:

# ls -la
drwxr-xr-x  11 qmailq  qmail  512 Sep 18 15:34 .
drwxr-xr-x   8 root    wheel  512 Jul 16 04:36 ..
drwx------   2 qmails  qmail  512 Sep  9 15:11 bounce
drwx------  25 qmails  qmail  512 Jan 19  2008 info
drwx------   2 qmailq  qmail  512 Sep 18 13:55 intd
drwx------  25 qmails  qmail  512 Jan 19  2008 local
drwxr-x---   2 qmailq  qmail  512 Jan 19  2008 lock
drwxr-x---  25 qmailq  qmail  512 Jan 19  2008 mess
drwx------   2 qmailq  qmail  512 Sep 18 13:55 pid
drwx------  25 qmails  qmail  512 Jan 19  2008 remote
drwxr-x---   2 qmailq  qmail  512 Sep 18 13:55 todo

Make sure to transfer this directory to the new server and also chown the dirs with correct permissions.

The actual error is caused by missing file sendmutex which resides in /var/spool/qmail/lock/ directory

# ls -la /var/spool/qmail/lock
total 12
drwxr-x---   2 qmailq  qmail   512 Jan 19  2008 .
drwxr-xr-x  11 qmailq  qmail   512 Sep 18 15:34 ..
-rw-------   1 qmails  qmail     0 Jan 19  2008 sendmutex
-rw-r--r--   1 qmailr  qmail  1024 Sep 16 09:39 tcpto


If this file does not exist just create it with the following command (make sure also to change the file owner):

# touch /var/qmail/queue/lock/sendmutex
# chown qmails:qmail /var/qmail/queue/lock/sendmutex

NetBSD 6.1 filesystem notes.

The filesystem must be not mounted

root@farnsworth ~ # umount /disks/disk1
root@farnsworth ~ # umount /disks/disk2
root@farnsworth ~ # file -s /dev/wd1a
/dev/wd1a: Unix Fast File system [v1] (little-endian), last mounted on /disks/disk1, last written at Tue Jul 16 09:38:49 2013, clean flag 1, number of blocks 122096646, number of data blocks 121138363, number of cylinder groups 644, block size 32768, fragment size 4096, minimum percentage of free blocks 5, rotational delay 0ms, disk rotational speed 60rps, TIME optimization
root@farnsworth ~ # file -s /dev/wd2e
/dev/wd2e: Unix Fast File system [v1] (little-endian), last mounted on /disks/disk2, last written at Sat Jul 20 02:00:58 2013, clean flag 1, number of blocks 78142160, number of data blocks 77529094, number of cylinder groups 412, block size 32768, fragment size 4096, minimum percentage of free blocks 5, rotational delay 0ms, disk rotational speed 60rps, TIME optimization
root@farnsworth ~ #

Also you can do it with dumpfs:

root@farnsworth ~ # dumpfs -s /dev/wd1a
file system: /dev/rwd1a
format  FFSv1
endian  little-endian
magic   11954           time    Sat Jul 20 02:21:25 2013
superblock location     8192    id      [ 51e514a9 69cdbabc ]
cylgrp  dynamic inodes  4.4BSD  sblock  FFSv2   fslevel 4
nbfree  15140244        ndir    1       nifree  30334972        nffree  18
ncg     644     size    122096646       blocks  121138363
bsize   32768   shift   15      mask    0xffff8000
fsize   4096    shift   12      mask    0xfffff000
frag    8       shift   3       fsbtodb 3
bpg     23699   fpg     189592  ipg     47104
minfree 5%      optim   time    maxcontig 2     maxbpg  8192
symlinklen 60   contigsumsize 2
maxfilesize 0x004002001005ffff
nindir  8192    inopb   256
avgfilesize 16384       avgfpdir 64
sblkno  8       cblkno  16      iblkno  24      dblkno  1496
sbsize  4096    cgsize  32768
csaddr  1496    cssize  12288
cgrotor 0       fmod    0       ronly   0       clean   0x02
wapbl version 0x1       location 2      flags 0x0
wapbl loc0 488401024    loc1 131072     loc2 512        loc3 3
flags   wapbl
fsmnt   /disks/disk1
volname         swuid   0

Differences between FFSv1 and FFSv2

From newfs man page -O option:

0    4.3BSD; This option is primarily used to build root file systems that can be understood by older boot ROMs.  This generates an FFSv1 file system with level 1 format.
1    FFSv1; normal Fast File System, level 4 format. Also known as `FFS', `UFS', or `UFS1'.  This is the default.
2    FFSv2; enhanced Fast File System, suited for more than 1 Terabyte capacity.  This is also known as `UFS2'.

from fsck_ffs man page:

FFS1 level 0 = inode 4.2/4.3BSD static table
FFS1 level 1 = dynamic table
FFS1 level 2 = 32bit UID/GID, compact symlinks
FFS1 level 3 = free segment maps
FFS1 level 4 = FFS2 style superblock (allows WAPBL)
FFS2 level 5 = 64bit addresses, 64bit timestamps, birthtime, ext attributes 

FFSv2 is suitable for large disk (1TB)

Here is example how to change default percentage of disk space held back from normal users to 1%. Default is 5% and this is too much for large disks.

root@farnsworth ~ # tunefs -m 1 /dev/wd1a
tunefs: tuning /dev/rwd1a
tunefs: minimum percentage of free space changes from 5% to 1%
tunefs: should optimize for space with minfree < 5%

String replacing in VIM.

To replace a string with vim do the following:

:%s/string/replace string/g

% - means search and replace in whole file
If you want to replace stuff in certain portion of the file you can do following:

:20,50 s/string/replace string/g

:20,50 will search and replace string in rows between 20 and 50

Escaping special chars with \. Look at following example. I need to replace /usr/src/ with /var/log/archive in one of my scripts:

:%s/\/usr\/src/\/var\/log\/archive/g

Fix console font on NetBSD 5.1/macppc

One of annoying things about booting NetBSD 5.1 on PowerMAC G4 after install is console font which is too big to fit in screen. Today I found solution on port-macppc mailing list. You can see it here http://mail-index.netbsd.org/port-macppc/2011/10/18/msg001479.html.

Edit your kernel config file to look like this:

#options    FONT_GALLANT12x22
options       FONT_VT8x8

then recompile and reboot.

Installing NetBSD 5.1 on PowerMAC G4

I've tried to install NetBSD 5.1 on PowerMAC G4 (grey) following these instructions: http://wiki-static.aydogan.net/How_to_install_NetBSD_on_a_Power_Macintosh_G4_Grey but I had problems booting.

Here are some tips.
It seems that netbsd-INSTALL.gz image which comes from 5.1 installation CD is ok but netbsd-GENERIC.gz and others were not tested so they crashed when I rebooted after installation. So here what I've done to solve this after installation is done but can not boot to netbsd:

Hold Alt+Win+o+f when starting PowerMAC to enter in OpenFirmware cmd line menu.
Boot with install netbsd image. Something like that:

0> boot cd:,\ofwboot.xcf \macppc\binary\kernel\netbsd-INSTALL.gz

When it boots you will be prompted for (I)nstall (S)hell ... choose 'S;.
Now from shell mount your root partition and chroot to it and set up your internet connection:

# mount /dev/wd0a /mnt2
# chroot /mnt2
# ifconfig gem0 192.168.1.10 255.255.255.0 up
# route add default 192.168.1.1
# echo "nameserver 192.168.1.1" > /etc/resolv.conf

Now you can download and compile your own kernel. Get syssrc.tgz from ftp.netbsd.org untar it and go to sys/arch/macppc/conf. Use GENERIC config as if you try to compile POWERMAC_G5 or some of the others result will be the same - kernel crash on next reboot.

# cp GENERIC G4
# vi G4
# config G4
# cd ../compile/G4
# make depend
# make
# cp /netbsd /netbsd.old
# cp netbsd /netbsd

If you want your Xorg server to work you need to add these lines to kernel config file and recompile it:

options WSFB_FAKE_VGA_FB
options WSDISPLAY_COMPAT_PCVT # emulate some ioctls
options WSDISPLAY_COMPAT_SYSCONS # emulate some ioctls
options WSDISPLAY_COMPAT_USL # VT handling
options WSDISPLAY_COMPAT_RAWKBD # can get raw scancodes

Now you just use: X -configure and then test your config with: X -config xorg.conf.new