Thursday, November 21, 2019

Novell OES 11.1 shell commands hang without reason

Novell OES 11 linux shell commands hang for a very long time. It happened to me and I was able to fix it without restarting the whole server.

Server release info:
Novell Open Enterprise Server 11 (x86_64)
VERSION = 11.1
PATCHLEVEL = 1
Symptoms:
# strace -p26881
Process 26881 attached - interrupt to quit
connect(146, {sa_family=AF_FILE, path="/var/run/novell-lum/.nam_nss_sock"}, 35^C <unfinished ...>
All http services that are not using linux authentiction are working - iManager works, iMonitor works, Remote Manager is not working because it is for controlling linux services and require linux authentication.

It seems that something went wrong with namcd service. Most linux commands (like cron, id, ssh...) are trying to check through that socket for a user FDN from eDirectory. Linux User Managment (LUM) maps linux users to eDirectory users and every linux program executed is asking namcd (eDirectory Novell Account Management caching daemon) for information about current user. If namcd is not working it will just use local linux user db for that and if is working correctly you can check every user in eDir like this:

Note: admin is not local linux user but eDir user

Working LUM and namcd:
# id admin
uid=602(admin) gid=602 groups=602,601(sms smdr group)
Not working LUM and/or namcd:
# id admin
id: admin: No such user
The problem is when namcd is working but not returning any data through that socket. Then you get every linux command that checks for current user to hang forever.
# id admin
(hangs forever until you press ctrl+c)
Solution is to kill all hanged processes and then restart namcd
# rcnamcd restart
I was able to login via ssh in a strange way - it asks me for password and then hanged and I left it like this and after an hour I pressed ctrl+c and it showed me the desired shell on the remote server.

If you do this and it still does not work properly and you get messages like this in /var/log/messages
Nov 22 10:19:45 storage /usr/sbin/namcd[720]:  GetGIDsGroupListNumberOfGroupsOfWS: Error [32] in LDAP search while trying to find group FDNs with scope=base for cn=UNIX Workstation - storage,o=servers
You need to recreate nam.conf. For more information look here: http://geroyblog.blogspot.com/2013/04/novell-enterprise-linux-server-install.html

Thursday, March 28, 2019

Poweshell OpenFIleDialog works in ISE but hangs when started from command line

This code works in Powershell ISE but when started from command line and OpenFileDialog is called it hangs
$OpenFileDialog = New-Object System.Windows.Forms.OpenFileDialog
$OpenFileDialog.Title = "Please Select File"
$OpenFileDialog.InitialDirectory = $initialDirectory
$OpenFileDialog.filter = "TXT files (*.txt)| *.txt"
$OpenFileDialog.ShowDialog()
When we set .ShowHelp = $true it works
$OpenFileDialog = New-Object System.Windows.Forms.OpenFileDialog
$OpenFileDialog.Title = "Please Select File"
$OpenFileDialog.InitialDirectory = $initialDirectory
$OpenFileDialog.filter = "TXT files (*.txt)| *.txt"
$OpenFileDialog.ShowHelp = $true
$OpenFileDialog.ShowDialog()
and I don't know why but it works!

Friday, February 1, 2019

Fast import of billion lines to postgresql

This is the fastest way I've found about importing huge (3+ billion lines) text files in csv-like format in PostgreSQL. It requires python3 and psycopg2

#!/usr/bin/python3
#
# initial stuff by Cac Ko <email@gdpr_protected_stuff> 
# additional by <geroy@horizon9.org>
#
# import very large cvs-like file in postgre database
# 01.02.2019
#
# requires psycopg2 postgresql lib for python
# file format is:
# ----------------------------
# something12:otherthing2345
# something7134:otherthing4243
# .....
# License: Public Domain

import os
import sys
import psycopg2
from psycopg2.extras import execute_values

# use file in tmpfs for faster read/write
POS_FILE = '/run/.position'

# get current position from POS_FILE location
def getPosition():
    result = 0
    if os.path.exists(POS_FILE):
        with open(POS_FILE, 'rb') as f:
            try:
                result = int(f.read())
                print("result = %d" % result)
            except Exception as error: 
                print("Error read():", error)
                result = 0
    return result

# write to this file after each commit()
def storePosition(pos):
    with open(POS_FILE, 'w') as f:
        f.write(str(pos))

# uncomment the following 3 lines if you want perentage / position on stdout
#    y = pos/whole*100
#    print('{0:.2f}%'.format(y), end = '')
#    print(" pos=%s" % (str(pos)))


# store stuff in database
def storeInDb(line, bulk_data):
#    print(line)
    x = []
    x = line.split(':')
    sql = b'INSERT into secrets (somestuff1, somestuff2) VALUES %s'
    try:
        execute_values(cur, sql, bulk_data)
        connection.commit()
    except Exception as error:
        print("Error pri INSERT", error)
        pass

if __name__ == '__main__':
    try:
        if len(sys.argv) < 2:
            print('usage: ./large-file-import.py filename-to-import.csv')
            exit
        f = open(sys.argv[1], 'r', buffering=2000000, errors='replace')
        whole = os.path.getsize(sys.argv[1])
        pos = getPosition()
        print("Start reading from pos=%d" % pos)
        f.seek(pos)
        line = f.readline()
        try:
            bulk_count = 0;
            connection = psycopg2.connect(
                user="postgres", password="somestuff", host="127.0.0.1", port="5433",
database="postgres") cur = connection.cursor() bulk_data = [] split_line = [] while line: split_line = line.split(':') try: l = tuple(split_line) bulk_data.append(l) except (Exception) as error: print(error) pass storePosition(f.tell()) line = f.readline() bulk_count = bulk_count + 1 # store and commit in db after X lines if bulk_count == 9000: storeInDb(line,bulk_data) bulk_count = 0; bulk_data = [] #connection.commit() except (Exception, psycopg2.Error) as error: print("Error while connecting to PostgreSQL", error) finally: if(connection): cur.close() connection.close() print("PostgreSQL connection is closed") except KeyboardInterrupt: storePosition(f.tell()) if(connection): cur.close() connection.close() print("PostgreSQL connection is closed")
I am too lazy to explain how this script works and probably there are some errors but it works for me.


Friday, January 4, 2019

xorg.conf ModeLine for VGA connected ViewSonic monitors

Even with latest Linux distributions you can face the problem of Xorg not detecting your monitor.

My monitor is ViewSonic VG2030wm and it is capable of working at 1680x1050 resolution but Xorg can't detect that and highest resolution that I can use is 1024x768.

You need to add ModeLine so the Xorg can recognize it. Here is my Monitor section from xorg.conf:

Section "Monitor"
        Identifier   "ViewSonic"
        VendorName   "ViewSonic"
        ModelName    "VG2030wm"
        HorizSync    24.0 - 92.0
        VertRefresh  50.0 - 85.0
        Option       "DPMS"
        ModeLine "1680x1050" 170.00 1680 1784 1960 2240 1050 1053 1059 1089 +hsync +vsync
EndSection

Probably you will need just the ModeLine row but I am publishing the whole Monitor section.

You can find more ModeLine for certain monitors here: https://www.mythtv.org/wiki/Modeline_Database

Tuesday, December 4, 2018

Android 4.2.2 how to set default install path to external SD card

You don't need root access to do this. All you need is to enable Developer Options and then enable USB debugging. Once this is done connect adb to your phone:

root@bender:~# adb shell
shell@android:/ $ pm get-install-location
0[auto]
shell@android:/ $ pm set-install-location 2
shell@android:/ $ pm get-install-location
2[external]
shell@android:/ $ logout

root@bender:~#

as you can see from pm help page:

pm get-install-location: returns the current install location.
    0 [auto]: Let system decide the best location
    1 [internal]: Install on internal device storage
    2 [external]: Install on external media

You need to set this value to 2

Monday, March 19, 2018

Jboss / Wildfly errors when accepting socket java.io.IOException: Bad file descriptor problem

Jboss/Wildfly random crashes under heavy load:

Symptoms:

Jboss/Wildfly crashes randomly with  java.io.IOException: Bad file descriptor. Undertow HTTp server does not restart but admin console and everything else is working and nothing in log files.

How to debug it:

Put this in your configuration file (standalone.xml for Wildfly)
        <logger category="org.xnio.nio">
        <level name="DEBUG"/>
        </logger>
The error:
2018-03-03 17:19:15,273 DEBUG [org.xnio.nio.tcp.server] (default Accept) Exception accepting request, closing server channel TCP server (NIO) <69a407dd>: java.io.IOException: Bad file descriptor
    at sun.nio.ch.IOUtil.configureBlocking(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:264)
    at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:477)
    at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

What is the reason for that:

The reason is that you are probably running a BSD and the bug is in Java implementation of KQueueSelectorImpl.java in latest OpenJDK for BSD (openjdk-1.8.162 at the moment). This was tested on NetBSD 7.1.

Solution:

There is no solution at the moment. You need to move your Jboss/Wildfly to Linux. You can try changing default selector with java command line arguments (example is solution for Solaris):

java -Dxnio.nio.selector.provider=sun.nio.ch.PollSelectorProvider .......

At the moment the only solution to this is not to run it under BSD. Under Linux with EPollSelectorProvider everything works just fine.


Tuesday, February 13, 2018

How to filter 99.99% of ssh brute force attacks

Recently I've decided to experiment with ssh ciphers / key exchange algorithms to raise the security of my servers. This is the /etc/ssh/sshd_config I've got:

HostKey /etc/ssh/ssh_host_ed25519_key
Ciphers chacha20-poly1305@openssh.com
KexAlgorithms curve25519-sha256@libssh.org

If you don't have HostKey for Ed25519 generate it:

# ssh-keygen -t ed25519

You need to use recent version of ssh / pyTTY to be able to login to this server.
It seems that using only this Cipher/Kex filters all brute force scanners probably because they do not support it. I see only this kind of messages:

Feb 13 14:41:39 horizon9 sshd[22849]: SSH: Server;Ltype: Version;Remote: xxx.x.xx.xxx-53810;Protocol: 2.0;Client: libssh2_1.7.0
Feb 13 14:41:39 horizon9 sshd[22849]: fatal: ssh_dispatch_run_fatal: no matching cipher found [preauth]

More information about ciphers/algorithms read here:

https://cr.yp.to/ecdh.html#curve25519-paper
https://en.wikipedia.org/wiki/Salsa20
https://en.wikipedia.org/wiki/Poly1305