The problem

We recently hit a major performance problem with our nodejs application that took some investigation, firstly a brief overview of nodejs as I now understand it (which I didn’t before debugging this problem).

Background

• Your application code runs in a single thread. Your code will block all your other code from running.
• Four[1] IO threads are used to run IO operations asynchronously from your code.
• One more additional thread is used as the event loop (epoll)

 

This means that at any given time we can expect nodejs to be running with 6 threads in total.

Diagnosis

Back to our application… We found that while a long running database query was executing IO could completely lock up, including unrelated operations like file read/writes. During a problem phase I used gdb to inspect the state of the process and this is what I saw:-

(gdb) info threads
Id Target Id Frame
6 Thread 0x7ffff7fe4700 (LWP 23141) “SignalSender” 0x00007ffff7023420 in sem_wait ()
from /lib/x86_64-linux-gnu/libpthread.so.0
5 Thread 0x7ffff7f52700 (LWP 23153) “node” 0x00007ffff702418d in read () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7ffff7f11700 (LWP 23156) “node” 0x00007ffff7023cec in __lll_lock_wait ()
from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7ffff7ed0700 (LWP 23159) “node” 0x00007ffff7023cec in __lll_lock_wait ()
from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7ffff7e8f700 (LWP 23160) “node” 0x00007ffff7023cec in __lll_lock_wait ()
from /lib/x86_64-linux-gnu/libpthread.so.0
* 1 Thread 0x7ffff7fe6720 (LWP 23140) “node” 0x00007ffff6d670d3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6

Let’s see what each thread is and what’s doing:-

• Thread: #1: From the epoll_wait() call we can deduce this is the nodejs event loop.
• Thread: #5: By running a backtrace on the thread it was clear to see this is an active database query to our database server. This is obviously an IO thread.
• Threads: #2-#4: These threads are waiting for a lock to be released before they run. These are obviously also IO threads.
• Thread: #6: This is the thread for running our application code (also clear from performing a backtrace on the thread).

 

From that state we can see that all the IO threads are consumed with work so any future IO request must sit and wait for an IO thread to become free. This is why we are seeing all the IO block.

The question is why are we seeing 3 IO threads being needlessly “wasted” just to sit there and wait for a lock to release?

When we perform a backtrace on all of these 3 threads we see that they are blocked within the database library itself and thus we can finally understand the big flaw with our application design and the reason for this performance problem.

Our application is using a single database connection so it’s clear that this is what’s happening:-

1. A database query gets “dispatched” to an IO thread to run and does so.
2. Further database work comes in, gets “dispatched” to the remaining IO threads but they can’t run because the single database connection is already in use so it sits there, within the database library, waiting for the connection to become free.
3. With 4+ pieces of database work we have filled all of the IO threads so any other IO work must sit and wait for an IO thread to become free.

 

This is quite a fundamental design flaw with our application.

The Solution

Fortunately the solution is rather simple and we’ve refactored the application to use a database connection pool (courteousy of node-pool [generic-pool]) so that we now have more than one database connection to work with so that our database queries can run in parallel and not block waiting for each other.

With nodejs v0.10+ we can also take that one step further. In this version libeio as been replaced with a threadpool implementation within libuv. This introduces a new environment variable that allows you to increase the number of IO threads beyond 4 abcpaperwriter.com by setting the environment variable:-

 UV_THREADPOOL_SIZE (Range: 4 – 128, Default: 4).

We set this to around 1.5 * CPUs (in our case this was 12.)

Now we have more IO threads to work with and more database connections available so that queries can run in parallel.

Is there anything else we can do? Yes! To ensure that DB work cannot fully consume the IO threads and block file reading/writing we set the maximum pool size to be one less than the UV_THREADPOOL_SIZE. This means that only UV_THREADPOOL_SIZE – 1 database operations will ever occur in parallel which leaves one IO thread available at all times for our other IO operations (which are minimal) like file reading/writing.

Coding, Tech blocking, nodejs

SSPKD is a system to securely distribute your SSH public keys to multiple hosts.

Security is achieved through the use of your GPG key to sign authorized_keys file updates which are verified on each recipient machine before an update to the authorized_keys file takes place.

In the event an invalid signature is presented then the update is not performed, so if your ‘central’ sspkd server was compromised then an attacker would be unable to simply add their sshkey and push it onto all your other hosts.

SSPKD is currently an alpha-release and is available from github: http://github.com/kmdm/sspkd

Coding, Linux, Tech authorized_keys, distribution, public key, sharing, ssh

Introduction
Following some tinkering I’ve been doing with airtun-ng (and a bugfix I made to ticket #667 to support decrypting WPA CCMP AES QoS packets) it is (now) possible to live capture/sniff WPA traffic providing the WPA handshake is observed. In effect this allows firesheep (and sidejacking in general) to work on a WPA (PSK) network.

Step 1: Firesheep
Firstly you need to get firesheep completely setup, this is outside the scope of this guide – there are excellent instructions provided on the infamous github pull request #70. Once firesheep is setup and you’re able to pick an interface you should proceed.

Step 2: aircrack-ng
Now we can move onto patching aircrack-ng to support WPA networks, to do this you need to do do a checkout from svn, patch the code and then compile it yourself.

$ svn co http://trac.aircrack-ng.org/svn/trunk/ aircrack-ng
$ cd aircrack-ng/src
$ wget http://trac.aircrack-ng.org/raw-attachment/ticket/667/ticket-667.patch
$ wget http://trac.aircrack-ng.org/raw-attachment/ticket/74/airtun-ng-wpa.patch
$ patch -p0 < ticket-667.patch $ patch -p0 < airtun-ng-wpa.patch $ sudo apt-get build-dep aircrack-ng $ cd .. $ make

With any luck this should now compile successfully and your new binaries should be located in the src/ directory.

Step 3: Bringing it all together
Now we’re ready to use aircrack to live capture off WPA networks and feed the data into firesheep for analysis…

Open up a terminal and run (change adapter names and substitute values as appropriate):-

$ sudo airmon-ng start wlan0 CHANNEL
$ sudo src/airtun-ng -a BSSID -e ESSID -p WPAPSK mon0

Note: Used airmon-ng from the standard package installed version since I couldn’t find it in svn and didn’t investigate too much/far!

This should start a tap interface on at0 (or similar) on which the decrypted traffic is fed. Now open another console and run (change IP address if it conflicts with your local subnet):-

$ sudo ifconfig at0 10.10.10.10 up

The IP address is needed since in my experience Firesheep appears to require it. Now open up Firesheep, go into preferences, pick the at0 interface and hit the Start Capture button.

Now on your other wireless client.. disconnect and reconnect to the wireless network (so that the WPA handshake can be observed) and login to Facebook research paper help. With any luck Firesheep should see the session and double clicking on that session should access your Facebook account via sidejacking.

If this doesn’t work use wireshark to monitor the traffic on at0 to see if you’re getting any data at all, the eapol packets for the handshake should be visible.

That’s all folks!

Linux aircrack-ng, firesheep, sidejacking, ubuntu, wpa

Having searched on google I wasn’t able to find a PHP implementation of Shamir’s secret sharing, I was only able to find a perl implementation.

So for anyone else who’s after the same thing I’ve created a (simple) PHP class which implements Shamir’s secret sharing which you can download below, it’s strongly based on the perl implementation above and is also licensed under the GNU GPL.

(It’s missing some error-condition checks in the ::recover() method.)

There’s a simple test case in the class file which’ll explain its usage.

[Download Shamir’s Secret Sharing class]

Coding php, secret, shamir, sharing

I recently forgot my blacklist password while using handcent sms so investigated how possible it would be to reset this password without reinstalling the application.

It turns out it’s pretty straight forward but your phone must be rooted for this to work and you must have enabled usb debugging.

The process goes something like this…

First on your phone navigate to Settings -> Applications -> Manage Applications -> Handcent SMS and click the Force Stop button to close handcent sms.

Next we retrieve the preferences file where the password is stored:-

$ adb pull /data/data/com.handcent.nextsms/shared_prefs/
com.handcent.nextsms_preferences.xml

Then edit the file you just pulled, find and remove the line that looks something like:-

<string name="hc_password">0cc175b9c0f1b6a831c399e269772661</string>

Next we push that file back to the device as follows:-

$ adb push com.handcent.nextsms_preferences.xml /data/data/
com.handcent.nextsms/shared_prefs/com.handcent.nextsms_preferences.xml

Now open handcent sms and the password should be removed ready for you to set another!

Obviously the above commands should be entered on one line, line breaks have been added to wrap the content…

Tech adb, android, handcent

I’ve written a quick rough and ready tool that can extract the rom.zip file from a RUU update from HTC.

The instructions are located in the extractrom.c file itself and reproduced here for clarity:-

 * REQUIREMENTS
 * This program requires a patched unshield 0.6 to correctly support newer
 * installshield cab files.
 *
 * $ wget http://bit.ly/aWIjG2 (used bit.ly link for blog due to styling issues)
 * $ tar xzf unshield-0.6.tar.gz
 * $ cd unshield-0.6
 * $ patch -p1 < ../unshield.patch
 * $ ./configure --prefix=/usr && make && sudo make install
 * 
 * COMPILATION
 * $ gcc -Wall -lunshield -o extractrom extractrom.c 
 * 
 * USAGE
 * $ ./extractrom /path/to/RUU.exe
 *
 * If successful it will create rom.zip in your current directory.
 *

(pay some attention to the NOTES and TODO sections - at the moment the tool doesn't clean up its temporary directory...)

Example Usage:-

$ ./extractrom RUU_Bravo_O2_UK_1.20.207.1_Radio_32.36.00.28U_4.06.00.02_2_release_124865.exe 
Extracting data1.cab...
Extracting data1.hdr...
Extracting rom.zip...
Done!
$ ls -lh rom.zip
-rw-r--r-- 1 kenny kenny 147M 2010-06-24 13:06 rom.zip

You can download it from here: extractrom-0.1.tgz.

Update (2011-12): This is now maintained in github at: https://github.com/kmdm/unruu/

Linux android, HTC, rom.zip, RUU

Having recently purchased a subscription to iPredator I wanted to filter only certain traffic over the VPN and the rest, by default, over my normal ADSL account. My solution was to setup a SOCKS5 and HTTP proxy that went out over the VPN leaving the rest of the traffic to follow the default route as normal.

Please note this guide works for me on Ubuntu 9.10 (Karmic), your mileage may vary!

Step 1: Setting up the PPTP connection to iPredator

Firstly install the required packages:-

$ sudo apt-get install pptp-linux ppp

Obviously the first step is to actually get the VPN setup, so I created the peers file: /etc/ppp/peers/ipredator

pty "pptp vpn.ipredator.se --nolaunchpppd --nobuffer --loglevel 0"
name YOUR-USERNAME-HERE
remotename ipredator
file /etc/ppp/options.pptp
ipparam ipredator
nodefaultroute
persist

Next you have to populate your chap-secrets file with your password: /etc/ppp/chap-secrets:-

YOUR-USERNAME-HERE    ipredator   YOUR-PASSWORD-HERE        *

Finally edit the /etc/ppp/options.pptp file and uncomment the require-mppe-128 line. Now if you run “pon ipredator” and tail the syslog you should find that the VPN comes up although at the moment no traffic will route over it so go ahead and and run “poff ipredator”.

Step two: Setup the SOCKS5 server

Now we need to setup the SOCKS5 server, danted. First go ahead and install it:-

$ sudo apt-get install dante-server

Next edit the config file /etc/danted.conf and make it look something like this, changing 192.168.1.0/24 to your local subnet:-

logoutput: syslog

internal: eth0 port = 1080
external: ppp0

method: none
clientmethod: none

user.privileged: proxy
user.notprivileged: proxy
user.libwrap: proxy

client pass {
    from: 192.168.1.0/24 port 1-65535 to: 0.0.0.0/0
}

client pass {
    from: 127.0.0.1/32 port 1-65535 to: 0.0.0.0/0
}

client block {
	from: 0.0.0.0/0 to: 0.0.0.0/0
log: connect error
}

block {
	from: 0.0.0.0/0 to: 127.0.0.0/8
	log: connect error
}

pass {
       from: 192.168.1.0/24 to: 0.0.0.0/0
       protocol: tcp udp
}

pass {
	from: 127.0.0.1/32 to: 0.0.0.0/0
        protocol: tcp udp
}

block {
	from: 0.0.0.0/0 to: 0.0.0.0/0
        log: connect error
}

Finally since we will be controlling when this process starts we need to prevent it from starting at boot by running the following:-

$ sudo update-rc.d -f danted remove
$ sudo service danted stop

Step three: Setting up the HTTP Proxy

Firstly install the HTTP proxy (polipo):-

$ sudo apt-get install polipo

Now make your config file /etc/polipo/config look something like this, changing as appropriate:-

proxyAddress = "192.168.1.1"    # IPv4 only
proxyPort = 3128
allowedClients = 192.168.1.0/24
allowedPorts = 1-65535
                                            
socksParentProxy = "192.168.1.1:1080"
socksProxyType = socks5
chunkHighMark = 50331648
diskCacheRoot = ""
disableIndexing = true
disableServersList = true
dnsUseGethostbyname = yes
disableVia=true
censoredHeaders = from,accept-language,x-pad,link
censorReferer = maybe
maxConnectionAge = 5m
maxConnectionRequests = 120
serverMaxSlots = 8
serverSlots = 2
tunnelAllowedPorts = 1-65535

Finally since we’ll be controlling when this service starts, prevent it from starting on boot:-

$ sudo update-rc.d -f polipo remove
$ sudo service polipo stop

Step four: Setting up the VPN routes

Now that our services are configured we need to add the magic glue that makes their requests route out via the VPN – this will be done using an ip-up.d script and an ip-down.d script.

NB: Both these scripts assume that your proxy user has user id 13, please change accordingly if this is not the case.

Create the /etc/ppp/ip.up.d/ipredator script as follows (changing 192.168.1.0/24 as appropriate):-

#!/bin/bash

# Check for the iPredator VPN.
[ "$6" == "ipredator" ] || exit 0

# Setup the extra routing table for iPredator.
ip route add table 200 via $5
ip route add 192.168.1.0/24 dev eth0 table 200
ip rule add fwmark 0x50 table 200

# Mark all packets owned by the proxy user to go through iPredator.
iptables -t mangle -A OUTPUT --match owner --uid-owner 13 -j MARK --set-mark 80

# Start the daemons we use for this VPN.
service danted start
service polipo start

Next create the /etc/ppp/ip-down.d/ipredator similarly:-

#!/bin/bash

# Check for the iPredator VPN.
[ "$6" == "ipredator" ] || exit 0

# Stop the daemons we use for this VPN.
service danted stop
service polipo stop

# Remove the packet mangling rule.
iptables -t mangle -D OUTPUT --match owner --uid-owner 13 -j MARK --set-mark 80

# Remove the extra routing table for iPredator.
ip rule del fwmark 0x50 table 200
ip route del 192.168.1.0/24 dev eth0 table 200
ip route del table 200 via $5

Ensure both scripts are executable.

Step five: Testing the services.
To test firstly bring up the VPN by running “pon ipredator”, you should see the VPN come up and both polipo and danted should be started.

Check, by default, you get your ADSL IP:-

$ wget -q -O - http://www.whatismyip.com/automation/n09230945.asp

Next check that when routing via the proxy you get a different (VPN) IP:-

$ export http_proxy=192.168.1.1:3128 
$ wget -q -O - http://www.whatismyip.com/automation/n09230945.asp

If it all works congratulations, you now have a http and socks5 proxy routing traffic over your VPN while all other traffic routes out via your normal connection.

IF it fails, I’m afraid you’re stuck going through the logs etc… Sorry!

Linux ipredator, proxy, ubuntu, vpn

Been a while since my last post due to being extremely busy here over the past few months.

Recently we came across a little problem where one of our customers had a dedicated mailserver (courier, exim4) but had no ability to change their own passwords.

So I came up with two scripts, one bash (to do the actual password reset) and one php (as a front-end to the bash script). These two scripts are designed to be used in tandem to provide adequate input validation and security.

You can find these two scripts here: chpasswd.sh (backend bash script) and chpasswd.txt (frontend php script).

Please note these are simple scripts, in both the lack of error reporting and lack of styling / CSS however they should be functional and with the use of su’ing to the user and avoidance of a setuid root script they should be secure.

If you implement these you are strongly advised to also implement some anti-bruteforce code or in the very least restrict access to internal use only.

Coding, Linux Linux, password reset, php

The Problem
I took the plunge to upgrade my laptop to Jaunty yesterday and upon reboot the first thing I noticed was that my encrypted home partition was no longer being mounted. After enabling pam_mount’s debug option and tracing through the problem it turned out to be that mount.crypt was now unable to mount my encrypted partition.

mount.crypt fails to mount the partition because when it calls cryptsetup it fails to pass the keysize parameter so cryptsetup uses the default (256), which is fine if you used a 256 bit encryption key, but unfortunately I’d used a 128-bit encryption key so therefore cryptsetup fails to setup the encrypted volume correctly thus causing the subsequent call to mount to fail.

The Solution
Fortunately I don’t let little things like that stop me from getting things working. Firstly I created a patched libpam-mount package (LP: #367577) that does pass the -s option through to cryptsetup, this then allows mount.crypt to successfully mount the encrypted volume.

However, pam_mount was still not able to mount the volume when I logged in, thankfully it only required a config tweak to match the new options that are now passed to mount.crypt, so my new pam_mount.conf.xml config block now looks like this:-

<volume
    user="kenny"
    fstype="crypt"
    path="/dev/sda3"
    mountpoint="/home/kenny"
    options="cipher=aes,hash=ripemd160,fsk_cipher=aes-128-ecb,fsk_hash=md5,
keyfile=/home/kenny.key"
/>

(Annoyingly hash=ripemd160 has to be passed because cryptsetup defaults to that but mount.crypt defaults to and passes through “plain” unless told otherwise.)

Once this change was made (and the fixed packages installed), pam_mount was once again able to mount my encrypted home directory when I log in.

Linux encrypted, intrepid, jaunty, libpam-mount, ubuntu

After getting my Antec Black Fusion, I just couldn’t get it to work in Ubuntu with lirc 0.8.4a, so I went ahead and created some simple packages for lirc 0.8.5pre1 which do appear to work with this case / device.

Notice: Please note in preparing these lirc 0.8.5pre1 packages I’ve dropped a fair few of the Ubuntu specific enhancements to lirc since this is intended as a fix solely for adding support for this device. Debdiffs welcome. 😉

Firstly add my repository key to apt so that the packages authenticate:

$ wget -O - http://packages.kennynet.co.uk/repository.key | sudo apt-key
add -

(note: trailing dash)

Next grab the sources.list entries for my repository so you can download the 0.8.5pre1 packages:

$ sudo wget -O /etc/apt/sources.list.d/kennynet-testing.list
 http://packages.kennynet.co.uk/intrepid-testing.list

Now we’re ready to install the packages as follows:

$ sudo apt-get update
$ sudo apt-get install lirc lirc-modules-source

With these packages installed, you only need the following line in /etc/lircd.conf:-

include "/usr/share/lirc/remotes/imon/lircd.conf.imon-antec-veris"

I’ve also made a custom (somewhat hacky) startup script which’ll launch the required two lircd daemons for the two lirc devices created under /dev.

You can download the script here, just put it in /etc/init.d/: lirc-imon

Now make it executable and set it to run on startup:

$ sudo chmod +x /etc/init.d/lirc-imon
$ sudo update-rc.d -f lirc remove
$ sudo update-rc.d lirc-imon defaults 51

This’ll disable the default lirc init.d script and enable the lirc-imon one that I’ve created.

From then on I’d make sure it’s loaded the new modules / started lirc as follows then test it out using irw:-

$ sudo /etc/init.d/lirc stop
$ sudo modprobe -r lirc_dev lirc_imon
$ sudo /etc/init.d/lirc-imon start
$ irw

Press some buttons, they should now show up.

All works for me, let me know how it goes… there are numerous debug steps at each stage if it doesn’t work but I’ve left these out to try and keep the e-mail relatively short. We can go through those if you still experience problems. Please leave comments if you have any questions.

I’ll package the very latest imonlcd patch for lcdproc and upload that to my testing repositories soon and make another quick post.

Linux antec, imon, lirc, ubuntu