CompleteFusion

HAProxy is a great load balancer; however, it doesn't natively terminate SSL connections. Fortunately, stunnel can terminate SSL connections and seamlessly forward them to HAProxy. To be able to log the client's IP address, we'll need to patch stunnel to provide X-Forwarded-For headers to HAProxy.

The following instructions are based on Debian Squeeze, HAProxy 1.4.8-1, and stunnel 4.32.

To get started, download the stunnel source and X-Forwaded-For patch:

wget http://www.stunnel.org/download/stunnel/src/stunnel-4.32.tar.gz
wget http://haproxy.1wt.eu/download/patches/stunnel-4.32-xforwarded-for.diff

Before building stunnel, we need to install a few other packages:

apt-get install build-essential libssl-dev haproxy

To install stunnel:

tar -zxvf stunnel-4.32.tar.gz
cd stunnel-4.32
patch -p1 < ../stunnel-4.32-xforwarded-for.diff
./configure
make && make install
 
adduser stunnel
ln -s /usr/local/etc/stunnel /etc/stunnel
touch /var/log/stunnel.log
chown -R stunnel:stunnel /usr/local/etc/stunnel
chown -R stunnel:stunnel /var/run/stunnel
chown stunnel:stunnel /var/log/stunnel.log

Then we have to create the necessary files in /etc/stunnel: stunnel.conf, stunnel.pem, and the SSL certificate (server.crt) and key file (server.key). The server.crt and server.key are the SSL certificate and key which would normally go on a web server. The following stunnel.conf will accept incoming HTTPS connections on port 443, then forward it to port 81 as regular HTTP traffic – just change the ip address from 192.168.1.1 to your own.

Example /etc/stunnel/stunnel.conf:

cert=/etc/stunnel/stunnel.pem
setuid=stunnel
setgid=stunnel
pid=/var/run/stunnel/stunnel.pid
output = /var/log/stunnel.log
 
socket=l:TCP_NODELAY=1
socket=r:TCP_NODELAY=1
 
[https]
  cert=/etc/stunnel/server.crt
  key=/etc/stunnel/server.key
  accept=192.168.1.1:443
  connect=192.168.1.1:81
  xforwardedfor=yes

Next, we'll need to create our stunnel certificate file, stunnel.pem. More details about the certificate creation process are available at Using Certificates with stunnel, but you can just run the following command:

openssl req -new -x509 -days 365 -nodes -config /etc/stunnel/stunnel.conf -out /etc/stunnel/stunnel.pem -keyout /etc/stunnel/stunnel.pem

Finally, here's an example haproxy.cfg file (/etc/haproxy/haproxy.cfg) – just change your web server IPs:

global
        log 127.0.0.1 local0
        log 127.0.0.1 local1 notice
        user haproxy
        group haproxy
        daemon
        maxconn 20000
 
defaults
        log global
        option dontlognull
        balance leastconn
        clitimeout 60000
        srvtimeout 60000
        contimeout 5000
        retries 3
        option redispatch
 
listen http 192.168.1.1:80
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.1.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ http
        server webserver1 192.168.1.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.1.3 cookie webserver2 maxconn 5000
 
listen https 192.168.1.1:81
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.1.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ https
        server webserver1 192.168.1.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.1.3 cookie webserver2 maxconn 5000

Edited: I reported this issue to Debian and Christian Kastner patched it. Debian cron versions 3.0pl1-110 and higher should behave properly.

As reported in Ubuntu bug #151231 and at stackoverflow, cron jobs can fail silently when too much output is produced and an MTA is not installed.

Although the comments in the Ubuntu bug indicate that Debian is not affected because it includes the exim MTA by default, using a Debian VServer only installs the most basic packages – no MTA. I encountered this bug today while I was working on a backup script for a Subversion repository that runs in a Debian-based VServer. I previously discussed svn backupsand my script was using the same svnadmin dump technique I mentioned in that post.

When I ran the backup script manually, everything worked as intended and it produced a ~1.3GB backup file; however, when it ran under cron, the backup file was only ~2MB. Since the simplest solution is usually the correct one, I didn't initially suspect a bug in cron as the problem and tried to troubleshoot my script/environment. I eventually found the two links I listed above, where others have encountered the same bug.

It appears that the bug in cron causes it to silently fail when too much output is directed to STDERR without an MTA being present. In the case of svn backups, using the "quiet" flag (-q) for svnadmin dump is a possible workaround. In other cases, either redirecting STDERR to /dev/null or setting MAILTO="" in /etc/default/cron seem to be valid workarounds.

I've needed to recover deleted files on ext3, FAT, and NTFS file systems in the past, but I recently needed to recover a previous version of a text file I had overwritten by editing and saving it. I initially thought I might be able to recover it either by accessing the inode used by the previous version of the file, or by looking at ext3's journal.

Unfortunately, I had used nano to edit the file. Apparently, nano saves files by truncating and overwriting the file, reusing the same inode. Also, I quickly realized ext3's journal wouldn't help because my file system was mounted using data=ordered, not data=journal. From the ext3 FAQ:

1. data=journal: Journals all data and metadata, so data is written twice.
2. data=ordered: Only journals metadata changes.

Ultimately, I was able to recover the file with some help from stat, debugfs, and blkls from The Sleuth Kit. Before getting started, you'll need to install The Sleuth Kit. On Debian, it is available as a package, so: apt-get install sleuthkit

First, check the inode being used by the file: stat file.txt | grep Inode

This should return a line containing the inode, like: Inode: 1474575

Next, backup the file, then delete it:
cp file.txt file.old
rm file.txt

Run debugfs /dev/sda1, replacing /dev/sda1 with the hard drive the file is on. From the debugfs CLI, run stats and check its output for "Blocks per group". On my system, and most of the time, this is 32768. While still in the debugfs CLI, run imap <inode> to get the block: imap <1474575>. In my case, the block was 5898242.

Once you know the block the inode is in, and the number of blocks per group, create a block range: 5898242+32768-1 and use blkls to copy the block to a file: blkls /dev/sda1 5898242-5931009 > tmp.dat

Finally, open tmp.dat in your favorite text editor or use grep to search for the overwritten version of your file.

For more details about ext3 file systems and recovering deleted files:

1. Recovering Deleted Files on an ext3 File System
2. Data Recovery on Linux and ext3

As a followup to my AoE storage server posts (Part 1, Part 2), I recently needed to extend an LVM logical volume from 1TB to 1.3TB. As per the LVM howto, I followed these steps:

On the "client" (the server that had mounted the AoE share):
umount /backups

On the storage server:
vblade-persist stop 0 1
lvextend -L1331GB /dev/data1/backups
e2fsck -f /dev/data1/backups
resize2fs /dev/data1/backups
vblade-persist start 0 1

On the "client":
aoe-stat
mount /dev/etherd/e0.1 /backups

I recently created a new Subversion repository for a new personal project, and realized I needed to add it to my daily backup script. After a quick bit of searching via Google to refresh my memory, I found exactly what I thought I was looking for:

svnadmin hotcopy /svn /backups/svn --clean-logs

Indeed, O'Reilly's Version Control with Subversion states that svnadmin hotcopy will make a hot copy of a repository, and "is able to make a safe copy of the repository at any time, regardless if other processes are using the repository." However, further research seemed to indicate that the following may be a better option:

svnadmin dump /svn > /backups/svndump-`date %Y%m%d`.dmp

According to several posts on the Subversion mailing list, svnadmin's hotcopy "theoretically needs the correct machine architecture, version of bdb (if it's a bdb repo), and version of Subversion to work"; however, svnadmin's dump could "theoretically be loaded by any future version of Subversion". Whether or not that's actually true, since there are numerous "theoretical" qualifiers, I decided to go with svnadmin dump.

It is often useful to be able to retrieve and update exchange rates in an automated manner. Fortunately, the European Central Bank publishes its foreign exchange reference rates on a daily basis, and they're kind enough to provide an XML file with the data. Although the data uses the Euro as the base currency, we can easily recalculate the rates with US Dollars as the base currency.

For example, to get the current exchange rate for Euros to US Dollars, we can run the following:

1

curl --silent http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml | grep -e 'USD' | cut -d \' -f 4

As of May 22, 2009, the reference rate returned for US Dollars was 1.3972. To turn the USD per Euro rate of $1.3972/€ into Euro per USD (USD as the base rate), we simply take the inverse. To do so, we calculate 1/$1.3972, which equals €0.7157/$. To perform the calculation programatically and return the Euro per USD value:

1

echo "scale=4; 1/`curl --silent http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml | grep -e 'USD' | cut -d \' -f 4`" |bc -l

When calculating foreign exchange rates, one must consider the principle of arbitrage, which is expressed mathematically as A-to-C = (A-to-B)*(B-to-C). Therefore, GBP per USD = (GBP per EU)*(EU per USD). To perform the calculation programatically and return the GBP per USD value:

1
2
3
4
5
6

#!/bin/bash
USD_EU=`curl --silent http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml | grep -e 'USD' | cut -d \' -f 4`
GBP_EU=`curl --silent http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml | grep -e 'GBP' | cut -d \' -f 4`
EU_USD=`echo "scale=5; 1/$USD_EU" | bc -l`
GBP_USD=`echo "scale=5; $GBP_EU*$EU_USD" | bc -l`
echo $GBP_USD

For example, using the reference rates provided by the European Central Bank as of May 22, 2009, running our script to calculate the GBP per USD rate yields £0.62846/$.