CompleteFusion

HAProxy is a great load balancer; however, it doesn't natively terminate SSL connections. Fortunately, stunnel can terminate SSL connections and seamlessly forward them to HAProxy. To be able to log the client's IP address, we'll need to patch stunnel to provide X-Forwarded-For headers to HAProxy.

The following instructions are based on Debian Squeeze, HAProxy 1.4.8-1, and stunnel 4.32.

To get started, download the stunnel source and X-Forwaded-For patch:

wget http://www.stunnel.org/download/stunnel/src/stunnel-4.32.tar.gz
wget http://haproxy.1wt.eu/download/patches/stunnel-4.32-xforwarded-for.diff

Before building stunnel, we need to install a few other packages:

apt-get install build-essential libssl-dev haproxy

To install stunnel:

tar -zxvf stunnel-4.32.tar.gz
cd stunnel-4.32
patch -p1 < ../stunnel-4.32-xforwarded-for.diff
./configure
make && make install
 
adduser stunnel
ln -s /usr/local/etc/stunnel /etc/stunnel
touch /var/log/stunnel.log
chown -R stunnel:stunnel /usr/local/etc/stunnel
chown -R stunnel:stunnel /var/run/stunnel
chown stunnel:stunnel /var/log/stunnel.log

Then we have to create the necessary files in /etc/stunnel: stunnel.conf, stunnel.pem, and the SSL certificate (server.crt) and key file (server.key). The server.crt and server.key are the SSL certificate and key which would normally go on a web server. The following stunnel.conf will accept incoming HTTPS connections on port 443, then forward it to port 81 as regular HTTP traffic – just change the ip address from 192.168.1.1 to your own.

Example /etc/stunnel/stunnel.conf:

cert=/etc/stunnel/stunnel.pem
setuid=stunnel
setgid=stunnel
pid=/var/run/stunnel/stunnel.pid
output = /var/log/stunnel.log
 
socket=l:TCP_NODELAY=1
socket=r:TCP_NODELAY=1
 
[https]
  cert=/etc/stunnel/server.crt
  key=/etc/stunnel/server.key
  accept=192.168.1.1:443
  connect=192.168.1.1:81
  xforwardedfor=yes

Next, we'll need to create our stunnel certificate file, stunnel.pem. More details about the certificate creation process are available at Using Certificates with stunnel, but you can just run the following command:

openssl req -new -x509 -days 365 -nodes -config /etc/stunnel/stunnel.conf -out /etc/stunnel/stunnel.pem -keyout /etc/stunnel/stunnel.pem

Finally, here's an example haproxy.cfg file (/etc/haproxy/haproxy.cfg) – just change your web server IPs:

global
        log 127.0.0.1 local0
        log 127.0.0.1 local1 notice
        user haproxy
        group haproxy
        daemon
        maxconn 20000
 
defaults
        log global
        option dontlognull
        balance leastconn
        clitimeout 60000
        srvtimeout 60000
        contimeout 5000
        retries 3
        option redispatch
 
listen http 192.168.1.1:80
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.1.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ http
        server webserver1 192.168.1.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.1.3 cookie webserver2 maxconn 5000
 
listen https 192.168.1.1:81
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.1.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ https
        server webserver1 192.168.1.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.1.3 cookie webserver2 maxconn 5000

Edited: I reported this issue to Debian and Christian Kastner patched it. Debian cron versions 3.0pl1-110 and higher should behave properly.

As reported in Ubuntu bug #151231 and at stackoverflow, cron jobs can fail silently when too much output is produced and an MTA is not installed.

Although the comments in the Ubuntu bug indicate that Debian is not affected because it includes the exim MTA by default, using a Debian VServer only installs the most basic packages – no MTA. I encountered this bug today while I was working on a backup script for a Subversion repository that runs in a Debian-based VServer. I previously discussed svn backupsand my script was using the same svnadmin dump technique I mentioned in that post.

When I ran the backup script manually, everything worked as intended and it produced a ~1.3GB backup file; however, when it ran under cron, the backup file was only ~2MB. Since the simplest solution is usually the correct one, I didn't initially suspect a bug in cron as the problem and tried to troubleshoot my script/environment. I eventually found the two links I listed above, where others have encountered the same bug.

It appears that the bug in cron causes it to silently fail when too much output is directed to STDERR without an MTA being present. In the case of svn backups, using the "quiet" flag (-q) for svnadmin dump is a possible workaround. In other cases, either redirecting STDERR to /dev/null or setting MAILTO="" in /etc/default/cron seem to be valid workarounds.

I've needed to recover deleted files on ext3, FAT, and NTFS file systems in the past, but I recently needed to recover a previous version of a text file I had overwritten by editing and saving it. I initially thought I might be able to recover it either by accessing the inode used by the previous version of the file, or by looking at ext3's journal.

Unfortunately, I had used nano to edit the file. Apparently, nano saves files by truncating and overwriting the file, reusing the same inode. Also, I quickly realized ext3's journal wouldn't help because my file system was mounted using data=ordered, not data=journal. From the ext3 FAQ:

1. data=journal: Journals all data and metadata, so data is written twice.
2. data=ordered: Only journals metadata changes.

Ultimately, I was able to recover the file with some help from stat, debugfs, and blkls from The Sleuth Kit. Before getting started, you'll need to install The Sleuth Kit. On Debian, it is available as a package, so: apt-get install sleuthkit

First, check the inode being used by the file: stat file.txt | grep Inode

This should return a line containing the inode, like: Inode: 1474575

Next, backup the file, then delete it:
cp file.txt file.old
rm file.txt

Run debugfs /dev/sda1, replacing /dev/sda1 with the hard drive the file is on. From the debugfs CLI, run stats and check its output for "Blocks per group". On my system, and most of the time, this is 32768. While still in the debugfs CLI, run imap <inode> to get the block: imap <1474575>. In my case, the block was 5898242.

Once you know the block the inode is in, and the number of blocks per group, create a block range: 5898242+32768-1 and use blkls to copy the block to a file: blkls /dev/sda1 5898242-5931009 > tmp.dat

Finally, open tmp.dat in your favorite text editor or use grep to search for the overwritten version of your file.

For more details about ext3 file systems and recovering deleted files:

1. Recovering Deleted Files on an ext3 File System
2. Data Recovery on Linux and ext3

With the apt-cacher package in Debian Lenny, my web interface for the cache statistics wasn't being properly updated. Eventually, I tracked down the problem and figured out that I needed to modify the following section of apt-cacher-report.pl to look like this:

1
2
3
4
5
6
7
8
9
10
11
12

#parse logfile:
foreach $logfile_line (@logdata)
{
        #$logfile_line =~ s/ /\+/g;
        @line = split /\|/, $logfile_line;
        $req_date = $line[0];
#       $req_pid = $line[1];
#       $req_ip   = $line[2];
        $req_result = $line[3];
        $req_bytes  = 0;
        $req_bytes  = $line[4] if $line[4] =~ /^[0-9]+$/;
#       $req_object = $line[5];

Saving time and bandwidth are two things every system administrator loves. Implementing apt-cacher could save you a considerable amount of both, depending on the number of servers you have. As the name implies, apt-cacher caches packages and package lists for apt packaging systems on Debian or Debian-like systems. Here's how I recently implemented apt-cacher:

apt-get install apt-cacher

I edited /etc/apt-cacher/apt-cacher.conf

cache_dir=/path/to/cache/directory
admin_email=your@email.com
daemon_addr=IPADDRESS

I also edited /etc/default/apt-cacher and set:

AUTOSTART=1

Then ran:

/etc/init.d/apt-cacher start

I imported the existing files apt-get had locally archived:

/usr/share/apt-cacher/apt-cacher-import.pl -r /var/cache/apt/archives

Finally, I edited /etc/apt/sources.list on the apt-cacher server, and each client server:

deb http://hostname:3142/ftp.us.debian.org/debian/ lenny main non-free contrib
deb-src http://hostname:3142/ftp.us.debian.org/debian/ lenny main non-free contrib

deb http://hostname:3142/security.debian.org/ lenny/updates main non-free contrib
deb-src http://hostname:3142/security.debian.org/ lenny/updates main non-free contrib

As I mentioned in Debian NAS using AoE (Part 1), I recently needed to implement a low-cost, enterprise grade storage system for backups and to provide network attached storage for vserver-based Debian servers. I was able to build a suitable storage server with 13TB of raw storage for a reasonable price, and decided to use Debian with LVM2 and AoE to divide and share it across the network.

Before getting to the OS installation, I had to setup and build the RAID arrays in the HighPoint 3540 BIOS. I set up a RAID 1 mirror of the two 300GB drives to use for the OS. Then, I created a 6-disk RAID6 and a 7-disk RAID6, which left me with 4GB and 5GB of usage storage.

Next, I installed Debian Lenny from the Lenny netinst ISO onto the 300GB RAID 1 mirror I created, with no additional packages/package groups selected. After the installation was complete, I rebooted the server to install openssh server, lvm2, and AoE. Since I wanted to be able to use the storage server to run vserver instances if needed, I also installed the vserver kernel and tools:

apt-get update
apt-get install vserver-debiantools linux-image-2.6-vserver-amd64 lvm2 vblade vblade-persist aoe-tools

I configured LVM to recognize the physical volumes:

pvcreate /dev/sdb
pvcreate /dev/sdc

I created the volume groups data1 and data2 on the physical volumes:

vgcreate data1 /dev/sdb
vgcreate data2 /dev/sdc

I created two 1TB LVM logical partitions (/dev/data1/backups, /dev/data1/backups2):

lvcreate -L 1TB -n backups data1
lvcreate -L 1TB -n backups2 data1

I made the ext3 filesystem on them so they would be usable:

mkfs.ext3 /dev/data1/backups
mkfs.ext3 /dev/data1/backups2

I configured AoE (ATA over Ethernet):

modprobe aoe

vblade-persist setup 0 1 eth0 /dev/data1/backups
vblade-persist setup 0 2 eth0 /dev/data1/backups2

vblade-persist start 0 1
vblade-persist start 0 2

Finally, I verified they were up and available using:

aoe-discover
aoe-stat

Output from aoe-stat… :

e0.1 1099.511GB eth1 up
e0.2 1099.511GB eth1 up

To mount an AoE export on a remote server, I used the following (on the remote server):

modprobe aoe
apt-get install aoetools
aoe-discover
aoe-stat (should show the available AoE exports)
mkdir /mountpoint
mount /dev/etherd/e0.1 /mountpoint (replace e0.1 with the appropriate device from aoe-stat)