CompleteFusion

Before 2010 comes to a close, be sure to check out the sysadmin's Advent calendar, SysAdvent! Last year, I contributed an article about ATA over Ethernet (AoE), and I wrote about Website Performance Optimization this year.

HAProxy is a great load balancer; however, it doesn't natively terminate SSL connections. Fortunately, stunnel can terminate SSL connections and seamlessly forward them to HAProxy. To be able to log the client's IP address, we'll need to patch stunnel to provide X-Forwarded-For headers to HAProxy.

The following instructions are based on Debian Squeeze, HAProxy 1.4.8-1, and stunnel 4.32.

To get started, download the stunnel source and X-Forwaded-For patch:

wget http://www.stunnel.org/download/stunnel/src/stunnel-4.32.tar.gz
wget http://haproxy.1wt.eu/download/patches/stunnel-4.32-xforwarded-for.diff

Before building stunnel, we need to install a few other packages:

apt-get install build-essential libssl-dev haproxy

To install stunnel:

tar -zxvf stunnel-4.32.tar.gz
cd stunnel-4.32
patch -p1 < ../stunnel-4.32-xforwarded-for.diff
./configure
make && make install
 
adduser stunnel
ln -s /usr/local/etc/stunnel /etc/stunnel
touch /var/log/stunnel.log
chown -R stunnel:stunnel /usr/local/etc/stunnel
chown -R stunnel:stunnel /var/run/stunnel
chown stunnel:stunnel /var/log/stunnel.log

Then we have to create the necessary files in /etc/stunnel: stunnel.conf, stunnel.pem, and the SSL certificate (server.crt) and key file (server.key). The server.crt and server.key are the SSL certificate and key which would normally go on a web server. The following stunnel.conf will accept incoming HTTPS connections on port 443, then forward it to port 81 as regular HTTP traffic – just change the ip address from 192.168.1.1 to your own.

Example /etc/stunnel/stunnel.conf:

cert=/etc/stunnel/stunnel.pem
setuid=stunnel
setgid=stunnel
pid=/var/run/stunnel/stunnel.pid
output = /var/log/stunnel.log
 
socket=l:TCP_NODELAY=1
socket=r:TCP_NODELAY=1
 
[https]
  cert=/etc/stunnel/server.crt
  key=/etc/stunnel/server.key
  accept=192.168.1.1:443
  connect=192.168.1.1:81
  xforwardedfor=yes

Next, we'll need to create our stunnel certificate file, stunnel.pem. More details about the certificate creation process are available at Using Certificates with stunnel, but you can just run the following command:

openssl req -new -x509 -days 365 -nodes -config /etc/stunnel/stunnel.conf -out /etc/stunnel/stunnel.pem -keyout /etc/stunnel/stunnel.pem

Finally, here's an example haproxy.cfg file (/etc/haproxy/haproxy.cfg) – just change your web server IPs:

global
        log 127.0.0.1 local0
        log 127.0.0.1 local1 notice
        user haproxy
        group haproxy
        daemon
        maxconn 20000
 
defaults
        log global
        option dontlognull
        balance leastconn
        clitimeout 60000
        srvtimeout 60000
        contimeout 5000
        retries 3
        option redispatch
 
listen http 192.168.1.1:80
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.1.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ http
        server webserver1 192.168.1.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.1.3 cookie webserver2 maxconn 5000
 
listen https 192.168.1.1:81
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.1.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ https
        server webserver1 192.168.1.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.1.3 cookie webserver2 maxconn 5000

I recently migrated CompleteFusion to a new hosting provider, which meant switching from a Xen-based VPS to OpenVZ. Although OpenVZ VPS hosting is generally much less expensive, it often lacks swap space and requires special consideration of memory usage. Running OpenSSH, MySQL, and Apache2 with mod_php5 caused out of memory problems, even with 512MB RAM.

First, I replaced OpenSSH with Dropbear, which uses about 50% less memory while providing very similar SSH2 server capabilities. On Debian, installing dropbear was simple:

apt-get install dropbear
sed -i 's/NO_START=0/NO_START=1/g' /etc/default/dropbear
/etc/init.d/ssh stop && /etc/init.d/dropbear start

Although I could have made various configuration changes to tune Apache2 to use less memory, switching to a stock lighttpd configuration with fastcgi was enough to reduce memory usage to acceptable levels. I installed lighttpd on Debian as follows:

apt-get install lighttpd
apt-get install php5-cgi
echo "cgi.fix_pathinfo = 1" >> /etc/php5/cgi/php.ini
lighttpd-enable-mod fastcgil
/etc/init.d/lighttpd force-reload

On my OpenVZ VPS, I ran into a problem with starting lighttpd, where it reported that port 80 was in use, even though it wasn't. As it turns out, it was apparently because my container lacked IPv6 support. I resolved it by commenting out the following line in /etc/lighttpd/lighttpd.conf:

#include_shell "/usr/share/lighttpd/use-ipv6.pl"
These changes should lower your memory usage enough to run well inside of 512MB RAM, but additional configuration will probably be needed if you only have 256MB or less.

When I embarked on my first subversion to git migration recently, I thought it would be fairly easy — after all, migrating from CVS to subversion wasn't very difficult. Unfortunately, it wasn't as easy as I had thought, because I couldn't readily find reliable information to guide me. Since I had to spend entirely too time searching and testing various migration methods, I hope this post will help you avoid a similar fate.

The first step in migrating a repository from subversion to git is to get a list of committers from svn. There's no native way of requesting that information from subversion, but you can use svn log and grep the lines with revision data. If your subversion repository is on a remote host, you can use http to retrieve it and list all subversion commiters:

svn log --quiet http://www.foo.com/path/to/svn/ | grep '^r' | awk -F\| '{print $2}' | sort -u > committers.txt

If your subversion repository is hosted locally, just use file:///path/to/svn (note the three slashes in file:///, instead of two for http://).

After you have the list of subversion committers, create a users.txt file to map svn committers to git users, using the following format:

svnuser = Full Name <email@domain.com>

Although some people suggest using a basic git-svn migration or the svn2git scripts, I found the git-svn-abandon scripts to be the easiest and most sane. After downloading git-svn-abandon and putting the scripts somewhere in your PATH, you can start the migration process:

git svn clone --prefix=svn/ --stdlayout --authors-file=users.txt http://www.foo.com/path/to/svn/

Then cd into the directory the git-clone command just created and run:

git svn-abandon-fix-refs
git svn-abandon-cleanup

Finally, clone the new repository to make it sharable and configured for use:

cd ..
git clone --bare /path/to/git_svn/repo /path/to/shared_git/repo.git
cd /path/to/shared_git/repo.git
git --bare update-server-info
mv hooks/post-update.sample hooks/post-update

For more information, take a look at the git-svn-abandon author's article about Migrating from Subversion to Git.

Edited: I reported this issue to Debian and Christian Kastner patched it. Debian cron versions 3.0pl1-110 and higher should behave properly.

As reported in Ubuntu bug #151231 and at stackoverflow, cron jobs can fail silently when too much output is produced and an MTA is not installed.

Although the comments in the Ubuntu bug indicate that Debian is not affected because it includes the exim MTA by default, using a Debian VServer only installs the most basic packages – no MTA. I encountered this bug today while I was working on a backup script for a Subversion repository that runs in a Debian-based VServer. I previously discussed svn backupsand my script was using the same svnadmin dump technique I mentioned in that post.

When I ran the backup script manually, everything worked as intended and it produced a ~1.3GB backup file; however, when it ran under cron, the backup file was only ~2MB. Since the simplest solution is usually the correct one, I didn't initially suspect a bug in cron as the problem and tried to troubleshoot my script/environment. I eventually found the two links I listed above, where others have encountered the same bug.

It appears that the bug in cron causes it to silently fail when too much output is directed to STDERR without an MTA being present. In the case of svn backups, using the "quiet" flag (-q) for svnadmin dump is a possible workaround. In other cases, either redirecting STDERR to /dev/null or setting MAILTO="" in /etc/default/cron seem to be valid workarounds.

In December 2009, Google publicized their new public DNS resolution service; a week later, they also announced the open-source release of a DNS benchmarking tool called namebench, a DNS benchmarking tool. It is designed to "hunt down the fastest DNS servers available for your computer to use."

Since web traffic is measured in milliseconds, delays of 200-400ms for slow DNS resolution can be more meaningful than it would seem. Using namebench to benchmark a variety of DNS servers, it is possible to speed up browsing by finding a faster DNS resolver.

You could write a script to test DNS resolution via nslookup, or a similar query tool; however, namebench includes several interesting features: it uses your browser history to select a list of hosts to test against, appropriately handles multiple IPs of cache-sharing resolvers, and reports on resolvers that use DNS hijacking. At the end of its benchmarking process, namebench suggests which DNS resolver seems to be the fastest, and displays useful statistical information about the test results.