Bob Feldbauer - Sysadmin - CompleteFusion

Zero-Downtime Deployments with MySQL

I've been talking a lot about zero-downtime deployments with MySQL as your database backend (including schema changes/migrations!) recently:

I gave a 45 minute presentation called Zero Downtime Deployments with Database Migrations, from a DevOps conference that Vladimir Vuksan and I organized back in November 2012:

Additionally, I contributed a post on the same subject, titled Zero-Downtime MySQL Schema Changes, to this year's SysAdvent.

Feel free to drop me an email if you need help with implementing zero-downtime deployments or schema changes in your own applications.

Finally, here are a few more useful resources:

Etsy: Deployments using Master-Master MySQL
High Performance MySQL, "Replication" (Ch. 10 in 3rd edition, Ch. 8 in 2nd edition)
MySQL Load Balancing with HAProxy

January 11, 2013 0 comments Read More

Setup lxc on Debian

Using low-overhead virtualization methods like chroots, containers, etc. is a great way to gain most of the benefits of virtualization without the overhead of traditional hypervisor-based virtualization. Since 2011, lxc has become the standard, kernel-integrated lightweight virtualization method.

Setting Linux containers, or lxc, up on Debian is very straightforward:

Install lxc and bridge-utils

apt-get install lxc bridge-utils

Setup cgroup mount point

mkdir -p /cgroup
echo "cgroup /cgroup cgroup defaults 0 0" >> /etc/fstab
mount /cgroup

Make sure you have IP forwarding enabled

echo 1 > /proc/sys/net/ipv4/ip_forward

Modify your networking configuration

We'll setup an internal 10.0.0.1 network using dummy0 as the bridged device by adding the following to your /etc/network/interfaces:

auto dummy0
iface dummy0 inet static
  address 10.0.0.1
  netmask 255.255.255.0

auto br0
iface br0 inet static
  max_wait 0
  bridge_ports dummy0
  address 10.0.0.1
  netmask 255.255.255.0

Run the container setup script

Usage: lxc-debian.sh <distribution> <container-name> <container-number>

wget https://gist.github.com/raw/3008518/e3b87deb423ace3c67628fe501af79d46d9de04c/lxc-debian.sh
chmod +x lxc-debian.sh
./lxc-debian.sh wheezy test 2

Start the container and enter the console
(user and password are both root)

lxc-start -n test -d
lxc-console -n test

Optional: On the host, setup iptables rules to route to and from the container

This will route all traffic from the container out through the host's eth0, and route incoming traffic on port 10080/10443 on the host to port 80/443 on the container for a web server.

HOSTIP=$(ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}')
DESTIP=$(grep address /var/lib/lxc/test/rootfs/etc/network/interfaces | awk '{print $2}')
iptables -t nat -A POSTROUTING -s 10.0.0.1/24 -j SNAT --to-source $HOSTIP
iptables -t nat -A PREROUTING -m tcp -p tcp --dport 10022 -j DNAT -i eth0 --to-destination $DESTIP:80
iptables -t nat -A PREROUTING -m tcp -p tcp --dport 10443 -j DNAT -i eth0 --to-destination $DESTIP:443

June 27, 2012 0 comments Read More

SysAdvent – Website Performance Optimization

Before 2010 comes to a close, be sure to check out the sysadmin's Advent calendar, SysAdvent! Last year, I contributed an article about ATA over Ethernet (AoE), and I wrote about Website Performance Optimization this year.

December 7, 2010 1 comment Read More

SSL load balancing with HAProxy and stunnel on Debian

HAProxy is a great load balancer; however, it doesn't natively terminate SSL connections. Fortunately, stunnel can terminate SSL connections and seamlessly forward them to HAProxy. To be able to log the client's IP address, we'll need to patch stunnel to provide X-Forwarded-For headers to HAProxy.

The following instructions are based on Debian Squeeze, HAProxy 1.4.8-1, and stunnel 4.32.

To get started, download the stunnel source and X-Forwaded-For patch:

wget http://www.stunnel.org/download/stunnel/src/stunnel-4.32.tar.gz
wget http://haproxy.1wt.eu/download/patches/stunnel-4.32-xforwarded-for.diff

Before building stunnel, we need to install a few other packages:

apt-get install build-essential libssl-dev haproxy

To install stunnel:

tar -zxvf stunnel-4.32.tar.gz
cd stunnel-4.32
patch -p1 < ../stunnel-4.32-xforwarded-for.diff
./configure
make && make install
 
adduser stunnel
ln -s /usr/local/etc/stunnel /etc/stunnel
touch /var/log/stunnel.log
chown -R stunnel:stunnel /usr/local/etc/stunnel
chown -R stunnel:stunnel /var/run/stunnel
chown stunnel:stunnel /var/log/stunnel.log

Then we have to create the necessary files in /etc/stunnel: stunnel.conf, stunnel.pem, and the SSL certificate (server.crt) and key file (server.key). The server.crt and server.key are the SSL certificate and key which would normally go on a web server. The following stunnel.conf will accept incoming HTTPS connections on port 443, then forward it to port 81 as regular HTTP traffic – just change the ip address from 192.168.1.1 to your own.

Example /etc/stunnel/stunnel.conf:

cert=/etc/stunnel/stunnel.pem
setuid=stunnel
setgid=stunnel
pid=/var/run/stunnel/stunnel.pid
output = /var/log/stunnel.log
 
socket=l:TCP_NODELAY=1
socket=r:TCP_NODELAY=1
 
[https]
  cert=/etc/stunnel/server.crt
  key=/etc/stunnel/server.key
  accept=192.168.1.1:443
  connect=192.168.1.1:81
  xforwardedfor=yes

Next, we'll need to create our stunnel certificate file, stunnel.pem. More details about the certificate creation process are available at Using Certificates with stunnel, but you can just run the following command:

openssl req -new -x509 -days 365 -nodes -config /etc/stunnel/stunnel.conf -out /etc/stunnel/stunnel.pem -keyout /etc/stunnel/stunnel.pem

Finally, here's an example haproxy.cfg file (/etc/haproxy/haproxy.cfg) – just change your web server IPs:

global
        log 127.0.0.1 local0
        log 127.0.0.1 local1 notice
        user haproxy
        group haproxy
        daemon
        maxconn 20000
 
defaults
        log global
        option dontlognull
        balance leastconn
        clitimeout 60000
        srvtimeout 60000
        contimeout 5000
        retries 3
        option redispatch
 
listen http 192.168.1.1:80
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.1.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ http
        server webserver1 192.168.1.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.1.3 cookie webserver2 maxconn 5000
 
listen https 192.168.1.1:81
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.1.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ https
        server webserver1 192.168.1.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.1.3 cookie webserver2 maxconn 5000

September 21, 2010 4 comments Read More

Lower memory usage on OpenVZ VPS

I recently migrated CompleteFusion to a new hosting provider, which meant switching from a Xen-based VPS to OpenVZ. Although OpenVZ VPS hosting is generally much less expensive, it often lacks swap space and requires special consideration of memory usage. Running OpenSSH, MySQL, and Apache2 with mod_php5 caused out of memory problems, even with 512MB RAM.

First, I replaced OpenSSH with Dropbear, which uses about 50% less memory while providing very similar SSH2 server capabilities. On Debian, installing dropbear was simple:

apt-get install dropbear
sed -i 's/NO_START=0/NO_START=1/g' /etc/default/dropbear
/etc/init.d/ssh stop && /etc/init.d/dropbear start

Although I could have made various configuration changes to tune Apache2 to use less memory, switching to a stock lighttpd configuration with fastcgi was enough to reduce memory usage to acceptable levels. I installed lighttpd on Debian as follows:

apt-get install lighttpd
apt-get install php5-cgi
echo "cgi.fix_pathinfo = 1" >> /etc/php5/cgi/php.ini
lighttpd-enable-mod fastcgil
/etc/init.d/lighttpd force-reload

On my OpenVZ VPS, I ran into a problem with starting lighttpd, where it reported that port 80 was in use, even though it wasn't. As it turns out, it was apparently because my container lacked IPv6 support. I resolved it by commenting out the following line in /etc/lighttpd/lighttpd.conf:

#include_shell "/usr/share/lighttpd/use-ipv6.pl"
These changes should lower your memory usage enough to run well inside of 512MB RAM, but additional configuration will probably be needed if you only have 256MB or less.

June 17, 2010 0 comments Read More

Migrating from subversion to git

When I embarked on my first subversion to git migration recently, I thought it would be fairly easy — after all, migrating from CVS to subversion wasn't very difficult. Unfortunately, it wasn't as easy as I had thought, because I couldn't readily find reliable information to guide me. Since I had to spend entirely too time searching and testing various migration methods, I hope this post will help you avoid a similar fate.

The first step in migrating a repository from subversion to git is to get a list of committers from svn. There's no native way of requesting that information from subversion, but you can use svn log and grep the lines with revision data. If your subversion repository is on a remote host, you can use http to retrieve it and list all subversion commiters:

svn log --quiet http://www.foo.com/path/to/svn/ | grep '^r' | awk -F\| '{print $2}' | sort -u > committers.txt

If your subversion repository is hosted locally, just use file:///path/to/svn (note the three slashes in file:///, instead of two for http://).

After you have the list of subversion committers, create a users.txt file to map svn committers to git users, using the following format:

svnuser = Full Name <email@domain.com>

Although some people suggest using a basic git-svn migration or the svn2git scripts, I found the git-svn-abandon scripts to be the easiest and most sane. After downloading git-svn-abandon and putting the scripts somewhere in your PATH, you can start the migration process:

git svn clone --prefix=svn/ --stdlayout --authors-file=users.txt http://www.foo.com/path/to/svn/

Then cd into the directory the git-clone command just created and run:

git svn-abandon-fix-refs git svn-abandon-cleanup

Finally, clone the new repository to make it sharable and configured for use:

cd .. git clone --bare /path/to/git_svn/repo /path/to/shared_git/repo.git cd /path/to/shared_git/repo.git git --bare update-server-info mv hooks/post-update.sample hooks/post-update

For more information, take a look at the git-svn-abandon author's article about Migrating from Subversion to Git.

April 28, 2010 0 comments Read More