Post Tagged with: "linux"

Setup lxc on Debian

Setup lxc on Debian

Using low-overhead virtualization methods like chroots, containers, etc. is a great way to gain most of the benefits of virtualization without the overhead of traditional hypervisor-based virtualization. Since 2011, lxc has become the standard, kernel-integrated lightweight virtualization method.

Setting Linux containers, or lxc, up on Debian is very straightforward:

Install lxc and bridge-utils

apt-get install lxc bridge-utils

Setup cgroup mount point

mkdir -p /cgroup
echo "cgroup /cgroup cgroup defaults 0 0" >> /etc/fstab
mount /cgroup

Make sure you have IP forwarding enabled

echo 1 > /proc/sys/net/ipv4/ip_forward

Modify your networking configuration

We'll setup an internal 10.0.0.1 network using dummy0 as the bridged device by adding the following to your /etc/network/interfaces:

auto dummy0
iface dummy0 inet static
  address 10.0.0.1
  netmask 255.255.255.0
 
auto br0
iface br0 inet static
  max_wait 0
  bridge_ports dummy0
  address 10.0.0.1
  netmask 255.255.255.0

Run the container setup script

Usage: lxc-debian.sh <distribution> <container-name> <container-number>

wget https://gist.github.com/raw/3008518/e3b87deb423ace3c67628fe501af79d46d9de04c/lxc-debian.sh
chmod +x lxc-debian.sh
./lxc-debian.sh wheezy test 2

Start the container and enter the console
(user and password are both root)

lxc-start -n test -d
lxc-console -n test

Optional: On the host, setup iptables rules to route to and from the container

This will route all traffic from the container out through the host's eth0, and route incoming traffic on port 10080/10443 on the host to port 80/443 on the container for a web server.

HOSTIP=$(ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}')
DESTIP=$(grep address /var/lib/lxc/test/rootfs/etc/network/interfaces | awk '{print $2}')
iptables -t nat -A POSTROUTING -s 10.0.0.1/24 -j SNAT --to-source $HOSTIP
iptables -t nat -A PREROUTING -m tcp -p tcp --dport 10022 -j DNAT -i eth0 --to-destination $DESTIP:80
iptables -t nat -A PREROUTING -m tcp -p tcp --dport 10443 -j DNAT -i eth0 --to-destination $DESTIP:443
June 27, 2012 0 comments Read More
SSL load balancing with HAProxy and stunnel on Debian

SSL load balancing with HAProxy and stunnel on Debian

HAProxy is a great load balancer; however, it doesn't natively terminate SSL connections. Fortunately, stunnel can terminate SSL connections and seamlessly forward them to HAProxy. To be able to log the client's IP address, we'll need to patch stunnel to provide X-Forwarded-For headers to HAProxy.

The following instructions are based on Debian Squeeze, HAProxy 1.4.8-1, and stunnel 4.32.

To get started, download the stunnel source and X-Forwaded-For patch:

wget http://www.stunnel.org/download/stunnel/src/stunnel-4.32.tar.gz
wget http://haproxy.1wt.eu/download/patches/stunnel-4.32-xforwarded-for.diff

Before building stunnel, we need to install a few other packages:

apt-get install build-essential libssl-dev haproxy

To install stunnel:

tar -zxvf stunnel-4.32.tar.gz
cd stunnel-4.32
patch -p1 < ../stunnel-4.32-xforwarded-for.diff
./configure
make && make install
 
adduser stunnel
ln -s /usr/local/etc/stunnel /etc/stunnel
touch /var/log/stunnel.log
chown -R stunnel:stunnel /usr/local/etc/stunnel
chown -R stunnel:stunnel /var/run/stunnel
chown stunnel:stunnel /var/log/stunnel.log

Then we have to create the necessary files in /etc/stunnel: stunnel.conf, stunnel.pem, and the SSL certificate (server.crt) and key file (server.key). The server.crt and server.key are the SSL certificate and key which would normally go on a web server. The following stunnel.conf will accept incoming HTTPS connections on port 443, then forward it to port 81 as regular HTTP traffic – just change the ip address from 192.168.1.1 to your own.

Example /etc/stunnel/stunnel.conf:

cert=/etc/stunnel/stunnel.pem
setuid=stunnel
setgid=stunnel
pid=/var/run/stunnel/stunnel.pid
output = /var/log/stunnel.log
 
socket=l:TCP_NODELAY=1
socket=r:TCP_NODELAY=1
 
[https]
  cert=/etc/stunnel/server.crt
  key=/etc/stunnel/server.key
  accept=192.168.1.1:443
  connect=192.168.1.1:81
  xforwardedfor=yes

Next, we'll need to create our stunnel certificate file, stunnel.pem. More details about the certificate creation process are available at Using Certificates with stunnel, but you can just run the following command:

openssl req -new -x509 -days 365 -nodes -config /etc/stunnel/stunnel.conf -out /etc/stunnel/stunnel.pem -keyout /etc/stunnel/stunnel.pem

Finally, here's an example haproxy.cfg file (/etc/haproxy/haproxy.cfg) – just change your web server IPs:

global
        log 127.0.0.1 local0
        log 127.0.0.1 local1 notice
        user haproxy
        group haproxy
        daemon
        maxconn 20000
 
defaults
        log global
        option dontlognull
        balance leastconn
        clitimeout 60000
        srvtimeout 60000
        contimeout 5000
        retries 3
        option redispatch
 
listen http 192.168.1.1:80
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.1.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ http
        server webserver1 192.168.1.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.1.3 cookie webserver2 maxconn 5000
 
listen https 192.168.1.1:81
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.1.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ https
        server webserver1 192.168.1.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.1.3 cookie webserver2 maxconn 5000
September 21, 2010 4 comments Read More
cron bug: silently fails when too much output is produced

cron bug: silently fails when too much output is produced

Edited: I reported this issue to Debian and Christian Kastner patched it. Debian cron versions 3.0pl1-110 and higher should behave properly.

As reported in Ubuntu bug #151231 and at stackoverflow, cron jobs can fail silently when too much output is produced and an MTA is not installed.

Although the comments in the Ubuntu bug indicate that Debian is not affected because it includes the exim MTA by default, using a Debian VServer only installs the most basic packages – no MTA. I encountered this bug today while I was working on a backup script for a Subversion repository that runs in a Debian-based VServer. I previously discussed svn backupsand my script was using the same svnadmin dump technique I mentioned in that post.

When I ran the backup script manually, everything worked as intended and it produced a ~1.3GB backup file; however, when it ran under cron, the backup file was only ~2MB. Since the simplest solution is usually the correct one, I didn't initially suspect a bug in cron as the problem and tried to troubleshoot my script/environment. I eventually found the two links I listed above, where others have encountered the same bug.

It appears that the bug in cron causes it to silently fail when too much output is directed to STDERR without an MTA being present. In the case of svn backups, using the "quiet" flag (-q) for svnadmin dump is a possible workaround. In other cases, either redirecting STDERR to /dev/null or setting MAILTO="" in /etc/default/cron seem to be valid workarounds.

April 8, 2010 0 comments Read More
Recover an Overwritten File on ext3 File System

Recover an Overwritten File on ext3 File System

I've needed to recover deleted files on ext3, FAT, and NTFS file systems in the past, but I recently needed to recover a previous version of a text file I had overwritten by editing and saving it. I initially thought I might be able to recover it either by accessing the inode used by the previous version of the file, or by looking at ext3's journal.

Unfortunately, I had used nano to edit the file. Apparently, nano saves files by truncating and overwriting the file, reusing the same inode. Also, I quickly realized ext3's journal wouldn't help because my file system was mounted using data=ordered, not data=journal. From the ext3 FAQ:

  1. data=journal: Journals all data and metadata, so data is written twice.
  2. data=ordered: Only journals metadata changes.

Ultimately, I was able to recover the file with some help from stat, debugfs, and blkls from The Sleuth Kit. Before getting started, you'll need to install The Sleuth Kit. On Debian, it is available as a package, so: apt-get install sleuthkit

First, check the inode being used by the file: stat file.txt | grep Inode

This should return a line containing the inode, like: Inode: 1474575

Next, backup the file, then delete it:
cp file.txt file.old
rm file.txt

Run debugfs /dev/sda1, replacing /dev/sda1 with the hard drive the file is on. From the debugfs CLI, run stats and check its output for "Blocks per group". On my system, and most of the time, this is 32768. While still in the debugfs CLI, run imap <inode> to get the block: imap <1474575>. In my case, the block was 5898242.

Once you know the block the inode is in, and the number of blocks per group, create a block range: 5898242+32768-1 and use blkls to copy the block to a file: blkls /dev/sda1 5898242-5931009 > tmp.dat

Finally, open tmp.dat in your favorite text editor or use grep to search for the overwritten version of your file.

For more details about ext3 file systems and recovering deleted files:

  1. Recovering Deleted Files on an ext3 File System
  2. Data Recovery on Linux and ext3
February 17, 2010 3 comments Read More
Extending an LVM Logical Volume

Extending an LVM Logical Volume

As a followup to my AoE storage server posts (Part 1, Part 2), I recently needed to extend an LVM logical volume from 1TB to 1.3TB. As per the LVM howto, I followed these steps:

On the "client" (the server that had mounted the AoE share):
umount /backups

On the storage server:
vblade-persist stop 0 1
lvextend -L1331GB /dev/data1/backups
e2fsck -f /dev/data1/backups
resize2fs /dev/data1/backups
vblade-persist start 0 1

On the "client":
aoe-stat
mount /dev/etherd/e0.1 /backups

July 26, 2009 0 comments Read More
Backup a Subversion Repository with svnadmin dump

Backup a Subversion Repository with svnadmin dump

I recently created a new Subversion repository for a new personal project, and realized I needed to add it to my daily backup script. After a quick bit of searching via Google to refresh my memory, I found exactly what I thought I was looking for:

svnadmin hotcopy /svn /backups/svn --clean-logs

Indeed, O'Reilly's Version Control with Subversion states that svnadmin hotcopy will make a hot copy of a repository, and "is able to make a safe copy of the repository at any time, regardless if other processes are using the repository." However, further research seemed to indicate that the following may be a better option:

svnadmin dump /svn > /backups/svndump-`date %Y%m%d`.dmp

According to several posts on the Subversion mailing list, svnadmin's hotcopy "theoretically needs the correct machine architecture, version of bdb (if it's a bdb repo), and version of Subversion to work"; however, svnadmin's dump could "theoretically be loaded by any future version of Subversion". Whether or not that's actually true, since there are numerous "theoretical" qualifiers, I decided to go with svnadmin dump.

June 3, 2009 2 comments Read More