CompleteFusion

In my last post, I mentioned I would be contributing to the SysAdvent calendar. Google Analytics seems to indicate my posts on ATA over Ethernet (AoE) have been pretty popular lately, so I decided to write about AoE for my SysAdvent article. Check it out: Storage: ATA over Ethernet.

No comments

As I mentioned in Debian NAS using AoE (Part 1), I recently needed to implement a low-cost, enterprise grade storage system for backups and to provide network attached storage for vserver-based Debian servers. I was able to build a suitable storage server with 13TB of raw storage for a reasonable price, and decided to use Debian with LVM2 and AoE to divide and share it across the network.

Before getting to the OS installation, I had to setup and build the RAID arrays in the HighPoint 3540 BIOS. I set up a RAID 1 mirror of the two 300GB drives to use for the OS. Then, I created a 6-disk RAID6 and a 7-disk RAID6, which left me with 4GB and 5GB of usage storage.

Next, I installed Debian Lenny from the Lenny netinst ISO onto the 300GB RAID 1 mirror I created, with no additional packages/package groups selected. After the installation was complete, I rebooted the server to install openssh server, lvm2, and AoE. Since I wanted to be able to use the storage server to run vserver instances if needed, I also installed the vserver kernel and tools:

apt-get update
apt-get install vserver-debiantools linux-image-2.6-vserver-amd64 lvm2 vblade vblade-persist aoe-tools

I configured LVM to recognize the physical volumes:

pvcreate /dev/sdb
pvcreate /dev/sdc

I created the volume groups data1 and data2 on the physical volumes:

vgcreate data1 /dev/sdb
vgcreate data2 /dev/sdc

I created two 1TB LVM logical partitions (/dev/data1/backups, /dev/data1/backups2):

lvcreate -L 1TB -n backups data1
lvcreate -L 1TB -n backups2 data1

I made the ext3 filesystem on them so they would be usable:

mkfs.ext3 /dev/data1/backups
mkfs.ext3 /dev/data1/backups2

I configured AoE (ATA over Ethernet):

modprobe aoe

vblade-persist setup 0 1 eth0 /dev/data1/backups
vblade-persist setup 0 2 eth0 /dev/data1/backups2

vblade-persist start 0 1
vblade-persist start 0 2

Finally, I verified they were up and available using:

aoe-discover
aoe-stat

Output from aoe-stat… :

e0.1 1099.511GB eth1 up
e0.2 1099.511GB eth1 up

To mount an AoE export on a remote server, I used the following (on the remote server):

modprobe aoe
apt-get install aoetools
aoe-discover
aoe-stat (should show the available AoE exports)
mkdir /mountpoint
mount /dev/etherd/e0.1 /mountpoint (replace e0.1 with the appropriate device from aoe-stat)

3 comments

Recently, I needed to implement a low-cost, enterprise grade storage system for backups and to provide network attached storage for vserver-based Debian servers. After considering the requirements and projected short-term future growth, I decided to build a storage server with at least 12TB of raw storage, which would allow me to create two 4TB RAID6 arrays.

Ultimately, I was able to get 13TB of raw storage for less than $6300 (~$484 per TB, compared to many vendors starting prices of $1000 or more per TB).

Hardware

1. Supermicro CSE-933T-R760B Black 3U Rackmount Case – $769.99
2. Supermicro MBD-X7DBN-O Dual LGA 771 Intel 5000P Extended ATX Motherboard – $359.99
3. 2x Intel Xeon E5405 Harpertown 2.0GHz LGA 771 80W Quad-Core Processor – $224.99 ($449.98 for 2)
4. 4x Kingston 8GB (2 x 4GB) 240-Pin DDR2 667 FB-DIMM ECC RAM – $212.99 ($851.96 for 4)
5. 13x Western Digital RE3 1TB 7200 RPM SATA Hard Drive – $189.99 ($2469.87 for 13)
6. 2x Western Digital VelociRaptor 300GB 10000 RPM Hard Drive – $229.99 ($459.98 for 2)
7. HighPoint RR3540 RAID Controller – $799.99

Total: $6161.76 (plus shipping)

After reviewing the current Linux distributions targeted towards NAS systems, like FreeNAS and OpenFiler, I decided to stick with a simple system using Debian with LVM2 and AoE. My next post, Debian NAS using AoE (Part 2), will dig into the details of the OS and software setup.

3 comments