Creating a flying Linux-NAS

Submitted by tschinke on Thu, 04/16/2009 - 01:58

Today I tried to set up a system which trades of data security and performance using Ubuntu. That's the setup:

Hardware

Gigabyte GA/MA69V-S2 running an AMD 690V chipset
Athlon X2 4200+
4 GB of RAM
8GB USB-Stick
4x WD EADS 1TB HDDs

Software

Ubuntu Linux 8.10 64bit edition on USB-Stick
mdadm RAID5 configuration
JFS as file system

Tutorial for setting up the RAID5

First you have to install mdadm

sudo apt-get install mdadm

After this use cfdisk to create Linux Raid Partition on your disk. cfdisk is the same like fdisk, but much more comfortable. Create first a primary partition (which is then Linux) and modify the type to Linux Raid.

sudo cfdisk

Now we are ready to fire up our raid, using mdadm

 sudo mdadm --create --verbose /dev/md0 --chunk=128 --level=5 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

As you can see, I changed the chunk size from 64k up to 128k. There are sites out there which states to use 1024k, but this degrades my performance down to 30-40MB/s. Please make sure you really use the partitions (sda1 etc) of your devices. Note that there is maybe a bug, so mdadm driver was not loaded, so you get a message like mdadm: error opening /dev/md0: No such device or address. For this start the service by hand

sudo modprobe md

Now you can go away and do something more important, because the array is now syncing and lasts for hours (depends on your drives performance, e.g. 3 hours). You can take a look at the status with

cat /proc/mdstat

If array has synced successfully we can go on, without any disturbance. You can choose a file system of your choice. I tried ext3 and jfs on several systems, and the most stable one was jfs for me, but this maybe only due to a personal taste.

 sudo jfs_mkfs /dev/md0

Now we are ready to mount and for creation of a test-directory

sudo mkdir /mnt/dataRaid5
sudo mount /dev/md0 /mnt/dataRaid5/
sudo mkdir /mnt/dataRaid5/common
sudo chmod g+rw /mnt/dataRaid5/common

Fixating mdadm configuration

First we have to make sure that our md kernel module ist loaded at startup, to avoid modprobe command. Open the modules config and add the md entry.

sudo gedit /etc/modules

If you rebootet already or later boot up with another system, which don't know about your raid, then you can reassemble your raid automatically with this command

sudo mdadm --assemble --scan

Instead you can try it by hand

sudo mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

Now we will write a new mdadm.conf file like this

sudo su
mdadm --detail --scan > /etc/mdadm/mdadm.conf

Don't forget to ensure that the monitor-service is configured properly using

sudo dpkg-reconfigure mdadm

But sadly I can't get the stuff to assemble automatically at startup. I think this is due to a wrong order, e.g. loading md module earlier then udev devices or either the sata stuff. My workaround for this consists of using a new start-script, which you can find at the bottom. So copy it to /etc/init.d and tell Ubuntu, that there is a new script and modify the rights for execution

sudo update-rc.d my_raid defaults
sudo chmod 755 /etc/init.d/my_raid

Pimp it up

extend the stripe cache to 32mb and the readahead cache to 256mb

sudo echo 32768 > /sys/block/md0/md/stripe_cache_size 
sudo blockdev --setra 262144 /dev/md0

Performance

hdparm

A simple test with hdparm is showing some impressive results

sudo hdparm -tT /dev/md0
/dev/md0:
 Timing cached reads:   1758 MB in  2.00 seconds = 879.20 MB/sec
 Timing buffered disk reads:  658 MB in  3.00 seconds = 219.03 MB/sec

Copy and Paste

This is not really measured, just copied some gigabytes of data within the raid. This will reach an through output of 67MB/s (so an absolute IO of 134MB/s)

bonnie++

A synthetic benchmark with bonnie

cd /mnt/dataRaid5/common
bonnie++ -q >results.csv
cat results.csv | bon_csv2html >results.html

You can grab the results as attachment at the end of this post, but in short the results for sequential reads are about 222mb/s by block and sequential write will reach about 175mb/s by block. Rewrites degrades to 106mb/s.

Samba

Samba transfer rate are still horrible. Transferring from/to another ubuntu 8.10 won't reach more then 30MB/s. This is not due to the cheap realtek onboard solutions (read/write 57/74 MB/s).

Conclusion

The Bonnie benchmarks shows a good performance for reading sequential blocks (222 MB/s) which is quite reasonable, assuming one drive reaches about 80MB/s => 3x80MB/s=240MB/s. But writing sequential blocks still lacks of performance because of only reaching 175MB/s. Reason may be CPU parity calculation if it won't make use of multiple cores.

Sadly samba performance is still unusable because of a read/write performance just being around 30MB/s. This may be due to a slow disk at the client side (nc test degrades to 46MB/s when writing to disk, inluding smb-overhead 30 MB/s may be reasonable).

Attachment	Size
my_raid	266 bytes
results.html	2.85 KB

tschinke's blog