As a follow on to my post about choosing RAID, here is a quick guide to setting up RAID-5 using mdadm. In theory this should apply to any distro that has mdadm, but I was already running Ubuntu 9.10 (just upgraded yesterday actually!).

Note: RAID-5 requires a minimum of 3 drives, and all should be the same size. It provides the ability for one drive to fail without any data loss.

Usable space = (no. of drives - 1) * size of smallest drive

In my set up I started with 3x 1.5TB drives, giving 3.0TB usable space. I have now grown it to 4x 1.5TB drives, giving 4.5TB usable space.

Required software

To create, format and resize the array we need to use various filesystem manipulation tools, but they should all come with Ubuntu. The only software you should need to install is mdadm.

sudo aptitude install mdadm

Initial setup

1. Preparing the drives

Before we jump into creating the actual RAID array, we first need to prepare partitions for the array to use. This can be done using a GUI tool such as GParted (available in the Ubuntu repositories), but I prefer using the terminal.

To get a list of available drives/partitions:

sudo fdisk -l

This will output, for each drive you have, something along the lines of:

Disk /dev/sda: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000f1f05

There may well be slightly more information available too if you already have some partitions made. From here note the name of the drives you wish to use (for example, /dev/sda). For each drive, create a partition and mark it as a RAID partition.

sudo fdisk /dev/sda

This will open up fdisk, the partition manager. If you already have any partitions on the drive you should first delete them (obviously this will erase any data on them!). To create one partition that is the size of the whole drive:

Command (m for help): n
Command action
	e	extended
	p	primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-182401, default 1): [blank]
Last cylinder or +size or +sizeM or +sizeK (1-182401, default 182401): [blank]

When choosing the first and last cylinder simply hitting return will use the default values (which are probably the values you actually want).

Next we will mark the partition as being part of a RAID array, allowing mdadm to automatically detect it:

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid auto)

So far our changes haven’t actually been written to disk, so finally, issue the command to write the changes to disk.

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Remember to repeat this step for each drive you wish to use in the array.

2. Creating the array

To create the array, we use the mdadm create flag (who’d have guessed that!). We also need to specify what RAID level we want, as well as how many devices and what they are.

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1

The verbose flag tells it to output extra information. In the above command I am creating a RAID-5 array at /dev/md0, using 3 partitions. If you already have a RAID array set up then you may need to use /dev/md1 for example. The number of partitions you are using, and their names will probably be different, so do not just copy and paste all of the command above. Note that the partition name is something like /dev/sda1, whereas the drive name is something like /dev/sda; the 1 refers to the partition number.

While the array is being built you can view its status in the file /proc/mdstat. Here the watch command comes in handy:

sudo watch cat /proc/mdstat

This will output the contents of the file to the screen, refreshing every 2 seconds (by default). While the array is being built it will show how much of the “recovery” has been done, and an estimated time remaining.

Now that we have set up the array, we need to edit the mdadm configuration so it knows how to reassemble it when the system boots.

sudo nano /etc/mdadm/mdadm.conf

There should already be some sample configuration here:

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=31283299:0c118170:8b4eb9a8:4e935bf2

The ARRAY line probably doesn’t already exist. It is used to describe an array (such as the one we just created). While similar, your array will not be the same as mine above so do not just copy and paste it! Luckily, mdadm provides a utility to fetch this line exactly (by looking for partitions marked as belonging to a RAID array, as we did when preparing the drives):

sudo mdadm --detail --scan

If you add the –verbose flag here it will also output the devices which are part of the array. Because we marked the partitions as being part of a RAID array mdadm can find them automatically so this isn’t required in our case, but it may be if you have more than one RAID array in your machine.

Now, I decided to wait for the array to finish syncing before moving on, but you should be able to continue and format the array while it syncs if you wish. In my case, building the array with 3x 1.5TB drives took around 6 hours.

3. Creating and mounting the filesystem

Now that the array is built we need to format it. What filesystem you choose is up to you, but I would probably recommend ext3. Note: For my array I chose to use ext4. So far I have not had any problems, but ext4 is still fairly new and there are still various bug reports going around mentioning data loss [bugzilla.kernel.org] (which I conveniently didn’t notice until after!).

sudo mkfs.ext3 /dev/md0

This will take a while, especially if your array is large.

If you chose to use ext2/3/4 you should also be aware of reserved space. By default ext2/3/4 will reserve 5% of the drives space, which only root is able to write to. This is done so a user cannot fill the drive and prevent critical daemons writing to it, but 5% of a large RAID array which isn’t going to be written to by critical daemons anyway, is a lot of wasted space. I chose to set the reserved space to 0%, using tune2fs:

sudo tune2fs -m 0 /dev/md0

Next we should add the array to the fstab, so that it will automatically be mounted when the system boots up. This can be done by editing the file /etc/fstab. For more detailed information, see the fstab Wikipedia article.

sudo nano /etc/fstab

Your fstab should already contain a few entries (if it doesn’t something is wrong!). At the bottom add a line similar to the following:

/dev/md0	/mnt/raid	ext4	defaults	0	0

I chose to mount my array on /mnt/raid, but you may well wish to mount it somewhere else. If the folder you chose doesn’t exist you will need to create it. As I said earlier I chose to use ext4, but here you will need to enter whatever filesystem you chose earlier.

Now, mount the array.

sudo mount -a

This will mount anything mentioned in the fstab that isn’t currently mounted but should be (hopefully your array!).

You should now have a working RAID-5 array! Next I would strongly suggest you investigate how to look after and monitor it, being able to cope with one drive failure is no use if you don’t notice when it fails and let a second fail!

Growing

One of the advantages of software RAID is the flexibility it gives you, that would normally only be available from high end (expensive) RAID cards. This includes the ability to grow an existing array (only for certain RAID levels), which means if you run out of space you can easily plug in a new drive and keep going.

1. Growing the array

Growing a RAID-5 array with mdadm is a fairly simple (though slow) task. First you will need to prepare the new drive in the same we we prepared the initial drives (step 1, above). To start the actual growing of the array we then add the new drive to the array as a spare:

sudo mdadm --add /dev/md0 /dev/sdd1

Then we grow the array onto this device. Because I had 3 drives before, the new drive obviously makes 4. Make sure to change this to whatever number you now have!

sudo mdadm --grow /dev/md0 --raid-devices=4

As when creating the array, we can follow the progress by checking the file /proc/mdstat.

sudo watch cat /proc/mdstat

Again, I decided to wait for the operation to finish before attempting to grow the filesystem, but you should be able to do it while the array syncs.

2. Growing the filesystem

Before we can grow the filesystem we need to unmount it:

sudo umount /dev/md0

Now that it is unmounted we can run a filesystem check to make sure everything is in order:

sudo fsck.ext3 -f /dev/md0

If any issues arise, let it attempt to fix them for you.

Once the filesystem has been checked we can perform the resize. If you decided to use something other than ext2/3/4 this may be done differently, but for the ext2/3/4 filesystems we use the resize2fs tool.

sudo resize2fs /dev/md0

This will automatically grow the filesystem to the new size of the device.

You can now remount the drive and start filling it up again!

Sources

• Software RAID 5 in Ubuntu with mdadm
• Growing a RAID5 array – MDADM
• Linux Raid Wiki
• Recovering reserved space in ext2 and ext3 filesystems

If you spot any errors, or have any problems then please leave a comment and let me know!

Having been virtually out of hard drive space for the past couple of months I finally decided to do something about it, and purchased 3 new 1.5TB Seagate Barracuda hard drives. I chose these because I already had one and thought 4 the same would make sense, but if you are buying all new then the 1.5TB Western Digital Caviar Green is maybe a better option; it is 5400rpm rather than 7200rpm, the performance loss shouldn’t be an issue for a basic file server, but the power saving and quieter operation may be appreciated.

I decided to create a software RAID-5 array using the 4 drives, to give a total capacity of 4.5TB with the ability to handle 1 drive failure.

What is RAID?

RAID stands for Redundant Array of Inexpensive Disks, and is a way of organizing data over multiple drives to provide higher performance and/or reliability than available from a single drive.

What is RAID not?

Although RAID is redundant and can recover from drive failure, it alone is not a backup solution! If you delete a file on a RAID array it is deleted from all drives in that array, similarly if your data is corrupted then the corrupted data will be mirrored across the whole array.

Hardware RAID vs Fake RAID vs Software RAID

Before getting started you should think about how best to implement your RAID array, because once it’s done it can be hard (if not impossible without enough spare disks to back up all your data) to change.

Hardware RAID

Hardware RAID is usually considered the “real” implementation of RAID. It requires a dedicated RAID card, which handles maintaining the array; this has the advantage that the CPU is not required so performance is usually better. The disadvantages however are that you are then restricted to the RAID card you have; the implementations of RAID isn’t always compatible between different models, and almost definitely isn’t between different vendors, and the cost; hardware RAID is not cheap!

Fake RAID

Most motherboards these days come with RAID support, which is known as Fake RAID. What’s fake about it? There is no dedicated chip to maintain the array as with “real” RAID. This means that the processing is handed off to the CPU, so the performance benefit is no longer there. On the upside, it is cheap!

Software RAID

The third option when implementing RAID is Software RAID, which, as the name suggests, is done totally in software. This means, like with Fake RAID, the CPU is required to do the processing. The advantages however are it’s portability, and cost (free!). Because the implementation is entirely software based, there is no restriction on hardware any more; you can move the drives between computers without any problems. There is however now a restriction on software; if you created the RAID array under Linux (using mdadm), it will not be accessible under Windows (or any other OS that doesn’t have the same implementation of mdadm available). It is also worth mentioning that mdadm includes many features which can only be found on the high end (expensive) RAID cards, such as the ability to grow or shrink a live array.

Levels of RAID

Another thing you will need to decide on before starting is what level of RAID you require. For a detailed review of the different levels see the Wikipedia article about Standard RAID levels. I am using RAID-5 because it provides some redundancy but without sacrificing too much capacity (usable size = (num of drives – 1) * size of smallest drive), however there is some speculation that as disk sizes are increasing, RAID-5 is becoming less reliable. When I next get paid I may consider upgrading to RAID-6 (which, using software RAID, is easy!).

Follow up: Software RAID-5: using mdadm in Ubuntu 9.10.