Software RAID advice

[N3G4T1v3]

Hey all

So I'm building up another NAS and I was hoping someone could help with a recommendation of a software RAID implementation

At the moment, I'm running 2 HP N40L Micro-Servers with each running 4x2TB, using Snapraid for parity
Before, When I had just the one Micro-server, I was running 4x2TB in RAIDz1, and I loved it, but I was struggling with FreeBSD, which was my host OS, as I was really new to *nix in general, especially to the server world
I eventually got frustrated with the OS, and decided to move to Ubuntu, and exported the RAIDz array to Ubuntu, which worked very well
I eventually ran into an odd issue, when 2 of the drives began to fail due to bad sectors, I couldn't replaced the drive that I had removed with the new drive
When I eventually came right, and rebuilt the 2 new drives, but every time I rebooted the server, the 2 drives would resilver
So I became annoyed and opted for something different, and thought the WD greens would prefer being able to spin down during unused periods
So I went for Snapraid, but haven't enjoyed it at all, but it has been working

Anyway, that was a little off topic

So, my new system:

CPU: AMD A4 5300 [I'm considering the 5400k, but I don't think the 200MHz is worth the price bump]
Motherboard: ASrock FM2A85X Extreme4-M ATX
RAM: 2x4GB DDR1333
PSU: Gigabyte GT 850W
HDD: 4x3TB WD RED

The file server will be mostly used to stream files, and store backups of software and what not, and I would really like to move back to RAID5 again.
What I'm hoping to do this time around to expand my array as I require more space, as now I have a case that has enough space, and it would suit my budget better to be able to just buy an extra drive when space become low
I may eventually increase the array to 7 drives in total

I would love to use ZFS again, but you can't expand an array, and the issues I had on ubuntu makes me nervous (Although, I would probably move to another OS if I were), but the features and simplicity of use from ZFS was A+
Snapshot raid seemed like a good idea for WD greens, as I don't think they liked the 24/7 operations, lol

So that leaves me with MDADM
I've looked into it basically, and it seems it does what I'm looking for, but there seem to be a fair amount of variables that I need to look at, it's definitely more complex than setting up a ZFS array
Seeing that this array may exceed 16TB, what would the best filesystem to use? It seems the ext4 has limitations above 16TB, but read there was a 64bit extension for ext4 the allows for 16+TB or something like that, or what would be a better file system?
How does the file system respond to an array being grown? I've read that people generally tweak the file system for the array, when expanding it, can the setting be modified? Or how does the filesystem respond to the increase in array size?

It seems to me that MDADM can do what I would like, with some effort on the original setup
I was kind of considering ZFS, and creating a 2nd 3x3TB raidz1 vdev to add to the pool if I needed more space in the future, but I'm note sure, but it would be an option

 

[marsboer]

EXT4 works fine with filesystems above 16TB, but you need kernel version > 3.7 and e2fsprogs > v1.42.6 for online resizing to more than 16TB given that you created the filesystem with the 64bit flag.

If you are creating a fresh filesystem > 16TB I think you actually can create it out-of-the-box even with kernel 3.2 (I use Debian Stable with the default kernel on my 18TB ext4 fs)

mdadm is not hard. You don't need partitions or anything. Examples:

Create your array and watch the build progress:

mdadm --create /dev/md/raid6-0 --level=raid6 --raid-devices=8 /dev/sd[b-i]

watch -n5 cat /proc/mdstat
# alternatively
mdadm --detail /dev/md/raid6-0

Parity raid operations is RAM starved in the default configuration. You should increase stripe_cache_size for a healthy increase in write performance without integrity issues (this is not write cache). Put this in a boot file (you can change it on the fly) like /etc/rc.local (adapt the path to match your system). You should set this immidiately after having created the RAID as it will make the rebuild go a lot faster.

echo 8192 > /sys/block/md127/md/stripe_cache_size

The default is 256. This is the number of 4K blocks of RAM used PER DISK for parity calculations. 8192x4K = 32MB per disk in the array. More is not better. I do not get any performance increase from higher values, just waste of RAM.

If you have been messing around and set faulty disks and then recreated the array you will probably notice that there is no progress. Then you have to make the array writeable again:

mdadm --readwrite /dev/md/raid6-0

When the rebuild is complete you may create your filesystem:
64bit will be set automatically if the fs is >16TB, but if it is less you need to set it yourself to be able to resize the fs to >16TB later as this flag cannot be changed later:

mkfs.ext4 -m1 -O 64bit /dev/md/raid6-0

Then mount it as usual.

On Debian, and I presume modern Ubuntu, you need to add the array to /etc/mdadm/mdadm.conf and update initramfs to make the array boot automatically:

mdadm --examine --scan >> /etc/mdadm/mdadm.conf
update-initramfs -u

You should also configure a working SMTP-setup on the system and the correct mail address in /etc/mdadm/mdadm.conf to get mails whenever the mdadm agent detects an issue with the array. On debian the array is automatically set for scrubbing once a month, and it is probably the same on ubuntu.

 

[N3G4T1v3]

Awesome, thanks, that clears up a lot of my questions

Would you recommend using Debian over Ubuntu?
When setting up the filesystem, would you need to specify the stride and stripe-width? Or are default settings ok?

I like the idea of setting up mdadm to email me when there is an array issue, that's very useful

 

[marsboer]

I prefer Debian because it is more thorougly tested as a whole. It is the little things that are mainly of concern when using it for server purposes like Ubuntu shipping with open-vm-tools that fails to compile the kernel modules on their shipping default kernel 3.5, not having a real standardized utility for mirroring etc. Nothing important for most people at home.
I would not go so far as to recommend Debian over Ubuntu as a general advice as I feel it is just personal preference. Use what you are comfortable with.

You do not have to specify the stride and stripe-width if you are creating the filesystem from scratch as those are automatically calculated for ext4. However, if you grow your array with additional disks later you do manually have to recalclulate the stripe witdth (it is normally not necessary to recalculate the stride as it only depends on chunk size and block size and not the number of drives).

To do this do the following:
1. Get the current stride and stripe width with

tune2fs -l /dev/<arrayfs>

2.
Verify the stride:

<array chunk size> / <fs block size>

# Example: 512k chunk size and 4k block size
512k / 4k = 128

- get the chunk size from mdadm --detail /dev/<array>
- get the block size from the tune2fs output

3. Verify / recalculate correct stripe width if resizing array with more/less drives

RAID stride * (<number of drives in array> - <number of parity disks>)

# Example: 11 disks in a RAID6 array with RAID stride of 128:
128 * (11 - 2) = 1152

3. Set new stripe width if necessary:

tune2fs -E stripe-width=1152 /dev/<arrayfs>

 

[bleomycin]

marsboer has some excellent advice. MDADM isn't very complicated at all and is perfect for what you're trying to achieve. If you are looking for something debian based to graphically manage your NAS have a look at openmediavault as well.

 

[N3G4T1v3]

Thanks marsboer, this will definitely help me in the future
Very excited about a mdadm array in my NAS

marsboer has some excellent advice. MDADM isn't very complicated at all and is perfect for what you're trying to achieve. If you are looking for something debian based to graphically manage your NAS have a look at openmediavault as well.

Thanks for the advice, OMV's webui is very interesting, been playing with it in a VM to see if I like it, and to see how the array setup works
It seemed simple enough, but I'm finding some odd issues, like getting samba shares with a user to work, that are now making me consider just sticking with a regular server OS, as I have at least figured out the config on my other servers, but I'll approach the OMV forums first, I may have missed something