Optimizing mdraid

[Red Squirrel]

I am building a large storage server (24 bay + FC card with around 50ish drives, though those will rarely be turned on). This server will be 100% dedicated to storage, and possibly ldap/kerberos as I want to experiment with that for better file security, but for now it's only going to be strictly storage. So NFS, Samba, possibly iSCSI. I will be using mdadm for the raid arrays.

Anyone know some good tips for optimization? I know I can set the rebuild times to be faster. How much faster is too fast? This is a fairly beefy machine. Xeon E3 1230 V2 Quad Core Processor 3.3GHZ, 8GB of ECC ram. So may as well put those resources to good use.

Also, what about chunk sizes when setting up an array, what is the general rule of thumb here? If mostly bigger files (Ex: VMs) then bigger chunk size, if more a mix of files, not as big of a chunk size? Is there a magic value that is all around good? Most tutorials seem to say like 32 or 64. What if I jack that up to like 512, 1024? higher?

Any other goodies I can optimize? Also, I will most likely read up on NIC teaming and team both gigabit nics. I have a managed switch so it should be able to support it. Any optimizations I can do on the network side? This will be a file server and not a SAN, so it's going to be on the same network/vlan as everything else.

 

[JoeComp]

There should be no reason for you to change the default mdadm chunk size from 512KiB. The only reason to change it is for people who are doing a lot of small writes, then they could make the chunk size quite small, but this obviously does not apply to your usage.

For mdadm performance optimization, usually there are only two parameters you ever need to tune:

1) /sys/block/md#/queue/read_ahead_kb : theoretically, it seems like this should be equal to one or two stripes worth of data to get the best performance, but in some testing I did a while ago I needed at least 24576 for a six drive RAID-6 to max out the sequential read speed. I have seen some people recommend as high as 65536, and some old 3ware support posts mention 16384. The default is 1536. I suggest starting at the default (1536) and testing performance. Then try 32768 to see if performance increases. If so, double that and test again until performance tops out.

2) /sys/block/md#/md/stripe_cache_size : the size of the write cache mdadm will use is equal to 4KiB x N x stripe_cache_size , where N is the number of drives in the RAID. Again, theoretically it seems like two stripes should be sufficient to reach top speed (i.e., stripe_cache_size = 2 x chunk_size / 4KiB), but I found that I needed to set stripe_cache_size to 2048 for top speed, which was 16 stripes with 512KiB chunk size. You could start with 256, test performance, then try 2048 and if the performance increases, double that again until performance stops increasing. Note that you can take up a lot of your RAM with large stripe_cache_size values if you are not careful.