raid 5 vs raid 6 performance with Linux MD raid

R

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,223
I have 2 failing drives out of 5 in my raid 5 array and realized, I need raid 6. Just wondering, will I notice any major performance downfall by going to raid 6? Will the fact that there's an extra spindle make up for it? Ex: if I go from 5 drives in raid 5 to 6 in raid 6.

Conviniently I ordered 2 drives to add to my backup pool but as an emergency they will go towards replacing the failing ones. Part that sucks is I bought those not too long ago, glad they're cheap.
 
At work I get 300+ MB/s reads and 200+ MB / s writes on linux software raid6 on 4+ year old dual core athlons X2 with 6 to 8 drives that are more than 3 years old and a less than 7% cpu load on a single core when reading or writing. Make sure you extend the default stripe cache from 256 to at least 1024. I usually set this to 2048.

/sys/block/mdX/md/stripe_cache_size

where X is the raid array minor (1, 2 ...)


John
 
Good to know. My cache size is 256 though, is there a way to modify that? I will probably have to rebuild the array right?
 
Interesting, so I can just add that in my startup. I'll probably get extra performance I never knew I could get, sounds like it makes a big difference.

I'll wait till I replace those drives though. Chances are it wont do anything to speed up their failure but better safe then sorry.
 
This will have a dramatic effect on rebuild times as well. Last week I needed to rebuild an 8 x 500GB drive raid 5. The rebuild was going to take around 2 hours @ around 57MB/s. After raising the cache to 16384 the time went down to 70 minutes and the rate was 110 MB/s. I have never seen over 100 before on a rebuild. This is on a core2quad that I am going to move to raid6 and resize to 2TB by swapping in 2TB hitachis for the 500GB seagates..

BTW, there is a trade off here. If you lost power during a large write the and the entire stripe cache was not flushed (this happens pretty quickly) you can get corruption. This is why the default is small. At home and at work I have all of my linux software raids on UPSs so that minimizes the chance of that.
 
hmm what kind of corruption, just that particular file that was being written, or like serious file system corruption? I have a UPS (it's not working now mind you - need new batteries) but still don't want to risk total disk corruption, but if it's just localized to one or few files then I can live with that risk. Restore them from backup, and call it a day, if ever it happens.
 
drescherjm said:
This will have a dramatic effect on rebuild times as well. Last week I needed to rebuild an 8 x 500GB drive raid 5. The rebuild was going to take around 2 hours @ around 57MB/s. After raising the cache to 16384 the time went down to 70 minutes and the rate was 110 MB/s.
Click to expand...

Note that the amount of memory used is a multiple of the stripe_cache_size. I believe the RAM = stripe_cache_size x 4KiB x num_disks_in_RAID. So you were probably using 16384 x 4KiB x 8 = 512 MiB of RAM.

When I did some tests with a RAID 6 of 2TB Samsung F3 HDDs a while back, I found that I did not see much improvement over 2048. Since 2048 x 4KiB = 8MiB, and I had a 512KiB chunk, that means the cache was holding 16 full stripes in RAM. Another way of looking at it is that 8MiB is, very roughly, 8 tracks on the HDDs.
 
For me it nearly doubled the performance in the rebuild but I am using 64K chunks. On one old server (2004 era 64 bit dual processor opteron 2.0GHz) I was using 512K chunks but I found that this reduced performance on a few filesets that I tested years back so now I just go with the default for all raid arrays (I have 12 to 15 arrays at the moment).
 
Last edited:
Red Squirrel said:
hmm what kind of corruption, just that particular file that was being written, or like serious file system corruption? I have a UPS (it's not working now mind you - need new batteries) but still don't want to risk total disk corruption, but if it's just localized to one or few files then I can live with that risk. Restore them from backup, and call it a day, if ever it happens.
Click to expand...

I have never seen it corrupt anything (and I have had many arrays) but it is a possibility. The stripe cache will contain dirty data (unflushed writes) for a few seconds after a write. If you had an instant off this data would be lost even filesystem structures. The default 256 operates the same way. This is in no way as bad as running a HW raid card without the BBU because HW raid cards hold the dirty cache even longer than md raid.

BTW, this is called the raid "write hole"
 
Last edited:
drescherjm said:
For me it nearly doubled the performance in the rebuild but I am using 64K chunks. On one old server (2004 era 64 bit dual processor opteron 2.0GHz) I was using 512K chunks but I found that this reduced performance on a few filesets that I tested years back so now I just go with the default for all raid arrays (I have 12 to 15 arrays at the moment).
Click to expand...

What doubled performance? You mean going from 256 to 16384 stripe_cache_size? My point was that you may not have needed to go so high -- maybe something in between would have been just as fast.

Also, 512 KiB chunk size IS the default for mdadm now. It has been for more than a year.
 
What doubled performance?
Click to expand...

The rebuild time was cut nearly in half when I set the stripe cache size to 16384 versus the 256 default.

You mean going from 256 to 16384 stripe_cache_size? My point was that you may not have needed to go so high -- maybe something in between would have been just as fast.
Click to expand...

I tried 1024, 2048, 4096, 8192, 16384 and saw large gains on each in the rebuilding rate. The first few did have the most impact. During the test I forced the 100000 in as the sync_speed_min so it would go as fast as it could.

Also, 512 KiB chunk size IS the default for mdadm now. It has been for more than a year.
Click to expand...

Did not know that. Thanks. I do not generally build many new arrays since I have 12 to 15 of them already at work. I am now mostly swapping in larger drives to existing arrays and growing them..
 
Last edited:
Is there any disadvantage of going too high? Just wondering what the happy medium is. Or should I increase a bit at a time and benchmark and see what setting is best?
 
The disadvantages are increased memory usage and a slightly higher chance of data loss in an instant power off or kernel panic.
 
Or should I increase a bit at a time and benchmark and see what setting is best?
Click to expand...

Do things that make disk activity and look at the stripe_cache_active. If you poll it enough it will give you a better understanding of what is going on.
 
You will have lower performance with RAID 6 due to the double parity being used, especially if encryption is used.

The whole point of RAID 6 is the double parity, in other words, it will allow up to 2 drives to fail without losing the array.

RAID 5 is a bit faster, but will only allow one disk to fail.

Honestly, RAID should only be used if you can sacrifice the disk space for the double parity and if the data is that mission critical. Otherwise, RAID 5 is more than good enough.

What do you need such high data transfer rates for? If you are just streaming HD videos or copying files, 60MB/s+ is far more than enough.

If you are doing video editing though, 100MB/s+ is probably more up your alley. ;)
 
When I put the rest of my 9 hitachis in the currently 8 drive raid 5 array that I am was talking about rebuilding in this thread and make it linux software raid6, I am expecting to get 300 to 500 MB/s writes and probably 600 to 700 MB/s reads and that at 7% CPU usage on one core of the core2quad server. Although the server only has 2 gigabit ports so its not like we will use all of that bandwidth. Still waiting for cheap ethernet 10G.
 
Last edited:
You must log in or register to reply here.
Back
Top