• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Best RAID level for random reads?

H

hutchingsp

Limp Gawd
Joined
Dec 24, 2006
Messages
150
As subject, be grateful for peoples thoughts on the best RAID level for random/semi-sequential read access.

There would be a minimum of 8 spindles.
 
Raid 10. Although raid6 will be better for large sequential reads with the same amount of disks.
 
Are you sure about the large seq reads? I would have thought a smart raid10 implementation would be streaming data from all N spindles, no?
 
danswartz said:
Are you sure about the large seq reads? I would have thought a smart raid10 implementation would be streaming data from all N spindles, no?
Click to expand...

He is right.

Indeed with a fast/newish controller raid6 will be best for random reads and sequential reads. Many controllers I have seen (3ware, LSI, etc...) will not read from all disks (only half) when doing sequential reads but they usually will for raid6.

Also raid6 will be just as fast as raid10 for random reads and you get to use more disk space (and a shortstroked raid6 of the same size raid10 would have been would give even greater performance then raid10).

The only time raid10 performs significantly better than raid6 is when the array is degraded or when doing small random writes. Oh or its just using an old/and or crappy performance controller that cant do raid6 worth a darn. Random reads they both should be pretty par though.
 
It really isn't fair to blame raid10 performance for brain-dead controllers. If this is the norm, yet another reason to do software raid (or zfs).
 
danswartz said:
It really isn't fair to blame raid10 performance for brain-dead controllers. If this is the norm, yet another reason to do software raid (or zfs).
Click to expand...

I also saw this behavior on mdadm as well. The only controller I have seen read from all disks during sequential reads is Areca. The norm appears to be sequential reads only read from half the disks. I haven't ever done raid10 on zfs so I dont know how that behaves.

Also this is the case from various forms of software raid1 I have seen as well (fakeraid) like nvidia, intel raid. Areca is also the only form of raid1 I have seen that gave double sequential read speed as well.
 
Also another reason to go raid6 is it provides far greater chance of array failure and/or lost data during a rebuild. Some controllers wont rebuild when the other disk has errors in a raid10 array (like 3ware).

If you don't need fast random writes I say go raid6.
 
This would be a "proper" Dell PERC H700 (or equivalent) controller with a ton of cache (which won't help much for reads I grant you).
 
hutchingsp said:
This would be a "proper" Dell PERC H700 (or equivalent) controller with a ton of cache (which won't help much for reads I grant you).
Click to expand...

Then its an LSI controller and raid6 will likely be better sequential but I don't have a lot of experience on how older LSI and perc controllers perform in raid6. I would say try creating the array both ways and see which is faster.
 
houkouonchi said:
I also saw this behavior on mdadm as well. The only controller I have seen read from all disks during sequential reads is Areca. The norm appears to be sequential reads only read from half the disks. I haven't ever done raid10 on zfs so I dont know how that behaves.

Also this is the case from various forms of software raid1 I have seen as well (fakeraid) like nvidia, intel raid. Areca is also the only form of raid1 I have seen that gave double sequential read speed as well.
Click to expand...

I would also quibble with the claim that raid6 provides 'far better protection'. If you have, say, an 8-disk raid10, with a single drive failure, the odds of a second failed drive taking you down is what, 1/7? I guess everyone has to make their assessment based on job mix and redundancy requirements. I've heard of folks who really need the random performance and don't care about wasted space using 3-way mirror raid10s using zfs :)
 
Last edited:
Just looking at a Dell PDF. It centres on Equallogic but I would expect the same to hold true for any enterprise RAID controller.

http://i.dell.com/sites/content/business/smb/sb360/en/Documents/wp-eql-how-to-raid.pdf

It suggests there's not likely to be that much in it either way.

My initial reaction when posting the thread was to use RAID50 as I still need half decent random write performance, though read performance is likely to be my biggest bottleneck for this application.
 
RAID50
Click to expand...

That would probably be fine for 8 to 10 drives over that I would worry about reliability and the chance of UREs. Although I am not sure raid50 would be any better than raid6 with a modern raid system and 8 to 14 drives.
 
Last edited:
danswartz said:
I would also quibble with the claim that raid6 provides 'far better protection'. If you have, say, an 8-disk raid10, with a single drive failure, the odds of a second failed drive taking you down is what, 1/7? I guess everyone has to make their assessment based on job mix and redundancy requirements. I've heard of folks who really need the random performance and don't care about wasted space using 3-way mirror raid10s using zfs :)
Click to expand...

No its not 1/7th. Also its just one drive getting stressed instead of the entire array which means more likely to fail. I have seen well over 20 arrays (in a group over over 1000 servers) that have had failed arrays due to being raid10 and one drive failing and during the rebuild the other drive in the group also failing. I have seen maybe 2 times where a raid10 array survived where raid6 wouldn't have (3 disks failed).

Also with raid10 you are really susceptible to data loss due to bad sectors and what not as you only have one source for the data where as with raid6 you still have redundancy after a single disk failure and it can recover from read errors and other errors on disks during the rebuild which raid10 does not deal with well on most controllers.
 
Well, I don't want to argue about HW controllers - my knowledge there is not great. One wonders how healthy those secondary failed drives were to fail under a rebuild. If the array was being scrubbed regularly, hard to imagine that would have happened. Everything you say makes me more convinced to go the software route... BTW, it sucks when you are the guy responsible for those 20 failed arrays, but I'm not sure the numbers are meaningful. 20 arrays out of how many failed ones is more useful, no?
 
Last edited:
danswartz said:
Well, I don't want to argue about HW controllers - my knowledge there is not great. One wonders how healthy those secondary failed drives were to fail under a rebuild. If the array was being scrubbed regularly, hard to imagine that would have happened. Everything you say makes me more convinced to go the software route... BTW, it sucks when you are the guy responsible for those 20 failed arrays, but I'm not sure the numbers are meaningful. 20 arrays out of how many failed ones is more useful, no?
Click to expand...

Everything is backed up so some data is lost but most is restored on to a new array. Most of the failures happen because during the rebuild the other drive in the pair starts running into bad sectors. If its bad enough it can cause the disk to keep getting kicked and fail the array or make the machines unstable enough it can't be used in production.

There isn't usually a scrub/check mode on most raid controllers for non parity raids anyway. Also on heavily loaded machines will this will effect performance quite a bit they likely wont be regularly checked/scrubbed in the enterprise either. 3ware seems really bad as it rarely will complete a rebuild even on just a few bad sectors. Even if you tell it to ignoreecc (the bad sectors/read errors) it sometimes will complete but still 80% of the time it just eventually crap out with rejecting I/O to offline device.

I think in many of these cases if the machine was just left alone it might rebuild but when the machine is constantly doing heavy I/O with the rebuild process the other drive craps out too much during the rebuild. We don't have a ton of raid6 machines but so far not one of those have lost its array due to drive failures. Also a lot of cases its because the raid controller waits too long to actually fail the drive and the problem is both drives are in bad health alot of times.

The average user doesn't usually keep track of re-allocated sectors and disk health either so they are pretty much just as susceptible to this. Either way when a drive fails on raid10 and you get unreadable sectors (even if it will rebuild) you will have data loss/corruption. There is no denying that fact. In the same case on raid6 you would likely be safe unless the extremely unlikely case of the same area of the drive being damaged on multiple disks.

There are many things I still suggest raid10 for (like MySQL). Raid6 does have its defects but if the machine is mostly doing random reads raid6 should perform just as well and its a lot more space efficient and IMHO better protection of your data/file-system.
 
Yeah, I see your point. On the other hand, it seems odd to say you need raid6 because the data is mission critical, but you never scrub the array for performance reasons, so half the drives in the array could be going bad for all you know. This is why the zfs best practices guide says sata drive pools should be scrubbed weekly, sas drives monthly.
 
hutchingsp said:
As subject, be grateful for peoples thoughts on the best RAID level for random/semi-sequential read access.

There would be a minimum of 8 spindles.
Click to expand...

In theory, RAID 1 (same data on all spindles) should provide best performance for pure reads.

houkouonchi said:
The average user doesn't usually keep track of re-allocated sectors and disk health either so they are pretty much just as susceptible to this.
Click to expand...
Reallocation's only purpose is to make HDDs look better, isn't it?
Given that SMART monitoring isn't integrated into OSs (at least in Windows), it's not weird that most users don't pay attention.
 
XTF said:
Reallocation's only purpose is to make HDDs look better, isn't it?
Given that SMART monitoring isn't integrated into OSs (at least in Windows), it's not weird that most users don't pay attention.
Click to expand...

I find that disks cause raid problems, don't want to rebuild/faulting the array, etc.. well before the disk is smart failing usually and yeah most OS's do have it built in and will inform you only when it is smart failing though. I usually try to replace disks as soon as they have over 50-80 re-allocated sectors as its usually on its way to death after that point.

Anyway ran into yet another array that failed due to it being raid10 instead of raid6:

Code: 
2011-11-18 07:13:25  Enc#1 Slot#8     Device Failed
2011-11-18 07:13:25  Raid Set # 000   RaidSet Degraded
2011-11-18 07:13:23  ARC-1222-VOL#001 Stop Rebuilding       052:02:22
2011-11-18 07:13:23  ARC-1222-VOL#001 Volume Failed
2011-11-18 07:13:23  ARC-1222-VOL#000 Volume Degraded
2011-11-18 07:13:22  Enc#1 Slot#8     Reading Error
2011-11-18 07:13:15  Enc#1 Slot#8     Reading Error
2011-11-18 07:13:06  Enc#1 Slot#8     Reading Error
2011-11-18 07:12:59  Enc#1 Slot#8     Reading Error
2011-11-18 07:12:53  Enc#1 Slot#8     Reading Error
2011-11-16 03:11:00  ARC-1222-VOL#001 Start Rebuilding
2011-11-16 03:11:00  ARC-1222-VOL#000 Complete Rebuild      002:11:40
2011-11-16 00:59:20  ARC-1222-VOL#000 Start Rebuilding
2011-11-16 00:59:15  Raid Set # 000   Rebuild RaidSet
2011-11-16 00:59:15  Enc#1 Slot#7     Device Inserted
2011-11-16 00:56:43  Enc#1 Slot#7     Device Removed
2011-11-14 12:24:09  Enc#1 Slot#7     Device Failed
2011-11-14 12:24:08  Raid Set # 000   RaidSet Degraded
2011-11-14 12:24:06  ARC-1222-VOL#001 Volume Degraded
2011-11-14 12:24:06  ARC-1222-VOL#000 Volume Degraded
2011-11-14 12:23:14  Enc#1 Slot#7     Time Out Error
2011-11-14 11:30:36  Enc#1 Slot#7     Reading Error
2011-11-14 11:11:09  Enc#1 Slot#7     Reading Error
2011-11-13 11:08:36  Enc#1 Slot#7     Reading Error
2011-11-13 10:59:21  Enc#1 Slot#7     Reading Error
2011-11-13 07:13:35  Enc#1 Slot#7     Reading Error
2011-11-10 15:55:53  Enc#1 Slot#7     Reading Error
2011-11-10 11:51:21  Enc#1 Slot#7     Reading Error
2011-11-10 11:24:54  Enc#1 Slot#7     Reading Error
2011-11-09 12:10:54  Enc#1 Slot#7     Reading Error
2011-11-09 12:10:49  Enc#1 Slot#7     Time Out Error
2011-11-08 10:50:45  Enc#1 Slot#7     Reading Error
2011-11-08 10:50:39  Enc#1 Slot#7     Reading Error
2011-11-07 10:48:42  Enc#1 Slot#7     Reading Error
2011-11-07 10:48:35  Enc#1 Slot#7     Reading Error
2011-11-05 10:37:19  Enc#1 Slot#7     Reading Error
2011-11-05 02:25:22  Enc#1 Slot#7     Reading Error
2011-11-04 10:25:31  Enc#1 Slot#7     Reading Error
2011-11-03 09:53:16  Enc#1 Slot#7     Reading Error
2011-11-03 02:43:49  Enc#1 Slot#7     Time Out Error
2011-10-30 22:14:17  H/W MONITOR      Raid Powered On
2011-10-30 22:13:11  H/W MONITOR      Raid Powered On
2011-10-30 21:23:34  Enc#1 Slot#7     Time Out Error
2011-10-29 16:03:02  Enc#1 Slot#7     Reading Error
2011-10-28 10:43:14  Enc#1 Slot#7     Reading Error
2011-10-28 10:10:09  Enc#1 Slot#7     Reading Error
2011-10-28 10:10:01  Enc#1 Slot#7     Reading Error
2011-10-25 07:08:37  H/W MONITOR      Raid Powered On
2011-10-25 06:28:19  H/W MONITOR      Raid Powered On
2011-10-25 06:21:26  H/W MONITOR      Raid Powered On
2011-10-07 23:19:30  H/W MONITOR      Raid Powered On

Yeah it might have been saved had the bad disk been discovered when it started throwing errors before it was kicked from the array, by that time it was really too late. As soon as the other slot in the mirror pair was stressed due to the rebuild it started crapping out to.
 
You must log in or register to reply here.
Back
Top