Z87 RAID 5 slow as sin

[Yushatak]

I was downloading a new game on Steam and started getting lagspikes (in fact, it was hard to even get the game to download). I was confused. I looked in task manager and found disk use at 100%. However, I was getting under 10MB/s write. Confused, I tried a benchmark and got ~200MB/s write through the task manager's opinion...

Using the Intel on-board RAID, DZ87KLT-75K board. RAID 5 with 4x 750GB Seagate Momentus XT 750GB Hybrid disks (for 2TB storage total with parity). It's fully initialized, BTW.

How come it suddenly "maxes out" when under real world use and doesn't with the benchmark? I can understand if I was really overloading it, but it should easily be able to do 100-150MB/s..

I don't want to reinstall, but I will if I must and set up RAID 0 (data loss be damned) to not deal with this crap.

 

[Yushatak]

It gets good theoretical speed for sequential, but everything else is in the shitter - especially the small size 4K stuff.

I presume that's where my performance deficit resides, but why.. What can I do? Would changing the stripe size (64K, left it at default) help?

(and yes, using Win8.. the idea was for the power benefits of Haswell but it has issues coming out of sleep so its for nothing now.. )

Oh I should also mention that the BIOS says they are "3.0gb/s", when I know they are SATA-600 drives on a SATA-600 controller.. not sure if that's related to using them in RAID, or what.. Either way, while that's not "good" and might be *a* problem, it's not the cause of *this* problem..

 

[drescherjm]

but everything else is in the shitter - especially the small size 4K stuff.

To me this looks about right. You can not control what gets into the cache of the Momentus XT so if your benchmark does not get cached in the NAND you will have laptop hard drive like performance raided. And yes I do have 4 Momentus XT drives so I do have experience with these. Although in my case its with the original 7200RPM model.

 

[dvsman]

IIRC the XT 750gb are also 7200 drives. it's the models after that that are stepped down to 5400 - but with supposedly improved buffer / logic.

I just run raid 0 and offload data to NAS or my external storage (USB 3.0 is pretty quick any ways). But then again I run my 2 750gb XT drives as storage and use RAID0 SSDs for boot / games.

 

[drescherjm]

Here is a benchmark of a single Momentus XT on a laptop (I will look at a desktop later today - provided I have time). I killed the AV and waited till the disk activity was constantly below QD1.




One theoretical point to consider (since we do not know the algorithm the Momentus XT uses to choose what goes in the cache) about raiding these is that for any piece of data it is essential that all 4 drives ( 3 out of 4 will work in raid 5 at least some of the time) cache the data otherwise the raid will work at the speed of the slowest device. You are definitely using the cache on your 4k 32 queue depth reads although it appears not really that much on the other benchmarks. I am not totally sure why the 512K number is essentially single disk speed. Perhaps we are hitting the point I bring up here.

 

[drescherjm]

My desktop CDM for a 500GB MomentusXT is a little better (I expect due to a faster desktop i7 versus core2 laptop and the drive is not the OS drive on the desktop):



I was thinking part of your problem is probably the motherboard raid5. Although things have improved over the years (it used to be horrendous) MB raid5 still is much slower than true HW raid (with a 512MB stripe cache and BBU) or even linux or zfs software raid especially when it comes to write performance.

 

[Yushatak]

I get WAY better results if I run the tests 9 times - I imagine the SSD caches kick in when you do that..

4K random stuff is still a weak point, though.

How much improvement should I expect if I go to RAID 0?

 

[Yushatak]

Steam downloads go up to or around the full 19.2MB/s (my net speed) then die down slowly, sit there, and then start up again. Eventually it evens out and does around 10MB/s, but with about 15-20MB/s total disk use per task manager it says its at 99-100% use.. which makes no sense, since it can also be at 200MB/s and say that.. Confused.. Once it evens out it will die down again, rinse and repeat after X amount of time. I wouldn't care, but it actually has the symptoms of full disk usage.

 

[Lost-Benji]

My thoughts:

Turn off the RAID and set BIOS to AHCI (you should be in AHCI for the cache and NCQ to work properly)
Set the drives using Storage Spaces to Double mirror or Parity. Test again.

Parity will be slow on writes as most know but can be sped up if you use the power protected commands.

 

[staticlag]

have you gotten smart info from the disks?

It sounds like one disk is lagging the whole array.

 

[tvm777]

get to rma maybe

 

[Tup3x]

If you want speed, you can forget RAID5 on software/firmware RAID. Only option in that case is RAID0.

 

[Lost-Benji]

RAID-0 good for speed but NO redundancy, RAID-10 is what is needed.

 

[Yushatak]

My thoughts:

Turn off the RAID and set BIOS to AHCI (you should be in AHCI for the cache and NCQ to work properly)
Set the drives using Storage Spaces to Double mirror or Parity. Test again.

Parity will be slow on writes as most know but can be sped up if you use the power protected commands.

RAID setting implies AHCI, so the cache should be going. Googled to find out what NCQ is.. That plus the caching might actually make the software RAID a good idea - the hardware can then do its hardware things, and the reads/writes would be distributed anyway.. But then why is hardware RAID so popular? Hadn't even heard of Storage Spaces (Windows 8.. woo...), so basically software RAID but more Linux-style than previous Windows and might get good performance?

have you gotten smart info from the disks?

It sounds like one disk is lagging the whole array.

Had that thought, but nothing detected on any of them.

 

[drescherjm]

But then why is hardware RAID so popular?

One reason is windows software raid and windows bios raid have always underperformed often with raid5 or raid6 having worse than single drive write performance. The second reason used to be CPU usage however for over a decade a single core of a CPU is really all that has been needed in good software raid to perform ratd6 calculations needed for reading or writing (rebuilds need more) so the CPU argument in my opinion is weak at best. Although I admit there are cases that this still applies because of the poor raid implementions on these systems. The basic problem is the stripe cache or lack of a stripe cache forcing every write of less than a full stripe to read then calculate parity then write back immediately. True HW raid and linux software raid buffer this such that the writes do not happen immediately. On the case of linux software raid this comes at a reliability cost meaning if the power goes out in the small time window that linux waits you can and will have data loss. HW raid uses a battery backup unit to protect against that.

 

[Yushatak]

The Intel RST software lets you set up write caching varieties besides the Windows in-built one, I have mine on write-back, which I assume would also pay attention to striping and parity when choosing how much and what and when to write.. Guess not!

I'm going to wipe it and try out the SW RAID. With my box (Haswell 4Ghz 4770K, 2400Mhz RAM, etc.) I think that any performance "hit" would be minimal. I'll also go for RAID 0 or equivalent if it's present in SW RAID, to maximize the benefit while changing everything..

 

[drescherjm]

Try making the stripe size or chunk size smaller. Also if possible increase the size of the write back cache. And when you remove the drives from the array look at the SMART raw data on each drive to see if there are any problems like reallocated sectors or worse problems like pending sectors.

 

[Yushatak]

I made it 4KB and RAID 0 hardware (I tried the Storage Space spanned and while better than 4 disk mobo RAID 5 for 4K r/w it wouldn't let me install the OS to it so it defeated itself..) and am getting amazing speeds now.

We'll see if it holds up in real world, but it feels better so far for sure just setting up drivers and crap.. :3

It's interesting that 512K write is so much higher than 512K read.. lol

 

[Yushatak]

And now that the drivers are in it runs like shit. Tried messing with cache, and while the speeds vary it has absolute crap for 4K read performance. I'm going to wager a guess that the SSD cache held the benchmark data the first time by whatever luck..

I uninstalled the driver and the 4K read performance still sucks, so that was coincidence. The write performance is so much better, but about half the read.

One of my larger concerns is that the array won't get detected as 2.7TB, cutting off at 2TB.. The Intel RST tool saw the size properly.

I suppose I can kill THREE birds with one stone by doing RAID 1+0 - the volume wont be larger than 2TB, I'll get data redundancy, and the read speed should double.

Gonna give that a go..

Edit:

Random 4K is worse than ever and seq. read is down at 91!

Says it's initializing, though.. I better wait until it's happy before writing it off.

 

[staticlag]

I'd say it still sounds like one disk is lagging the whole array.

I would benchmark the disks individually and see if one has a higher latency than the others. If one drive takes even a small amount longer than the others it can really degrade performance of the whole array.

 

[Lost-Benji]

RAID setting implies AHCI, so the cache should be going.

You are making assumptions. Intel took years to even get TRIM to pass through on the RST platform, AHCI is NOT always passed through.

Googled to find out what NCQ is..

Do you know what AHCI actually is or did you google it as well?

But then why is hardware RAID so popular?

It is ONLY popular when used with drives designed for it, then it allows hardware acceleration of the caching and calculations of data.

Hadn't even heard of Storage Spaces (Windows 8.. woo...), so basically software RAID but more Linux-style than previous Windows and might get good performance?

Dude, you need to get out more....

I made it 4KB and RAID 0 hardware (I tried the Storage Space spanned and while better than 4 disk mobo RAID 5 for 4K r/w it wouldn't let me install the OS to it so it defeated itself..)

You shouldn't be installing the OS on that array anyway, not as RAID-5. Too much performance hit, stay with RAID-10 or set the drives as a pair of mirrored arrays (can be done after OS install to a single disk or via RST in the BIOS prior to install.

And now that the drivers are in it runs like shit. Tried messing with cache, and while the speeds vary it has absolute crap for 4K read performance. I'm going to wager a guess that the SSD cache held the benchmark data the first time by whatever luck..

I uninstalled the driver and the 4K read performance still sucks, so that was coincidence. The write performance is so much better, but about half the read.

You have just installed a fresh OS and you haven't given it a chance to settle (needs a few hours really) or initialized the volume... You are your own worst enemy dude.

One of my larger concerns is that the array won't get detected as 2.7TB, cutting off at 2TB.. The Intel RST tool saw the size properly.

Google that one too please. No OS install to volume larger than 2TB, OS still uses MBR.

I suppose I can kill THREE birds with one stone by doing RAID 1+0 - the volume wont be larger than 2TB, I'll get data redundancy, and the read speed should double.

RAID-10 it, clear the disks, install Win7 or Win8, then let the system run for an hour or so until you see the background tasks settle. Once done, install latest STABLE release of the drivers/RST. Now go into "SERVICES" and set Intel RST to "AUTOMATIC" rather than Delayed.
Reboot system.
Now when system has settled again, make sure you have ALL device drivers for the system installed, if you ask how to do this then please stop, walk away and get someone who knows what they are doing.
Now, assuming everything is loaded and happy, you need to initialize the volume as well as enable caching (not enabled by default as you are NOT native AHCI). Caching should be done AFTER the initialization.

Once done, test what you like but also keep this little bit of wisdom, synthetic benchmarks are just that, SYNTHETIC. Try actually driving the system and using it real-world rather than getting caught up on tests that rarely ring truth and usually biased.

 

[Yushatak]

Anybody know a tool (bootable perhaps) that can benchmark disks in an array using free space without destroying the array/partition/etc.? I think I'll be happy with the 1+0, and I don't want to wipe it without knowing if it will be worth the reinstall. Dediicated most of my day yesterday to reinstalling Windows repeatedly, lol.

I'm getting way more consistent real world performance, so as long as the read speeds go up post-initialization I'm satisfied.

Bowever, I'm still quite concerned/upset at the SATA2 speeds with my SATA3 ts and disks.. I realize that disks almost never xome close to the theoretical maximums but it bothers mee.. :/

I'll try disconnecting some disks and see if the speeds jump up, but Intel pretty clearly said that all the ports did 6gbit..

Sorry for typos, on my iPhone and my keyboard tweaks arent friends with this site..

 

[drescherjm]

Bowever, I'm still quite concerned/upset at the SATA2 speeds with my SATA3 ts and disks.. I realize that disks almost never xome close to the theoretical maximums but it bothers mee.. :/

Hard drives (except for the 1TB / platter drives) have a hard time maxing out SATA I let alone SATA II since the media transfer rate is the limitation. The only chance to max out SATA III is the cache but that will not happen at all in a Momentus XT since it uses a single NAND chip which in no way could max out SATA II. Remember real SSDs get their high sequential performance by raiding 8+ NAND chips. For a non hybrid hard drive could theoretically max out SATA III but since the cache is 64MB or less there is little chance of that.

 

[DJ_Datte]

Hi,

Looking at your stats, I am not sure what kind of performance you are expecting. You are not using SSDs.

A normal drive can produce around ~100 4k iops, you are getting about 1500kb/sec /4 = ~375 iops on a RAID5, that's pretty damn decent performance.

Also, don't forget due to raid5s write-read-write, you are giving up 50% of your iops in the parity operation that R5 does at all times when doing writes.

I think you should be pretty happy with that performance, there is nothing to complain about in the numbers presented across 4 drives.

Edit: Also, raid5 scrambles files, it's absolutely uncertain how much help the hybrid part of the drives can even help at that stage. It's a very unpredictable operation from the drives point of view.

 

[Yushatak]

I think I expressed that poorly - it's not the speed I'm upset with not being SATA3, it's my BIOS telling me that my drives are SATA2..

 

[DJ_Datte]

Your best bet for iops & perf is raid10. You do give up some space, but it's cheap, and redundant and reliable.

 

[Yushatak]

Calm down, bro! With no offense to you intended (you do know your shit) - I'm not an imbecile. I don't explain every step of what I've done/do or my understanding thereof, so don't assume I don't know anything. xD

You are making assumptions. Intel took years to even get TRIM to pass through on the RST platform, AHCI is NOT always passed through.

Do you know what AHCI actually is or did you google it as well?

I know what AHCI is, "Advanced Host Control Interface", modern protocol and standard for disk communication up from what IDE and it's elder friends were using for what seems like an eternity. RAID on this board also includes AHCI being enabled - on most Intel setups it is, to my knowledge, but mine explicitly says so in the BIOS. Unless it's lying or being vague and it only has a partial implementation, etc., of course - not impossible.

It is ONLY popular when used with drives designed for it, then it allows hardware acceleration of the caching and calculations of data.

I'm relatively new to the RAID world, didn't have the money fo setups like this in the past and I've absorbed all the new knowledge like a sponge.

Dude, you need to get out more....

I stayed away from Windows 8 for a while. My job focus is primarily prorgamming, and my home boxen are Win7. While I've used Win8 plenty to know how it functions internally and be experienced with it, I hadn't done too much with RAID and didn't even know it had made changes.

You shouldn't be installing the OS on that array anyway, not as RAID-5. Too much performance hit, stay with RAID-10 or set the drives as a pair of mirrored arrays (can be done after OS install to a single disk or via RST in the BIOS prior to install.

This is something I learned through this experience. Didn't know enough about real-world implementation of RAID, only the theoretical.

You have just installed a fresh OS and you haven't given it a chance to settle (needs a few hours really) or initialized the volume... You are your own worst enemy dude.

I had thought it had initialized when I initially wrote that post, and then realized it had not when I installed the RST drivers (and edited the post). As I said when I noticed that it was still initializing, I am waiting for it to finish that before judging it.

Google that one too please. No OS install to volume larger than 2TB, OS still uses MBR.

XP and above (granted XP needs patching/etc. so I don't expect that behavior from *it*) support GPT, so I figured that it would intelligently choose to use it. Didn't know that the OS disk wouldn't. My largest single drive before this was precisely 2TB, so I had not tried installing an OS on anything larger before that. Also, I did Google around and did not find that tidbit of information while I did. I had suspected that to be the case, though, so it makes perfect sense.

RAID-10 it, clear the disks, install Win7 or Win8, then let the system run for an hour or so until you see the background tasks settle. Once done, install latest STABLE release of the drivers/RST. Now go into "SERVICES" and set Intel RST to "AUTOMATIC" rather than Delayed.
Reboot system.
Now when system has settled again, make sure you have ALL device drivers for the system installed, if you ask how to do this then please stop, walk away and get someone who knows what they are doing.
Now, assuming everything is loaded and happy, you need to initialize the volume as well as enable caching (not enabled by default as you are NOT native AHCI). Caching should be done AFTER the initialization.

Don't insult me by implying that I might not know how to install device drivers. If you read even half of my text on this thread, you'd see that I plainly know far more than that level of stuff, and I even talk about installing drivers a number of times. I was doing very quick tests by just changing the array, reinstalling Windows, and running benchmarks with nothing but the included drivers for Win8. Things are relative, so I figured that was OK, and it did show me differences and let me investigate performance. I came to the conclusion that 1+0 was the best option, and that's why I'm still on it now. Caching *IS* enabled by default, and it enables itself prior to the (automatic, it does this itself once RST is installed) initialization, unless you mean a different caching. I also had checked the services and found RST on delayed - I started it manually when necessary, and after a reboot or two it began starting normally. I'll swap it to non-delayed, though, per your advice.

Once done, test what you like but also keep this little bit of wisdom, synthetic benchmarks are just that, SYNTHETIC. Try actually driving the system and using it real-world rather than getting caught up on tests that rarely ring truth and usually biased.

I know - I'm not being "caught up". I know my data usage patterns, and most games that I play load hundreds of tiny files. I was experiencing slower-than-normal load times on several games, which is one of the things that prompted this investigation in the first place. I figured 4K read benchmarks were an appropriate approximation, the best I'd get in a quick scenario without installing a game and playing it for a while (and trying to be consistent) each time for comparison. Once I found a suitable candidate (like this setup) I'd check real-world.

Again, I mean no offense by countering some of your points, but please don't insult me - I hold several certifications, years of experience, and I program and do IT for a living. I'm not some clueless buffoon, and you cannot safely make assumptions based on what I didn't explicitly say I've done in my posts - I was assuming that anyone reading would take it for granted that I performed any logical steps I didn't mention.

 

[ND40oz]

You are making assumptions. Intel took years to even get TRIM to pass through on the RST platform, AHCI is NOT always passed through.

AHCI is not always passed through? What are you talking about, TRIM is a command, AHCI is specification.

If you have RAID enabled on an Intel chipset, AHCI is also enabled, it's been that way since ICH6.

http://www.intel.com/support/chipsets/imsm/sb/CS-015988.htm

If you are using a SATA hard drive, set your BIOS to RAID mode. RAID mode provides the greatest overall flexibility and upgradeability because it allows your system to be RAID ready and enable AHCI.

 

[Yushatak]

Benchmark with fully initialized 1+0:

Not as amazing as that freak benchmark on RAID 0, but that didn't hold up, lol.

This is really stable, the performance doesn't fluctuate or dip, etc.. Everything is above 1MB/s (I'm looking at YOU 4K R/W...) so that's good.

I'm pretty content with this, but I still want to know why my drives are detected as SATA-300 instead of SATA-600. :|

 

[jasonelmore]

i dont know if i know what i'm talking about, but check your sata cables. i heard some sata cables are not thick enough for sata 6 speeds. just a thought. You could also try diff ports but with 4 drives i doubt you have many left on the native chipset

 

[ep0x73]

Raid 10 should be just as fast as 0 with redundancy. I think those Seagate drives are holding you back.
Get 4 RE drives or ES drives and you should see much improvement.
Desktop drives are fine for 0 or 1 but for 5 or 10 you must use enterprise drives that are designed for raid.