RAID 5 array writing at 10MB/Sec?

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
This is really starting to bother me. I have a server with six 1TB hard drives setup in two RAID 5 arrays. Both arrays are identical. Three drives, 64kb striped, write-back enabled, advanced performance enabled. The hard drives are about 2 years old and were originally in a different server with the same RAID setup. Since then, they have been in a server that I have been using for less than a year. The motherboard is the ASUS P6T Deluxe V2, and I am using the onboard Intel RAID. I never had an issue until about a month ago when I decided to rebuild the server.

The RAID arrays were never changed or modified. The first array contains C: and D: and the second array is E:. Prior to rebuilding, I backed up all the data to the second array just in case.

I then reinstalled Server 2008 on C: and set it up similar to before. Since all the UserData was stored on C: by default, I went to move the data back and that is when I noticed it was writing extremely slow. I verified the array, and there were no issues. All volumes are defrag'd once a week. I haven't found anything unusual that would cause this problem.

I would imagine the issue would either be one of the hard drives, or the RAID controller. Any ideas, or how I can go about testing? I really don't feel like deleting the RAID array and re-creating it, but at this point I don't know what else to check. All drivers and the setup is the same as before.

See screen shots below.

Disk C:
C.jpg


Disk D:
D.jpg


Disk E:
E.jpg


IRST
IRST.jpg
 
That's normal on RAID5. You need write-back buffering and combine/split engine to make a driver write fast to RAID5. Could you check whether Intel write caching is enabled, and that you are running the most recent Intel RAID drivers?
 
sub.mesa said:
That's normal on RAID5. You need write-back buffering and combine/split engine to make a driver write fast to RAID5. Could you check whether Intel write caching is enabled, and that you are running the most recent Intel RAID drivers?
Click to expand...

It is not normal and never ran like that. You'll notice in the screen shots that the second array writes at 130MB/sec. My first array didn't have these issues before, and like I mentioned in my post, that write-back is enabled as well as any other performance options and the latest drivers are installed.
 
Disable the write caching and all Intel RAID5 write so slow; so it IS normal. You only get high writes on RAID5 with write-back.

By the way; how do you boot your system? From one of the RAID5 arrays? You have 6 HDDs on your onboard controller? You don't happen to have any pics do you? Can you tell me your motherboard make, though?
 
KapsZ28 said:
It is not normal and never ran like that. You'll notice in the screen shots that the second array writes at 130MB/sec. My first array didn't have these issues before
Click to expand...

Quick thoughts (if they're off, then I apologize):

Is the cluster size in the file system significantly different on the two arrays? Are C:/D:'s cluster sizes different from E:'s?

Could there be an alignment issue. (Although that was semi-covered on the first question above.)

I know on other RAID controllers a new array would run like ass 'til you had initialized the array. But you've verified the array, so that should cover it.

Since C: and D: are on the same array, is there anything happening on both at the same time that might cause write performance to plummet?

Also, are the RAID controllers identical on both (the new mobo and the old)? If not, maybe the new controller just doesn't like two logical drives on the array? Goes back to some sort of alignment issue possibility too between clusters of the file system and the 64K RAID stripes.

Just thoughts.
 
sub.mesa said:
Can you tell me your motherboard make, though?
Click to expand...

First post:

KapsZ28 said:
The motherboard is the ASUS P6T Deluxe V2, and I am using the onboard Intel RAID.
Click to expand...


sub.mesa said:
By the way; how do you boot your system? From one of the RAID5 arrays?
Click to expand...

He said:
KapsZ28 said:
The first array contains C: and D: and the second array is E:. Prior to rebuilding, I backed up all the data to the second array just in case.
Click to expand...


You always boot from C:, and it's on his first array along with D:. E:'s on the 2nd array.

please re-read the OP.


But this does bring up a question from the wording of your OP:

Is the first array the same array from the ol' mobo? Or did you break the array, make a new array on the new mobo, and then install 2008?

If you're starting clean anyway, break the array and make a new array.

If you are moving the ol' array between mobos and then installing, it might not like that. It might see some config info in the array data that it doesn't like. I suggest breaking the array, making a new one, and then start fresh.
 
Last edited:
Tolyngee said:
Quick thoughts (if they're off, then I apologize):

Is the cluster size in the file system significantly different on the two arrays? Are C:/D:'s cluster sizes different from E:'s?

Could there be an alignment issue. (Although that was semi-covered on the first question above.)

I know on other RAID controllers a new array would run like ass 'til you had initialized the array. But you've verified the array, so that should cover it.

Since C: and D: are on the same array, is there anything happening on both at the same time that might cause write performance to plummet?

Also, are the RAID controllers identical on both (the new mobo and the old)? If not, maybe the new controller just doesn't like two logical drives on the array? Goes back to some sort of alignment issue possibility too between clusters of the file system and the 64K RAID stripes.

Just thoughts.
Click to expand...

Cluster sizes and everything else is identical between the two arrays. When I built the new server, I started fresh with the arrays and never had performance problems. I was running several VMs on D without problems. Now I wouldn't be able to because of performance. Luckily I turned my old server into an ESXi box, so that doesn't matter.

Prior to reinstalling the OS, I was backing up all the data to my second array. For some unknown reason, which I never bothered to look into, the server crashed. Since I was 100% confident that everything on D was backed up to E, I decided not to break and recreate my RAID array. Probably a bad idea on my part, but I just have too much damn data to back up everything on E.

There is one thing different now which is odd, but is probably due to whatever the issue is on my first array especially since all the applications are running on that array. When I first log into the OS, the Intel Rapid Storage Technology takes 5-10 seconds before it is ready and shows the status as being good. This never happened before.

If it is an alignment issue, and re-creating the array will definitely fix it, would I be able to backup the partitions and just restore them on the new array? Or would I be stuck re-installing everything from scratch?
 
You can check the alignment with AS SSD (google!); it will say GOOD if alignment is good, and BAD if it's not. Good benchmark program too!
 
Tolyngee,

I think I lied and the cluster size of D is different from the other drives. What is the easiest way to verify cluster size and change it?
 
Delete the partition in Disk Management screen, killing off all data on that partition. Then create a new partition, and select the same clustersize as you did for the other volume.

Disk Management can be found via: click start, right click My Computer, click Manage, now click Disk Management.
 
Alright. And can you confirm that when using AS SSD or CrystalDiskMark (i recommend this one; you could do only sequential tests by clicking the button) you can confirm the bad write performance?

It may be that Intel's RAM writeback driver is not working for that array, but it could also be that this due to being your boot disk, would have background writes which significantly lower your test scores.
 
I ran both benchmark programs and they have the same results as the screen shots I posted using ATTO Disk Benchmark. The write performance on C and D is about 10 times slower than on E.
 
You can verify whether or not background writes are occurring just by watching disk resource monitor during a benchmark. Or use filemonNT or something.

But I suspect some of this is due to not breaking and re-creating your array.

If you move between two different RAID controller, this step is not just always recommended anyway, but most of the time is mandatory. Controllers between different manufs are most likely mandatory (they just don't store their config data the same way), but small difference between different controllers by the same manuf could make it mandatory, or at least recommended......

Well, like I said, if I was doing a clean install anyway, I would have just broken and re-created the array to be sure.

You can always clone the array off of there, break/re-create a new array, then clone it back. A bit of work though if it's not the issue.

I suspect this more, as you said it worked fine and fast before when it was the boot partition, so the background writes other places by the OS wouldn't add up. Unless 2008 is just a new beast and new experience in all of this... But resource monitor should verify that...
 
It could be old metadata, yes. That may be why the write-back isn't working. Trying to destroy the RAID array and re-creating it might be worth a try.

Also, it's possible one of the disks have cabling problems causing much lower performance. Can you see SMART information on the physical HDDs inside that Intel utility?
 
Cheers! Much better indeed!

One thing to keep in mind, though, is that write back in RAM is kind of dangerous. In some circumstances like a blue screen (crash) or power failure you could have a corrupt array/filesystem in some cases. So maintain a good backup at all times for your important data!
 
I guess I would say the RAID driver stopped working properly. Although write caching was enabled on the disk drive for volume0 and write-back was enabled within IRST, it didn't appear to really be running properly. I had tried turning the features on and off, rebooting, etc. and it made no difference. Then I looked at the individual drives in IRST (screen shot below) and it stated disk data cache was disabled on all the drives in my first array. Nothing would turn them back on, so all I did was simply re-install the exact same RAID driver, reboot, and verified it was now enabled. Then I ran the benchmark and everything looked good. And I can finally start moving data back to my first array.

DataCache.jpg
 
Yeah, I've experienced it with 3ware cards where although all of 3ware's info is suggesting you have updated your drivers, looking at each individual controller cards in device manager showed the truth.

In that circumstance, you had to manually install the driver for each card.

No idea why in the h*ll 3ware never fixed that. It was a potential disaster waiting to happen. Driver conflicts on a game? Maybe the game crashes. Three 12-port 3ware cards in your system? Two have old drivers, third has new? Driver conflict?

You get my drift...

Sorry that that suggestion hadn't crossed my mind earlier, but I assure it would eventually have been mentioned...

(But aren't all six ports of yours off the same controller? "reinstall drivers" just seems like such a cheap uneducated suggestion most of the time...)
 
Tolyngee said:
(But aren't all six ports of yours off the same controller? "reinstall drivers" just seems like such a cheap uneducated suggestion most of the time...)
Click to expand...

Exactly. That is why I really didn't think it was a driver issue since it wasn't affecting the second array. But as a tech it just reminds me to expect anything even if it doesn't seem to make sense.
 
FWIW - seeing this thread has made me reconsider the potential of the on-board Intel RAID controllers on mobos.

I never would have considered a parity RAID on one 'til now. I just assumed the performance would have to be abysmal.

Then again, on my gaming build, I just decided I no longer wanted to buy any drive below 1.5TB anymore, as the machine (and the drives) will one day be demoted to a file server, and by that time I'd be knocking myself for having yet another six lower-capacity (compared to 1.5/2TB/or great drives) drives in it to potentially sell off...)

NOT knocking your build, your intentions, etc. I admit I tried making a 3.0TB array out of mine (four-drive RAID 10) to verify it can do it, and while the Intel controller can make a 3.0TB array no problem, it is NOT bootable... Not sure what it's limits actually are, come to think of it...
 
I would love a good RAID controller, but technology keeps improving fast. My original server that I built a little over two years ago was mainly just to store movies and music for streaming on all my computers. Worked great with Windows Server 2003. When I wanted to move to Server 2008, it just didn't have the hardware capable of running it. So I decided to use the Gaming rig I built for PCWorld. Since then it basically became my file, testing, learning, experimenting, etc, server. Even with the i7-920 overclocked to 3.6GHz, it is virtually silent. Other than this odd issue, it has been running perfectly for nearly a year now. (Knock on wood.)
 
You must log in or register to reply here.
Back
Top