• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

RAID 5 speeds

K

Karandras

[H]ard|Gawd
2FA
Joined
Feb 16, 2001
Messages
1,873
Hey all,
I'm definitely not an expert on speeds of controllers or HDDs so I'm looking for a bit of input here before I go ahead and change the setup.

So I have a c2004 rackable sever with 4x 500gig 7200 SATA HDDs hooked to a 3ware 9500S-4LP controller. I'm running CentOS 5.6 as the host OS for VM Server 2.0.2 and I have two machines on there not doing much. However I wanted to change the size of the allocated hdd for one of the machines (delete and recreate it) so I removed it from the datastore and to delete 600gig of stuff it took about 12 minutes and top showed me the load was around 12-18 for 5 min. I googled how to test for hdd speeds but I'm not sure if this is any good:

[root@vmserver01 temp]# for i in 1 2 3; do hdparm -tT /dev/sda; done

/dev/sda:
Timing cached reads: 3704 MB in 2.00 seconds = 1853.53 MB/sec
Timing buffered disk reads: 162 MB in 3.03 seconds = 53.39 MB/sec

/dev/sda:
Timing cached reads: 3692 MB in 2.00 seconds = 1846.30 MB/sec
Timing buffered disk reads: 158 MB in 3.00 seconds = 52.61 MB/sec

/dev/sda:
Timing cached reads: 3696 MB in 2.00 seconds = 1847.81 MB/sec
Timing buffered disk reads: 160 MB in 3.01 seconds = 53.08 MB/sec

[root@vmserver01 temp]# dd if=/dev/zero of=/tmp/output.img bs=8k count=256k
262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 15.6371 seconds, 137 MB/s

I'm considering changing to a 1+0 configuration to increase the speed but I don't fully understand the results and if they are good or bad for the current setup. Any ideas or if the configuration change will affect the speed?

Thanks for the info!
 
Hah, I like that "Those numbers are very bad." funny. yes, it's in a 64bit PCI slot. Anything I should be tweaking in the bios or in centos? What kind of numbers should I be seeing with a 4 drive raid 5 sata setup?
 
drescherjm said:
I would expect to be somewhere around 2 drive performance for large reads and somewhere between 1 and 2 drive performance for large writes.

Can you use dd to do a read benchmark from the array. See my post here:
http://hardforum.com/showpost.php?p=1037564636&postcount=3
Click to expand...

Ok, took your advice and this is piss poor performance :-/

[root@vmserver01 ~]# dd if=/dev/zero of=/tmp/output.img bs=1M count=16000
16000+0 records in
16000+0 records out
16777216000 bytes (17 GB) copied, 498.305 seconds, 33.7 MB/s

Should I switch this to RAID 1+0 instead of RAID 5 to see if the card will perform better? I'll lose a lot of space but performance like this is useless.

Thanks.
 
Should I switch this to RAID 1+0 instead of RAID 5 to see if the card will perform better?
Click to expand...

I would try that. Do you have a BBU? Also the problem may be bus related (system not using full speed of 64 bit 100 MHz (or better) PCI-X)
 
Last edited:
With a load of 12-18 during the delete, not sure if 0+1 will help much since you are using a hardware card, but it might give us more information to go on.
 
drescherjm said:
I would try that. Do you have a BBU? Also the problem may be bus related (system not using full speed of 64 bit 100 MHz (or better) PCI-X)
Click to expand...

I'll go through the bios as well to see if there is something that is not set properly as well before I go ahead and change the array

adi said:
With a load of 12-18 during the delete, not sure if 0+1 will help much since you are using a hardware card, but it might give us more information to go on.
Click to expand...
Like drescherim mentioned, could be a bus issue, the CPU and Memory were not being utilized (CPU was under 2-5% and 3.5G of memory free)

I have a couple of things to take a look at, I'll get back to yall later this evening.

Thanks.
 
Well, checked ths bios but couldn't find anything out of place specially with the pci slots.

Are there any checks that i can do through linux to verify the bus speeds? I'll switch the array to rais 10 but that'll take a couple hours tonight.
 
For theoretical calculations, Raid 1 0 will give you 2x the write and 4x the read IOPS of a single drive.

Where raid 5 will give you 4x the read and 0.25 the write IOPS of a single drive. Of course, a real hardware raid 5 controller will increase the write IOPS with on board hardware / memory to help calculate the parity.
 
4x the read and 0.25 the write IOPS of a single drive
Click to expand...

It should be (n-1) X read and I am not sure about write IOPS. I believe you calculation assumes that the SATA controller can only access 1 drive at a time. This surely is not the case for my Intel rebranded LSI SAS HBAs that I use at work with linux software raid.
 
Also the read speed is assuming sequential. Random reads will tend to behave better with raid1/raid10.
 
drescherjm said:
It should be (n-1) X read and I am not sure about write IOPS. I believe you calculation assumes that the SATA controller can only access 1 drive at a time. This surely is not the case for my Intel rebranded LSI SAS HBAs that I use at work with linux software raid.
Click to expand...

I think his IOPS figure is almost correct for random writes smaller than the stripe width. The reason is that every write will require reading some of the disks (to compute the new parity), then writing to at least two disks (data and parity).

Although 0.25 may be pessimistic. If every write involves reading an entire stripe, and then writing an entire stripe, then I would think the figure should be closer to 0.5 than 0.25 the IOPS of a single drive. Also, in some cases you would not need to read the entire stripe nor write the entire stripe, just a portion of it, so that should bring the figure up even higher. But I do think it will be less than 1, and greater than 0.5, for random writes less than a stripe width.

That is all assuming that there is minimal cacheing. If there is a large write cache, then some of the writes that happen to be to the same stripe can be collected together, thus increasing the throughput.
 
Last edited:
So rebuilt the server in raid 1+0 and it's about triple the speed but still nothing like what drescherjm's datastore3 server is showing (819 MB/s)... I would like those speeds ;-)

Any other suggestions? Would using a basic SATA controller card and linux software raid be any better?

[root@vmserver01 ~]# dd if=/dev/zero of=/tmp/output.img bs=1M count=16000
16000+0 records in
16000+0 records out
16777216000 bytes (17 GB) copied, 184.109 seconds, 91.1 MB/s
 
Karandras69 said:
So rebuilt the server in raid 1+0 and it's about triple the speed but still nothing like what drescherjm's datastore3 server is showing (819 MB/s)... I would like those speeds ;-)
Click to expand...

819 MB/s was raw read performance and that is with 9 x 2TB last generation hitachi 7k2000s in linux software raid 6 (64 bit gentoo-sources-2.6.39) using a combination of motherboard ports and an Intel sas sasuc8i. The machine is a 2.83 GHz core2quad with 4GB of 800 MHz DDR2 it has an ASUS P5Q Pro motherboard.
 
Ha, yeah I understand that this server is 7 years old and I've beefed it up as much as I can (aside from a really good raid controller) but I'm wondering what I can do (if anything) to get these speeds up higher so I don't have issues with my VMs while they are running.

PS - What's the cost of your machine? Is it just file server or do you host off of it ( fiber channel to an ESX host?)

Thanks.
 
With a PCI RAID controller you probably won't get much higher. 3ware has always been the slowest controller but one of the most reliable.
 
Karandras69 said:
PS - What's the cost of your machine? Is it just file server or do you host off of it ( fiber channel to an ESX host?)

Thanks.
Click to expand...

It's mainly a fileserver for medical images but provides linux containers virtulization for 2 guests ( 2nd backup domain controller samba is one of the guests). We bought the server (mobo + case + powersupply + cpu + mem) in late 2008 for $600 to $700 minus hard drives. Originally I had 10 x 500 GB SATA drives in it connected to motherboard ports + a 4 port adaptec SATA II card. Earlier this year I purchased 10 2TB hitachi 7k2000 drives for around $110 each at the approved vendor. I replaced the adaptec SATA2 card with the ~$155 Intel SAS card + 2 x $15 or so SAS forward breakout cables.
 
make sure all drivers have updated firmware and that 3ware card does

i had really bad performance on my 3ware 9560? i think it was until i update firmware on the drives and the card.
 
MrGuvernment said:
make sure all drivers have updated firmware and that 3ware card does

i had really bad performance on my 3ware 9560? i think it was until i update firmware on the drives and the card.
Click to expand...

Found the tw_cli program from 3ware for linux and checked the firmware (too lazy to hook up a monitor and watch it boot to see the firmware ver :rolleyes:). It was something like 2.02 and the latest is 2.08 so a few versions behind. Updated to the latest (the tw_cli is a sweet cli for updating fw) New stats for RAID 10:

[root@vmserver01 temp]# dd if=/dev/zero of=/tmp/output.img bs=1M count=16000
16000+0 records in
16000+0 records out
16777216000 bytes (17 GB) copied, 146.751 seconds, 114 MB/s

Yay, broke 100MB/s ;-)

PS - No BBU on this unit :(
 
In all honesty, for a system from 2004, I'm not sure how you can expect much better.

Is this the controller in question? LINK

If so, I'm amazed you are getting the speeds you are. That controller is ancient and you may actually have better luck with software RAID at this point, especially with CentOS.

The 3ware 9500S hardware RAID controllers deliver in excess of 400 MB per second (MB/sec) sustained RAID 5 reads and over 100 MB/sec RAID 5 sequential writes with less than 3% CPU utilization.
Click to expand...

This is about the speeds we are seeing from your system.

Any further performance will only be gained by a newer, more powerful hardware RAID card, or by using software RAID (assuming your CPUs can handle it).

The card you are using isn't bad, it's just very antiquated by today's standards.

numbers arent so good my friend
Click to expand...
Hey, for a 2004 system, those numbers are actually pretty good with the hardware considered.
 
Even with the 3ware 9650-se controllers 3ware was dog slow with any parity raid. I saw better speeds putting the controller in JBOD mode and doing software raid (mdadm) over the controllers (significantly better speeds).
 
Yes that is the exact card that I have running in the system.

I was thinking, if it might be faster to put in a promise sata300 TX3 ata II pci controller then running software raid on CentOS. I have 2xOpteron 280, not sure how much CPU will be used or if these will be powerful enough to run raid5 or raid10.
 
In raid6 on the 9650's I saw like maybe 100 MB/sec reads and around 20MB/sec (not joking) writes in raid6. I am sure raid5 is significantly better but those cards are trash. Using mdadm I was able to get like 500-550 MB/sec reads and 180 MB/sec writes when putting the card in jbod mode and using software raid. Those cards are *PATHETICALLY* slow with parity raid.
 
houkouonchi said:
In raid6 on the 9650's I saw like maybe 100 MB/sec reads and around 20MB/sec (not joking) writes in raid6. I am sure raid5 is significantly better but those cards are trash. Using mdadm I was able to get like 500-550 MB/sec reads and 180 MB/sec writes when putting the card in jbod mode and using software raid. Those cards are *PATHETICALLY* slow with parity raid.
Click to expand...
Since the OP is not using Windows, enjoy the luxury of the software RAID daemon, MDADM.
 
Last edited:
houkouonchi said:
In raid6 on the 9650's I saw like maybe 100 MB/sec reads and around 20MB/sec (not joking) writes in raid6. I am sure raid5 is significantly better but those cards are trash. Using mdadm I was able to get like 500-550 MB/sec reads and 180 MB/sec writes when putting the card in jbod mode and using software raid. Those cards are *PATHETICALLY* slow with parity raid.
Click to expand...


i have the 9650SE with 8x1.5TB RAID6
i just did hdparm -t and read speeds are 294 MB/sec
it used to be faster, disks are nearly full now
 
john4200 said:
Good thing the OP said he was running linux. :D
Click to expand...

But that makes far too much sense! =head explodes= :eek:

Good call. OP, yes, use MDADM, I believe it will work to your advantage over the hardware RAID you are using at the moment.
 
Would it be better to use the 9500s or the promise sata300 TX3 controller for linux software raid?
 
Karandras69 said:
Would it be better to use the 9500s or the promise sata300 TX3 controller for linux software raid?
Click to expand...

Did you say what motherboard you have? Does it have any SATA ports? If so, use as many of those as you can.

I'm not familiar with either of the cards you mentioned. Do they both support pass-thru of the drives so that the OS can see each drive individually? /dev/sda, sdb, etc.?

If so, your best bet may to be use both cards with the drives divided between the cards. But if the PCI bus is the limiting factor, then forget that idea, since both cards would be sharing PCI bus bandwidth. But if the throughput of the card itself is the limiting factor, you might be better off using both.

Anyway, the main idea is to get all the drives connected in such a way that if you read or write to all the drives in parallel, you are getting the maximum possible total throughput. That is how you will get the best performance with mdadm.
 
Unfortunately with the 2u llayout of this server the hdds are right above the pci slots and I can only fit one riser card in, that's why im looking for best choice.

The mobo is HDAMA rev.g with no onbhoard sata ports :(

I will try with having the card as njust a controller and no raid and mdadm, see how that does with ther opterons.
 
Thanks everyone so far for the excellent info on this topic!

Now I'm trying to create a raid10 array using the disk that linux is installed to. Since the 4 disks are the only ones in the machine I don't have much of a choice unless it would be worth it to pick up an 8 gig flash drive and install centos on that then create the array from that drive.

Any linkies to help me get this process done properly so I can get the best results out of this machine?

Thanks!
 
I must have a magical 3ware 9560SE-8LPML considering how everybody else seems to get crap performance with the card. The two main issues I've seen people with the 9560SE is, A) the card is detecting the slot as having a PCI-E x1 link instead of PCI-E x4 (motherboard's fault) or B) Write Caching is disabled & StoreSave isn't set to Performance.

~382MB/s read & ~392MB/s write with a 1GB test at the beginning of a 4-drive 2TB 7K2000 RAID-5. Today I just added a 4-drive 3TB 5K3000 RAID-5 and early results are showing ~345MB/s read & ~350MB/s write (currently initializing, so I set priority to 1 for benchmarking). That is pretty much perfect scaling for RAID-5, single drive performance x (# of drives - 1). For RAID-6, perfect scaling would be single drive performance x (# of drives - 2) Maybe the card just sucks with 8-drive arrays and RAID-6? :confused:
 
Last edited:
So looks like the mdadm setup that I was hoping would be the best choice but it looks like having the raid10 on the 9500s will be my fastest option:


[root@vmserver01 ~]# dd if=/dev/zero of=/tmp/output.img bs=1M count=16000
16000+0 records in
16000+0 records out
16777216000 bytes (17 GB) copied, 637.537 seconds, 26.3 MB/s

[root@vmserver01 ~]# cat /proc/mdstat
Personalities : [raid10] [raid1]
md0 : active raid1 sdd1[3] sdc1[2] sdb1[1] sda1[0]
104320 blocks [4/4] [UUUU]

md1 : active raid10 sdd2[3] sdc2[2] sdb2[1] sda2[0]
6297088 blocks 256K chunks 2 near-copies [4/4] [UUUU]

md2 : active raid10 sdd3[3] sdc3[2] sdb3[1] sda3[0]
970020352 blocks 256K chunks 2 near-copies [4/4] [UUUU]

unused devices: <none>


That's the worst one yet :( Any adjustments that I can do to mdadm to get better results?
 
There is not much tuning for raid 0, 1, or 10. Looks like a bus speed problem. You can verify this by doing read benchmarks on all 4 drives simultaneously.
 
You must log in or register to reply here.
Back
Top