LSR megabenchmarks

[unhappy_mage]

Well, having lost my data array (not to worry, all I care about is backed up) I figure it's a perfect time to play with the disks and controller. I'll be testing STR and my own benchmark (I'll be writing it in the coming days, and posting source) on the following configurations:

• 3-disk raid 5, 32k stripe size
• 3-disk raid 5, 32k stripe size, degraded
• 3-disk raid 5, 4096k stripe size
• 3-disk raid 5, 4096k stripe size, degraded
• 3-disk raid 0, 32k stripe size
• 3-disk raid 0, 4096k stripe size
• 2-disk raid 1, 32k stripe size
• 2-disk raid 1, 32k stripe size, degraded
• 2-disk raid 1, 4096k stripe size
• 2-disk raid 1, 4096k stripe size, degraded
• Single disk.

Ambitious, I know It'll take me a while to get all this going, especially since I want to write my own tools so I know what they do. My goal is results that reflect the real world, rather than a completely sterile environment, so I'll be doing my usual things on this machine as well - streaming media (~1 mbit) off it, using it as a gateway, running FAH on it, etc. I'll do a single test where it's doing nothing extra, to see how much difference there is between sterile and dirty, but I have too much going through this machine to take it completely offline. Not to mention when I disconnect the network I have to use the actual keyboard, which is in an awkward location, rather than SSH. I'll be reporting any problems I encounter along the way. For now, let me just report my system configuration to get things started.
3*300GB Maxline III and 1*200GB Seagate 7200.7 in Supermicro CS-M35T-1 enclosure
Supermicro 370DER, 2GB ECC reg ram in 2 sticks, 2*866 mHz p3, 1u coolers, Seagate 9GB Scsi disk as boot drive.
Supermicro AOC-SAT2-MV8 using the driver integrated into the 2.6.16.20 kernel.

I won't be using a filesystem on these disks - filesystem performance is a whole barrel of fish that I may decide to investigate... once I'm done with these tests. But testing with a number of variables like that would take me forever, so I'll just stick with this for now

Any comments, suggestions, requests? I may be buying another disk in the near future, to do 0+1 testing, but we'll have to see.

 

[unhappy_mage]

Post reserved for result posting.

 

[longblock454]

Did I miss a post? How was the array lost?

ps. I look forward to the results!

 

[drizzt81]

Hiho U_M while I am crazy-busy with my thesis, if that bench is not going to be a Linux exclusive, I would love to run it on my little test array that I have at home

even if it is, I may be interested.

 

[unhappy_mage]

I may be doing read-write testing. Especially to see the difference in speed between raid 5 and 0.

lb454: Yeah, I posted in the Linux forum. Long story short, booted kernel with bad drivers, two disks drop, attempted to recover by writing new metadata and xfs_repairing it, borked horribly. Reformatted. Ah well, that's what backups are for.

 

[jen4950]

How do plan on running a degraded RAID 0 array?

I would be interested in the RAID 10 array for comparison. Also, although theoretically RAID 01 will perform the same, that might be another combo. Finally, if you do the the 4th disk, try a 4 disk RAID 5, and RAID 6 (or double parity) arrays (if your RAID card supports it).

 

[unhappy_mage]

Yep, I'm an idiot. Fixed.

My "raid card" is linux software raid. The actual card is just plain sata, all the parity calculations, striping, etc. are done in software.

 

[AreEss]

Yep, I'm an idiot. Fixed.
My "raid card" is linux software raid. The actual card is just plain sata, all the parity calculations, striping, etc. are done in software.

I believe I mentioned "I told you so" previously. If not, let me do so now.
Trusting Linux software RAID is like trusting that hooker who says she doesn't have VD. Doubly so for xfs_fsck.

Numbers should be interesting, but sadly irrelevant.

 

[unhappy_mage]

I believe I mentioned "I told you so" previously. If not, let me do so now.
Trusting Linux software RAID is like trusting that hooker who says she doesn't have VD. Doubly so for xfs_fsck.

Numbers should be interesting, but sadly irrelevant.

The problem wasn't LSR, it was the drivers for this controller. Marvell seems to be a little tight with their code; there's a driver that they built by ignoring the whole SCSI layer, and they don't seem to be planning to open it. Thus, I'm not using their drivers; the ones jgarzik is building are what I'm using.

May I assume that you'd like to see *BSD benchmarks instead or in addition? Solaris-x86? Some other OS? I've got spare disks around that I can put the OS on, and the rest of its functionality can probably stand to be down for a while (and I'd like to play ). Any idea how the Marvell 88SX6081 driver on anything else is?

I've written the basics of the STR test (although data reporting is still left to be done), and am working on a media streaming test - read sequentially in large chunks from many locations on the disk, each representing a single client playing a single high-bitrate file. I've done something like this before, but I'm abandoning that code - it's not very useful for what I want to check on. The idea will be to maintain the highest number of clients without starving anyone.

 

[AreEss]

The problem wasn't LSR, it was the drivers for this controller. Marvell seems to be a little tight with their code; there's a driver that they built by ignoring the whole SCSI layer, and they don't seem to be planning to open it. Thus, I'm not using their drivers; the ones jgarzik is building are what I'm using.

Marvell's not keen on drivers to begin with, and any driver on Linux can be considered anti-optimized at best due to the utter lack of proper DMA. Before you address the fundamentally flawed infrastructure, or rather, lack thereof.

May I assume that you'd like to see *BSD benchmarks instead or in addition? Solaris-x86? Some other OS? I've got spare disks around that I can put the OS on, and the rest of its functionality can probably stand to be down for a while (and I'd like to play ). Any idea how the Marvell 88SX6081 driver on anything else is?

Honestly, any "software RAID" benchmark is irrelevant and only good for single machine comparison between configurations. They hold no validity when applied to any other machine. Load patterns, hardware, user count, etcetera all will have noticable and significant impact on any software RAID regardless of the OS.
As far as Marvell drivers go; there are many reasons I dislike Marvell, and the people they have writing their so-called "drivers" is near the top of the list.

 

[longblock454]

Trusting Linux software RAID is like trusting that hooker who says she doesn't have VD.(

Uh? I've run Strip and 5 for years, while upgrading to each develoment release with no problems ever! Your comment above is misplaced for the majority of us.

Linux software raid isn't for corporate database servers! It performs well for it's intended purpose.

 

[unhappy_mage]

Marvell's not keen on drivers to begin with, and any driver on Linux can be considered anti-optimized at best due to the utter lack of proper DMA. Before you address the fundamentally flawed infrastructure, or rather, lack thereof.

I've heard you mention this before, and didn't ask at the time, but now I'm curious. I did some really simple benchmarks, like 'dd' simple, and it looks like I can get the 60ish MB/s (with a single disk) that I get under Windows with Linux as well. I can understand if it's coded in a bad way, or a solution that's not the correct one for the job, but it delivers good speeds no less.

What is a well supported card under BSD/Solaris? I have at my disposal an ICH5, sil3112, the Marvell, and a Highpoint 374+sata bridge card, and if a well-supported card is cheap I wouldn't mind picking up one in the name of science.

Honestly, any "software RAID" benchmark is irrelevant and only good for single machine comparison between configurations. They hold no validity when applied to any other machine. Load patterns, hardware, user count, etcetera all will have noticable and significant impact on any software RAID regardless of the OS.

This is true. But you can say the same about hardware raid, too. I'm attempting to describe my machine in enough detail that my results are replicable with an identical configuration, but it should definitely be understood that any other machine will get different results. And I'm intentionally throwing in some uncertainties - running "normal network traffic" through this machine may mean up to 7 mbits of trafic, usually in small packets.

Final note: whenever you write a multi-process program, make *sure* it's not a fork bomb. I just wrote code that was supposed to start 30 processes and terminate; instead it's doing... um... 30!, I think. Time for a reboot.

 

[AreEss]

I've heard you mention this before, and didn't ask at the time, but now I'm curious. I did some really simple benchmarks, like 'dd' simple, and it looks like I can get the 60ish MB/s (with a single disk) that I get under Windows with Linux as well. I can understand if it's coded in a bad way, or a solution that's not the correct one for the job, but it delivers good speeds no less.
What is a well supported card under BSD/Solaris? I have at my disposal an ICH5, sil3112, the Marvell, and a Highpoint 374+sata bridge card, and if a well-supported card is cheap I wouldn't mind picking up one in the name of science.

Show me someone that hasn't got good ICH5 support, and I'll show you a bunch of incompetent fools. ICH5's are well supported by everyone, though I can't say the same for RAID functionality necessarily.
And speed is absolutely nothing without integrity and stability. In a high load networking test, Linux dropped over 3700 packets, while Solaris x86 dropped 2, Windows dropped 1, and FreeBSD and NetBSD dropped 0. The sample was 10,000 packets; that means 37% packetloss on Linux.

This is true. But you can say the same about hardware raid, too. I'm attempting to describe my machine in enough detail that my results are replicable with an identical configuration, but it should definitely be understood that any other machine will get different results. And I'm intentionally throwing in some uncertainties - running "normal network traffic" through this machine may mean up to 7 mbits of trafic, usually in small packets.

Uhm, love to know where you're getting that from. If I take two LSI Logic Ultra320-2X's, they will perform identically no matter the CPU load or hardware configuration. Provided buses are in spec, I will get 80MB/sec out of the controller on dual Xeons or dual Opterons. I will get 80MB/sec if the Xeons are at 80% CPU load with calculations, and I will get 80MB/sec if the Opterons are at 0% CPU load doing nothing.
This cannot ever be said for software RAID; 1% CPU load can have a significant impact. Even with precisely identical hardware, supposedly minor version changes can and do have a dramatic impact on performance - both positive and negative. Changes to unrelated drivers can have significant negative impact as well. This is not true with hardware RAID; the only behavior which changes hardware RAID performance is generally load on the RAID. Unrelated loads don't affect it significantly, while on software RAID, unrelated loads can cripple the box in a heartbeat.

 

[Nostradamus]

And speed is absolutely nothing without integrity and stability. In a high load networking test, Linux dropped over 3700 packets, while Solaris x86 dropped 2, Windows dropped 1, and FreeBSD and NetBSD dropped 0. The sample was 10,000 packets; that means 37% packetloss on Linux.

Can you explain the "high load networking" test in more detail? I've never heard of a high load test with only 10000 packets.

 

[unhappy_mage]

And speed is absolutely nothing without integrity and stability. In a high load networking test, Linux dropped over 3700 packets, while Solaris x86 dropped 2, Windows dropped 1, and FreeBSD and NetBSD dropped 0. The sample was 10,000 packets; that means 37% packetloss on Linux.

Link to test please? I want to run this myself. Any details on hardware config you used or what config I'll need would be appreciated.

Uhm, love to know where you're getting that from. If I take two LSI Logic Ultra320-2X's, they will perform identically no matter the CPU load or hardware configuration. Provided buses are in spec, I will get 80MB/sec out of the controller on dual Xeons or dual Opterons.

Or dual P2's? Or a Cyrix 266? You said it doesn't depend on hardware configuration, so I reserve the right to use ridiculous hardware to prove my point. Oh, and 80 MB/s sounds pretty bad to me - I'm willing to believe that's the real number, though - unless that's a single disk.

And note - even when the Opterons are sitting completely idle, you don't get any faster transfers. Mightn't it be nice to get faster transfer speeds in that case? For my usage, and that of many home users, there's nothing essential running on this box, except the raid array. Thus, for me, it doesn't matter what amount of CPU running LSR takes.

I understand for your applications every bit of speed you can squeeze out compared to a competitor is important, but for streaming movies over LAN or dumping pictures to a storage location, performance isn't needed. Buying a grade faster CPU will give home users more performance in the usual case than adding that $600 scsi raid controller will.

 

[AreEss]

Link to test please? I want to run this myself. Any details on hardware config you used or what config I'll need would be appreciated.

Can't - really shouldn't even have said that much. NDA'd as it's part of product development. Suffice to say, the hardware was precisely identical down to disks.

Or dual P2's? Or a Cyrix 266? You said it doesn't depend on hardware configuration, so I reserve the right to use ridiculous hardware to prove my point. Oh, and 80 MB/s sounds pretty bad to me - I'm willing to believe that's the real number, though - unless that's a single disk.

As a matter of fact? Yes. If you have the bus, yes, it will be the same speeds. And no, 80MB/sec isn't an actual number, just one I tossed out at random. If I have a 64-bit 66MHz slot available on dual pII's (which I can get) and a 64-bit 66MHz slot available on quad Opterons (which I can obviously get), then yes, the transfers will be roughly the same. Your difference will be within margin of error - 1-3%.

I understand for your applications every bit of speed you can squeeze out compared to a competitor is important, but for streaming movies over LAN or dumping pictures to a storage location, performance isn't needed. Buying a grade faster CPU will give home users more performance in the usual case than adding that $600 scsi raid controller will.

Not really, especially if you're encoding or decoding on the fly or at all. Nor did I specify SCSI; only hardware RAID. REAL hardware RAID, not faker-crap. An Areca will do most home users fine, and provide significant benefit. You don't exactly have an LTO3 library lying around, and boy would it suck ass to lose Season 1 of the Sopranos. Sure, you can backup to CD or DVD, but I don't think you have a Pioneer 7K library either, so that means changing CD/DVD frequently. (And depressingly I've yet to see a DVD version of the venerable Pioneer DRM-62 family.)

The whole point of a hardware RAID card is not always performance; in most cases, it's real reliability versus false-percieved reliability. LSR is significantly less reliable than standalone disks by virtue of introducing multiple new failure points. Hardware RAID doesn't introduce any additional failure points over standalone disks, and has hardware solutions to multiple potential failure points.

 

[unhappy_mage]

Can't - really shouldn't even have said that much. NDA'd as it's part of product development. Suffice to say, the hardware was precisely identical down to disks.

Well, that's convenient. I once saw a network card that pushed 99.9% of theoretical 10gigE without dropping packets, but it's under NDA and I've said too much already.

Anyone can make ridiculous claims, but I'm allowed to post my numbers (albeit for a decidedly low-usage scenario):

# uptime
 16:12:56 up 22:21,  1 user,  load average: 2.01, 2.03, 2.00
root@fs:~# netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   1500 0   1521918      0      0      0  159982      0      0      0 BMRU
eth1   1500 0   1510025      0      0      0 4026762      0      0      0 BMRU

That's no packets dropped in 24 hours. A bad network driver (yes, I agree, Linux's drivers generally suck) could easily cause dropped packets.

As a matter of fact? Yes. If you have the bus, yes, it will be the same speeds. And no, 80MB/sec isn't an actual number, just one I tossed out at random. If I have a 64-bit 66MHz slot available on dual pII's (which I can get) and a 64-bit 66MHz slot available on quad Opterons (which I can obviously get), then yes, the transfers will be roughly the same. Your difference will be within margin of error - 1-3%.

Fair enough - I don't have hardware at my disposal to play with, you do. But my point was that making blanket statements with specific examples in the high dollar end of the field doesn't prove anything.

Not really, especially if you're encoding or decoding on the fly or at all. Nor did I specify SCSI; only hardware RAID. REAL hardware RAID, not faker-crap. An Areca will do most home users fine, and provide significant benefit. You don't exactly have an LTO3 library lying around, and boy would it suck ass to lose Season 1 of the Sopranos. Sure, you can backup to CD or DVD, but I don't think you have a Pioneer 7K library either, so that means changing CD/DVD frequently. (And depressingly I've yet to see a DVD version of the venerable Pioneer DRM-62 family.)

What encoder are you using? Most of the software ones I'm familiar with aren't really suitable for streaming media. And why transcode when you could send the original stream? Like I said, this is aimed at home users, and the cheaper the solution the better. An Areca card may indeed be "fine" for home users, but it's about $400 more than a software solution plus a non-raid sata card that uses the same sata chipset.

The whole point of a hardware RAID card is not always performance; in most cases, it's real reliability versus false-percieved reliability. LSR is significantly less reliable than standalone disks by virtue of introducing multiple new failure points. Hardware RAID doesn't introduce any additional failure points over standalone disks, and has hardware solutions to multiple potential failure points.

Hardware raid introduces new failure points - a piece of hardware, and a driver. Software raid introduces new failure points - a piece of hardware, and a driver. I fail to see the difference in reliability.

That said, I agree that well-done hardware raid has the potential to be more stable than software raid. But as Porgy and Bess say, "it ain't necessarily so". If 1% cpu usage is really cutting into your processing time significantly more than 1%, something's wrong.

 

[longblock454]

Hardware raid introduces new failure points - a piece of hardware, and a driver. Software raid introduces new failure points - a piece of hardware, and a driver. I fail to see the difference in reliability.

That said, I agree that well-done hardware raid has the potential to be more stable than software raid. But as Porgy and Bess say, "it ain't necessarily so". If 1% cpu usage is really cutting into your processing time significantly more than 1%, something's wrong.

Amen Brother!

This FC5 machine has been up for a month, on an old $189 Walmart Linspire PC.

Kernel Interface table
Iface       MTU Met    RX-OK        RX-ERR RX-DRP RX-OVR    TX-OK      TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0 206575100443     27      0      0 345816203487      9      0      0 BMRU

Not too shabby, the errors could have easily been caused by the Netgear switch!

 

[unhappy_mage]

First result is here. I'm pretty much finished with the meat of the STR test (binary here), although I'll be adding a method of generating the graph automatically for less annoyance factor. Yes, it gives a warning on compilation; I didn't feel like looking up how to properly do a conversion from char* to const char*. General usage syntax is:

gcc magemark-str.c -o mmstr
./mmstr /dev/devicefile

This will run the sequential read test on /dev/devicefile, and store its results in /tmp/resultsFile. I strongly suggest you set things up in such a way that /tmp is on another disk than the one you're testing, and to alleviate that possible problem I'll probably make it cache all the results until it's done testing. Here's my results for a 3-disk 32k block raid 5:

Right now the code posted does 32k reads in batches of 1000, and then calculates the megabytes per second for that interval. The number along the edge of the graph is in intervals; I did this graph the simple way (in gnuplot: "plot "/tmp/resultsFile" using 2:1 with dots") but I'll likely do some finangling to get it to display megabytes or something along the bottom (and standardize the axis scaling!).

Last but not least, I'm going on a week-long service camp starting tomorrow, so I'll be returning to this on Monday the 17th-ish. Any comments, suggestions, or more arguments are welcome.

 

[drizzt81]

currently working on porting this to win32.

 

[unhappy_mage]

Cool! I'd be glad to host your results, so it's all in one place.

I had a truly awesome week as a chaperone for a Catholic work camp, for anyone who's interested.

And last but not least, I'm adding an autogeneration of graphs feature - the program will generate the png files by itself, set the ranges properly, and that kind of thing. It'll also save a copy of the data so that one can generate larger-scale graphs with multiple results in them. As mentioned earlier, I'm also storing the results in memory until the testing is over to alleviate problems.

 

[drizzt81]

Cool! I'd be glad to host your results, so it's all in one place.

it's not going to happen for a while (mid-august), since I am unbelievably busy with my Thesis atm.