Something wierd with Areca raid6?

[zephyr]

I upgraded my server a couple of weeks ago to this;

Asus DSBV-DX Serverboard
Xeon 5410
12GB FB-DIMM

Areca 1280ML w/2GB cache
8xSeagate 1.5TB (ST31500341AS) in Raid6

I thing I am getting some strange numbers here, but maybe this is normal?
All tests was taken in a 3 min window.

I think this looks pretty good, but why is write much higher then read??

Suddenly I'm barely getting 350MB/s?

Thinks this looks kind of strange to since that line suggests that the controller or bus is full. That is not the case since the controller is in a x8 slot.

More info on the volume set

Is this normal? (As a glitch in the programs maybe?)
Or is there something going on here?

 

[Syntax Error]

Heh, I wish I got 350MB/s on my 3ware controller.

 

[AreEss]

Those numbers are perfectly normal, and honestly, pretty damn good for an Areca doing RAID6. You interpreted your results wrong; this is not an indicator of bus saturation.

Frankly, 350MB/sec out of SATA is phenomenal. The read drops are normal - you're on SATA. It's seek on a cache miss. If you want faster seek? Get SAS. Period. Sub-11ms average seek puts you on par with much more expensive arrays - the only comparable SATA option on cost is the LSI 8480 in RAID5. (Which for the record, will obliterate your throughput but only match in seek.)

 

[MrGuvernment]

ya 350MB AND with raid 6! double parity...... i know what drives i am getting for my 12 port Areca!

 

[AreEss]

ya 350MB AND with raid 6! double parity...... i know what drives i am getting for my 12 port Areca!

Double Parity does not mean double parity calculations. It means that parity writes are on two disks instead of one. Also, drives have very little to do with it. I can do these numbers with virtually any SATA drive out there.

 

[pissboy]

I have a 1231ML and 1261ML running 11 and 14 drive RAID5s, not really the same setup you have and all but your speeds seem on the slow side. My burst speeds are closer to 925MB/s and 1100MB/s. STR hovers in the 650-700MB/s and 550-600MB/s range. The 12 drive array is running Hitachi 1TB hdds, the 14 drive array has older Seagate 7200.10 500GBs.

On my old setup with the ARC-1230/1GB I was seeing 350-400MB/s STR speeds. The step up to the IOP341 was a huge jump, bringing the same array up to ~500-550MB/s STR.

Synthetic benchmarks are just that though. I found for what I do, simple file copies gave me a better idea of what I could do with the array. I started out with opening a ~4.5GB dvd iso (in dvd shrink) and saving it as a new copy. The app let me know how long the copy took. Initially with my 1230 and 2TB array of 250GB hitachis I was in the 55s range (~75MB/s). I was able to drop it under 30s with the 1231ML, 1GB cache, and 12 of those 500GB seagates.

Make sure to offset the partition you're using (I did 1024k) to align things up and get a nice usable boost in speed.

 

[AreEss]

I have a 1231ML and 1261ML running 11 and 14 drive RAID5s, not really the same setup you have and all but your speeds seem on the slow side. My burst speeds are closer to 925MB/s and 1100MB/s. STR hovers in the 650-700MB/s and 550-600MB/s range. The 12 drive array is running Hitachi 1TB hdds, the 14 drive array has older Seagate 7200.10 500GBs.

Thank you for demonstrating that you lack adequate understanding of RAID.

RAID5 and RAID6 scale with disk count. Thusly, his setup is absolutely within where it should be. He's also doing testing which is triggering cache misses. Your benchmarks are pretty much less than useless here, because the configuration is not even remotely comparable. Oh, and he's also RAID6 which does not directly compare to RAID5.

TL;DR - Your numbers are not only irrelevant and pointless here, but wrong for his configuration too.

 

[zephyr]

Thanks for many good answers here.

Seems like this is normal then?

 

[AreEss]

Thanks for many good answers here.

Seems like this is normal then?

Yeah, pretty much I'd write this down as "damn fine performance." Initial numbers really, are pretty much crap - you want to see a reasonable number that is very stable. As long as you stay at this 350MB/s number over time, you're beautiful. You might see a bit of movement changing your stripe segment size from 128K, but honestly, not enough that I would bother with the hassle of it. Other option would be to do NTFS alignment, which might peek you over 400MB/s on long sequential.
The thing to remember is that for all the marketing drivel out there, your typical SATA disk is still mechanically limited to ~60-70MB/s typical. So, if we take 6 data drives (your scaling factor) and subtract 10% for RAID overhead, 66MB/s per drive is 396MB/s, 10% drops you to ~357MB/sec. (Disclaimer: the 10% overhead is just my average estimate. It can vary wildly.) Can you go faster? Sure! Is it worth the effort for a home server, even here at [H]? Honestly, it really isn't, unless you do it before you put any data on it.

 

[pissboy]

Thank you for demonstrating that you lack adequate understanding of RAID.

RAID5 and RAID6 scale with disk count. Thusly, his setup is absolutely within where it should be. He's also doing testing which is triggering cache misses. Your benchmarks are pretty much less than useless here, because the configuration is not even remotely comparable. Oh, and he's also RAID6 which does not directly compare to RAID5.

TL;DR - Your numbers are not only irrelevant and pointless here, but wrong for his configuration too.

Thank you for showing the world what a dick you can be.

I said in the FIRST line of my post that my current setup was not the same. I HAVE had setups similar and his numbers do appear to be on the low side. I do have a ton of experience with the benefits and shortfalls of the Areca controllers. I first had a 1230 which I had to purchase directly from Taiwan, before Flickerdown and finally Newegg and others picked them up. With a set of old 250GB drives in RAID 6 on an Areca 1230 I was able to hit the numbers he's seeing currently. In case you might not know, the Areca 1230 was based on the Intel IOP333 chipset at 500mhz, the 1280 has the IOP341 at 800mhz. The difference between the two cards is huge. The Areca RAID controllers, like any other on the market do scale linearly with additional spindles, but they do max out the IOP at 8-10 drives.

Also you'll be delighted to know that on the Arecas we're talking about, the difference in speed from RAID5 to 6 is minimal at best.

http://www.areca.us/support/download/RaidCards/Documents/Performance/benchmark.zip
http://www.areca.us/support/downloa...erformance/IOP341_16HDD_PerformanceReport.zip
http://www.areca.us/support/download/RaidCards/Documents/Performance/IOP33x_HDDs_VS_Throughput.zip

 

[extide]

Double Parity does not mean double parity calculations. It means that parity writes are on two disks instead of one. Also, drives have very little to do with it. I can do these numbers with virtually any SATA drive out there.

Actually, it does mean double parity calculations. Say you have 6 drives, 1-6. Given a particular block, say its data chunks are on disks 1-4 and it's parity chunks on 5 and 6.

Raid 6 protects against a failure of ANY two drives right? Well that means that the parity chunk on disk 5 has to be different from disk 6 otherwise if you lost disk 2 and disk 3 (or any two disks in 1-4) you wouldn't be able to recover. If the chunk on 5 was identical to the chunk on 6 it might as well be raid 5. There is definitely more controller overhead for Raid 6. (That being said I am a HUGE fan of Raid 6 especially with these huge drives we are getting these days)

 

[unhappy_mage]

Double Parity does not mean double parity calculations. It means that parity writes are on two disks instead of one.

Uh, no. Parity is calculated in two ways (syndromes) P and Q, each of which is written to disk. The interesting thing about the Areca cards is that they have a relatively fast processor from Intel's IOP series, which lets them calculate the expensive Q syndrome at high speeds.

That said, I agree that pissboy's results are not particularly relevant. Those arrays are significantly wider (scaling factor 10 and 13) than the 8-disk raid6 (scaling factor 6) that the OP has. Take 925 MB/s and multiply by 6/10, and you get 555 MB/s.

OP: Are you sure you have it in the x8 with x8 link slot and not the x8 with x4 link one?

 

[AreEss]

Uh, no. Parity is calculated in two ways (syndromes) P and Q, each of which is written to disk. The interesting thing about the Areca cards is that they have a relatively fast processor from Intel's IOP series, which lets them calculate the expensive Q syndrome at high speeds.

Well obviously. (I probably could write a book on that, at this point.) The difference in Areca is the exact algorithm within the IOP, versus the algorithm used by just about everyone else. I don't know how much I can say about it without getting in trouble, but suffice to say that Areca has a slower processor than much more expensive arrays and performs PQ MUCH cheaper. Frankly, the IOP's aren't that phenomenal for parity calc.

That said, I agree that pissboy's results are not particularly relevant. Those arrays are significantly wider (scaling factor 10 and 13) than the 8-disk raid6 (scaling factor 6) that the OP has. Take 925 MB/s and multiply by 6/10, and you get 555 MB/s.

OP: Are you sure you have it in the x8 with x8 link slot and not the x8 with x4 link one?

You are way overestimating. Just because Areca does it way cheaper, doesn't mean it's RAID5 level. You also have to wait on two disk writes versus one for RAID5. So the actual calculations for RAID6 are significantly different. You're also ignoring mechanical limit - it's completely unreasonable to expect that number under sustained load. There isn't any SATA drive out there that can sustain 100MB/s - forget 150MB/s - under typical load. Your absolute best case is an average of about 66-70MB/s, as I said. (Please note; I have done a tremendous amount of testing with low to extreme high end gear, including IBM DS4700's and DS5300's with SATA. There are people who still want to know how I got the numbers I did.) With consumer SATA drives, a PCIe 4x or 8x (there is not enough difference to matter here - 240MB/s per lane) link, etcetera? 350MB/s is a good, reliable number. Especially if you're not doing consistent full stripe writes, which is probably the case here.

 

[pissboy]

Zephyr, I remembered something else you'll want to watch out for. The stock Areca ML cables are notorious for dying, I lost a 5TB array that way. I'm not the only one who's had those issues, 2cpu it chock full of information on the cards. IF for some reason 4 drives mysteriously disappear from the array make sure you disable the option to automatically remount the volume(s). If you don't do that, you'll end up with 2-3 volumes from the drives that at one point comprised a single volume. The cheapest insurance is to pick up some of the AMCC SFF-8087 to SATA forward breakout cables (if you are going to discrete drives/backplanes). The part number is CBL-SFF8087OCR-05M for the 1/2 meter cable and CBL-SFF8087OCR-10M for the 1meter cables.

 

[unhappy_mage]

Well obviously. (I probably could write a book on that, at this point.)

I think I'd start with writing a coherent and correct post on RAID 6 before moving on to the book.

The difference in Areca is the exact algorithm within the IOP, versus the algorithm used by just about everyone else. I don't know how much I can say about it without getting in trouble, but suffice to say that Areca has a slower processor than much more expensive arrays and performs PQ MUCH cheaper. Frankly, the IOP's aren't that phenomenal for parity calc.

The IOP341 used on many Areca cards has three ADMA units, one of which can perform P+Q calculation in hardware on up to 16 data streams (i.e., 16 source disks). It is fed by a 128-bit-wide, 400 mHz bus (i.e., 6400 MB/s transfer rate). This is obviously limited further by the host interface (2GB/s for x8 PCI express), but it's plenty fast to keep up.

You are way overestimating. Just because Areca does it way cheaper, doesn't mean it's RAID5 level. You also have to wait on two disk writes versus one for RAID5. So the actual calculations for RAID6 are significantly different.

The only difference between raid 5 and raid 6 speeds as far as the IOP is concerned is that there are three ADMA units that will handle raid 5, and only one that handles raid 6.

You're also ignoring mechanical limit - it's completely unreasonable to expect that number under sustained load. There isn't any SATA drive out there that can sustain 100MB/s - forget 150MB/s - under typical load. Your absolute best case is an average of about 66-70MB/s, as I said.

What does "sustained load" mean? Sequential reads? I regularly see 80+ MB/s sequential from single sata disks.

There are people who still want to know how I got the numbers I did.

I can believe that.

350MB/s is a good, reliable number. Especially if you're not doing consistent full stripe writes, which is probably the case here.

All the tests I see done are read-only.

 

[MrGuvernment]

good info here, i thought raid 6 did have more overhead why it can perform, sometimes around %30 slower then raid 5 in certain situations due to the extra work that has to be done.

 

[AreEss]

The IOP341 used on many Areca cards has three ADMA units, one of which can perform P+Q calculation in hardware on up to 16 data streams (i.e., 16 source disks).
-snip-

And this compares to hardware arrays how exactly? As I said; it does it cheaply, and on par with much faster hardware. Way to make a great many assumptions about how things operate. And making it completely impossible for me to even think about posting additional details anywhere here. When I say I can't say more, it's because I have information you don't have and can't get access to.

The only difference between raid 5 and raid 6 speeds as far as the IOP is concerned is that there are three ADMA units that will handle raid 5, and only one that handles raid 6.

Yes, that's totally an inconsequential difference.

What does "sustained load" mean? Sequential reads? I regularly see 80+ MB/s sequential from single sata disks.

It means real world which sequential is not. Period. The sole exception is TSM systems that were properly configured - and Windows servers with TSM properly configured? Beyond rare.

I can believe that.

Of course you can. The difference here is that one of us gets calls from real storage engineers asking how to do it.

 

[AreEss]

good info here, i thought raid 6 did have more overhead why it can perform, sometimes around %30 slower then raid 5 in certain situations due to the extra work that has to be done.

On most controllers, RAID5 to RAID6 is a 20-35% performance penalty. Areca is the only one out there where it's ~10% typical, data disk for data disk. Meaning a 6 disk RAID6 (4+2P) will typically only be ~10% slower than a 5 disk RAID5 (4+P). On the most common mid-range hardware, you'll see a much larger performance penalty.

 

[unhappy_mage]

And this compares to hardware arrays how exactly? As I said; it does it cheaply, and on par with much faster hardware. Way to make a great many assumptions about how things operate.

I was talking about the Areca, not hardware arrays. I misread your sentence as suggesting that the IOP was bad at generating PQ parity, when in fact it's done in dedicated hardware very quickly.

And making it completely impossible for me to even think about posting additional details anywhere here. When I say I can't say more, it's because I have information you don't have and can't get access to.

Ooh, scary. Let me know when the Freemasons have cleared the information for release.

If the information isn't available to the general public, then it's not very relevant to a general-public discussion.

Yes, that's totally an inconsequential difference.

If the task is killing flies, it doesn't really matter whether I have one shotgun or three. Unless I'm grossly misreading the datasheet, one ADMA unit can generate parity from 6400 MB/s of input (up to 16 source disks). This means that disks which transfer at less than 400 MB/s are not bottlenecked by the ADMA unit---one could (given a fast enough bus) take 6400 MB/s of input data and produce PQ data at 800 MB/s, then write the new 7200 MB/s stream to disk, with only one ADMA unit active.

It means real world which sequential is not. Period. The sole exception is TSM systems that were properly configured - and Windows servers with TSM properly configured? Beyond rare.

Okay, just checking. Not completely sequential, those are reasonable rates to get off a sata disk.

Of course you can. The difference here is that one of us gets calls from real storage engineers asking how to do it.

Oh, I see. People ask you questions, so you're right.

I honestly don't mind that you know more about this stuff than I do. But I do mind that you make assertions and fail to offer any proof or data or further reading. Appeal to authority only goes so far, and you've used up your credits.