RAID 5 Chunk Size

[Flapjack]

So I read a ton of stuff about RAID setup and chunk size. I even found a page with a bunch of benchmarks.

From what I read, I glean:

- Smaller chunks equal faster writes
- Smaller chunks equal slower reads
- Smaller chunks equal less wasted space (great for small files)

- Larger chunks equal slower writes
- Larger chunks equal faster reads
- Larger chunks are good for large media files (what I'm working with, since this is a media array)

After learning all this, I set the chunk size as large as it would go: 128K

What I did not expect is for this thing to be so damned slow. I set up five 1TB drives using a Sans Disk TR5M RAID tower (5-bay) and the included hardware RAID card (Sii 3132), connected over an eSATA connection.

I had 1.47TB of data to move to the new 4GB array (after parity stripe). After a full 24 hours, it's only 1/2 done!! I started the transfer last night at 10pm and at this rate, it won't be done until tomorrow night at 10pm (48 hours total!).

So I'm wondering if I shouldn't scrap the whole thing and choose a smaller chunk size? This is ridiculous!

 

[Flapjack]

A little more info...

The displayed transfer rate (by windows) is 8.86 MB/s (shown below).

The drive it's coming from is a dual bay USB enclosure with two 1.5TB drives set in "BIG" mode as a 3TB array with no fault tolerance.

I would suspect the USB connection at first, but then again, I don't remember it taking this long to write from the eSATA RAID to the enclosure. In fact, the enclosure software wasn't working correctly at the time and the drives were only showing up in Windows. At the time, I had three 1TB drives setup in a RAID0 software stripe in Windows.

 

[danman]

I doubt stripe size is your problem. That bay is probably really slow anyway. And eSATA is a heck of a lot faster than USB 2 is.

 

[Flapjack]

I pretty much agree with you, but like I said above... before I converted the RAID array from three 1TB drives in RAID 0 to five drives in RAID 5, I needed a place to move the data to. I copied the entire 1.47TB to the USB enclosure so I could add the new drives and rebuild the array in the new config. It only took a few hours.

This is even more compounded by the fact that writes are typically slower than reads, so if anything, reading back from the USB enclosure to the RAID box should've been faster, than the previous move... not slower.

There is something else at play here....

One theory I have...
I saw a review somewhere that someone posted it took HOURS to for a full format of his new array in Windows and that he wished he did a "quick format" instead. He then went onto to say that the drive would've been slower until it was written to anyways.

Exactly how does quick format work? This array had never been formatted, so how does this work? Could Windows be formatting the drive prior to copying data? Since I don't know how quick format works, I can't be sure....

 

[palmboy5]

Quick format doesn't check for bad sectors. As a result, you can write data to a quick formatted drive and have no idea that your data got corrupted by any bad sectors until you do a disk check or realize the file won't open (properly).

 

[Flapjack]

But does it slow it down at all? Could that potentially be the reason for the slow transfer?

I also sent a very detailed email to San Disk. Hopefully they come up with something...

 

[john4200]

I copied the entire 1.47TB to the USB enclosure so I could add the new drives and rebuild the array in the new config. It only took a few hours.

That is impossible. USB2 is 480 Mbps, or 60 MB/s. But due to overhead that speed is never achieved. Let us be generous and say that you achieved write throughput of 30 MB/s when you copied your data to the USB enclosure.

1.47e6 MB / (30 MB/s) / (3600 s/hr) = 13.6 hours

And unless you were copying nothing but very large files (~10MB or larger), you would not have achieved 30 MB/s. The latency on USB is high, and copying smaller blocks slows things down considerably.

 

[Flapjack]

That is impossible. USB2 is 480 Mbps, or 60 MB/s. But due to overhead that speed is never achieved. Let us be generous and say that you achieved write throughput of 30 MB/s when you copied your data to the USB enclosure.

1.47e6 MB / (30 MB/s) / (3600 s/hr) = 13.6 hours

And unless you were copying nothing but very large files (~10MB or larger), you would not have achieved 30 MB/s. The latency on USB is high, and copying smaller blocks slows things down considerably.

Ok, I can see your argument, and to your credit, I don't remember the exact time I started... but it was about the time the kids went to be (NLT 9pm). It certainly was not much more than overnight. I'm pretty sure I started it at night and it was done in the AM. I can't remember if I checked it before I left for work, so for the sake of argument, let's say I came home from work (5pm) and it was done. That's 20 hours. I know this was true because I was able to put a movie on for kids as soon as I got home while I was making dinner. That much, I remember.

At 24 hours, the transfer back was still not done. It was halfway done. So at the bare minimum, it more than half the speed of the write to the USB enclosure... and again, writes take longer than reads. I'm reading from a slower device and writing to a faster device. While the bottleneck should be the USB enclosure, it would appear that the bottleneck is the RAID box.

Your point about smaller blocks slowing things down is irrelevant. I've already stated I'm using the largest chunk/block size on the RAID array and the file sizes are anywhere from 700MB to 16GB in size (from compressed AVI DVD rips to high bit rate 1080p BluRay rips).

So back to the original problem... why would this be happening?

 

[john4200]

You do not understand how chunks (stripes) work on RAID, as evidenced by your original point about "smaller chunks equal less wasted space". The RAID chunk (stripe) size has no effect whatsoever on "wasted space". You are confusing file system block size with RAID chunk size -- two completely different things. So my point is NOT "irrelevant".

Also, you are incorrect that writes are always slower than reads. With random writes and a write back cache and/or NCQ, it is quite possible that the physical writes end up being nearly sequential, thus minimizing HDD track-to-track seeks. But depending on the order the data is read back, you could be doing a lot more seeks, thus making the reads slower.

Anyway, danman is correct, the RAID chunk size is not the bottleneck for your copy. Look elsewhere.

 

[white_devil]

The Sii 3132 is software RAID, not hardware, and is most likely the problem. This sounds like the performance hit a lot of RAID controllers take when write-caching is disabled. Per THIS PAGE write caching is configured in the registry, maybe give that a try.

*edit*
Take the screenshot while copying some porn over and have to black it out?

 

[Flapjack]

You do not understand how chunks (stripes) work on RAID, as evidenced by your original point about "smaller chunks equal less wasted space". The RAID chunk (stripe) size has no effect whatsoever on "wasted space". You are confusing file system block size with RAID chunk size -- two completely different things. So my point is NOT "irrelevant".

I am absolutely NOT confused. You are, in fact, the one who's confused... and your argument is still irrelevant. I findi your arrogance on this subject annoying. BEFORE posting here, I did some research on both ACM and IEEE, which you either need to be a confirmed computer scientist or engineer (of which I am both) to be allowed in (or a student of either major). I'd link the pages, but based on your lack of knowledge, I'm guessing you probably don't have access to either site. I'm here to solve a problem and to learn, if there's something to learn. When someone comes on here spouting like they know the truth and being completely wrong, I take issue with it.

For the sake of argument, I'll link you to a very well-written article at Tom's Hardware, by Patrick Schmid (THW editor-in-chief), which is about as technical as the "open" web gets.

Stripe Size Discussion - By Patrick Schmid
http://www.tomshardware.com/reviews/RAID-SCALING-CHARTS,1735-4.html

"Capacity Used
The stripe size also defines the amount of storage capacity that will at least be occupied on a RAID partition when you write a file. For example, if you selected a 64 kB stripe size and you store a 2 kB text file, this file will occupy 64 kB. Obviously, the stripe size defines the minimum amount of data that a RAID controller will distribute files across its hard drives, in my example, 64 kB. As long as a file can be written onto a single stripe or a single drive, you won’t have any advantage from running this particular RAID array."

Also, you are incorrect that writes are always slower than reads. With random writes and a write back cache and/or NCQ, it is quite possible that the physical writes end up being nearly sequential, thus minimizing HDD track-to-track seeks. But depending on the order the data is read back, you could be doing a lot more seeks, thus making the reads slower.

If you want to take my post literally, go right ahead. Of course, unless you want me to whip up a little chart for you that shows a variety of different after running a simulated benchmark (something I had to do a few classes back), I could do such. The fact of the matter here is that writes typically take longer than reads... period. What you're talking about is sequential, cached writes that is in fact not only rare, but most certainly unsustainable. You might have a burst write that empties the cache and stores data slightly faster than your typical read, but we're talking about large files here. Where NCQ shines is preserving the life of a drive. It works best with sequential reads (which are typically large files on a properly maintained disk) because it reduces the number of head movements the HDD has to make to access data. On paper, it's great. In reality, it's buggy, overhyped, and typically better off disabled (especially if you don't like losing data).

So here, we have another irrelevant point, as it is standard practice is to disable NCQ for RAID setups (check TechNet, if you don't believe me). I've run benchmarks in RAID 0, RAID 1 and RAID 5 and NCQ is a hit on all, regardless of whether or not I was running hardware or software RAID. I've even had serious system instability when running NCQ on a RAID array hosting the OS partition (but you already knew that):
http://www.evga.com/forumsarchive/tm.asp?m=114381

Anyway, danman is correct, the RAID chunk size is not the bottleneck for your copy. Look elsewhere.

I've already figured this out. I was second-guessing myself (in my first post) and have now started looking into other possibilities (in case you read the thread... as I've mentioned several).

I welcome further debate with you, but I'll leave it up to you whether you'd like to take this offline or continue via PM. I can safely assure you I know what I'm talking about here. I'm ready to keep this up, but I'd MUCH rather focus on figuring out what the bottleneck is.

The Sii 3132 is software RAID, not hardware, and is most likely the problem. This sounds like the performance hit a lot of RAID controllers take when write-caching is disabled. Per THIS PAGE write caching is configured in the registry, maybe give that a try.

I'll definitely look into this. As it stands, I'm about 3hrs from being finished writing the initial 1.47TB. Looks like it's right on track for the 48-50 hour guestimate I made before I posted.

Could you kindly explain how the Sii 3132 is software RAID? It is relying on the CPU? I highly doubt it, as CPU usage is sitting at a steady 95-97% idle, with the largest 2nd thread being the Logitech camera monitoring software running on my system (yes... I already disabled the cameras, as they were writing to the USB enclosure temporarily. The CPU usage is the system service still running in the background).

*edit*
Take the screenshot while copying some porn over and have to black it out?

No pr0n here. I have a beautiful fiancee, a healthy you-know-what life, and not enough free time to waste on pr0n. Besides, that shit got old for me back in 1996 when you couldn't mistype a website without getting some fake-ass-bitch and a million popups in your face. I'd opt for the real thing any day.

 

[Dangman]

Could you kindly explain how the Sii 3132 is software RAID? It is relying on the CPU? I highly doubt it, as CPU usage is sitting at a steady 95-97% idle, with the largest 2nd thread being the Logitech camera monitoring software running on my system (yes... I already disabled the cameras, as they were writing to the USB enclosure temporarily. The CPU usage is the system service still running in the background).

The specs for the Sil 3132 show no onboard CPU/Processor that handles all of the parity calculations. It uses the PC's CPU to do all of the parity caculations. In fact, I haven't found ANY info showing Silicon Image actually having an integrated XOR engine on ANY of their cards. All of their cards seems to be software driven.

Finally it says so in the Reference brief for that card:
http://www.siliconimage.com/docs/pb3132_SiI-pb-52_r1_FINAL-7-18-06.pdf

Note the references to a Silicon Image software RAID driver for RAID 0 and RAID 5.

 

[john4200]

For the sake of argument, I'll link you to a very well-written article at Tom's Hardware, by Patrick Schmid (THW editor-in-chief), which is about as technical as the "open" web gets.

Stripe Size Discussion - By Patrick Schmid
http://www.tomshardware.com/reviews/RAID-SCALING-CHARTS,1735-4.html

"Capacity Used
The stripe size also defines the amount of storage capacity that will at least be occupied on a RAID partition when you write a file. For example, if you selected a 64 kB stripe size and you store a 2 kB text file, this file will occupy 64 kB. Obviously, the stripe size defines the minimum amount of data that a RAID controller will distribute files across its hard drives, in my example, 64 kB. As long as a file can be written onto a single stripe or a single drive, you won’t have any advantage from running this particular RAID array."

That is actually incorrect. Like you, he is confusing RAID chunks with file system blocks.

All that RAID does is control the location of LBAs that are read and written from a group of drives. The file system is what determines which LBAs are used to store a file. There is no way for the RAID controller to know that one file has stopped and another has started (that is the job of the file system). All the RAID controller sees are a bunch of LBAs and data going to and from those LBAs. All the RAID controller does is stripe those LBAs across multiple drives. Consequently, the chunk size has absolutely no effect on "wasted space". In contrast, a large file system block size (i.e., the smallest unit that the file system keeps track of in an allocation table) can result in wasted space if there are many files stored that are smaller than the file system block size.

Note that chunk size is the term preferred by many linux users (particularly mdadm users) to indicate the amount of data read/written to each HDD before skipping to the next HDD in the array. Many people also call that the stripe size, but I prefer chunk size, since then a stripe is equal to the chunk size times the number of HDDs in the array. But you seem to be confusing RAID chunk size with the smallest allocation unit or block used by a file system.

There is really no point in discussing further with you since you "know" absolutely that you are correct.

 

[white_devil]

The specs for the Sil 3132 show no onboard CPU/Processor that handles all of the parity calculations. It uses the PC's CPU to do all of the parity caculations. In fact, I haven't found ANY info showing Silicon Image actually having an integrated XOR engine on ANY of their cards. All of their cards seems to be software driven.

Finally it says so in the Reference brief for that card:
http://www.siliconimage.com/docs/pb3132_SiI-pb-52_r1_FINAL-7-18-06.pdf

Note the references to a Silicon Image software RAID driver for RAID 0 and RAID 5.

Yup, what he said. I've never seen a Silicon Image card running hardware RAID either. Also, a quick check of Newegg shows Sil 3132 cards selling for ~$20-30 which definitely isn't getting you hardware RAID. I'd say the reason you're not seeing any CPU activity is because 8MB/s isn't exactly taxing.

 

[brandinb]

Ya trust me the raid chunk size has nothing to do with wasted space. It just determines the performance of different file sizes depending if they will be striped or not on a raid array.

Also that is a low performance silicon image software raid engine using a low performance port multiplier so you will get crap performance. If you want decent performance then let the controller work as a pass through and use windows to setup a jbod or seperate drives. Also if the controller supports raid 10 then performance would be decent there as the port multiplier bandwidth wouldnt be saturated by parity calcs.

good luck and perhaps dont act so cocky haha

 

[sub.mesa]

Silicon Image is fakeRAID just like JMicron and Promise FastTrak. It is 0% hardware and 100% software RAID. The only hardware is a Serial ATA controller and BIOS bootstrap support and the BIOS RAID setup; both which only write and read the 512 bytes of metadata stored on each disk connected to the Silicon Image controller.

Silicon Image is used in external "DAS" casings to provide RAID5 support, and also on the desktop it supports RAID5. But its implementation is comparable to the one in Windows: no write-back; always slow write performance and generally very low reliability; less than a single drive without RAID.

In my tests i can confirm that a low stripesize benefits sequential I/O, and a high stripesize benefits random I/O with striping RAID; including RAID5. I can also confirm that:

The stripe size also defines the amount of storage capacity that will at least be occupied on a RAID partition when you write a file. For example, if you selected a 64 kB stripe size and you store a 2 kB text file, this file will occupy 64 kB.

Is definately not true. This is filesystem limitation; the only limitation imposed by the raid array is the sector size; and it is still 512 bytes.

Watch some of my RAID0 stripesize tests (unfinished article):
http://submesa.com/data/raid/geom_stripe
http://submesa.com/data/raid/geom_stripe/page2

 

[Flapjack]

There is really no point in discussing further with you since you "know" absolutely that you are correct.

Do you have access to IEEE or ACM?

 

[vraa]

Your problem is the FAKEraid

Relax and have a cold beer

 

[Flapjack]

Your problem is the FAKEraid

Relax and have a cold beer

Ok. Assuming FakeRAID is the issue, what is a decent ([H] recommended) PCI-E RAID controller with eSATA support? Also, I'm going to go out on a limb here and assume I'll have to move all my data back because I'll have to rebuild the array... correct?

 

[Flapjack]

Man... I did some quick searching on NewEgg and didn't find anything spectacular. I could use a good, affordable recommendation.

I asked the company and they pointed me towards one of their products. It's more affordable than the ones I've been finding, not sure if I'm looking in the right places, though. They say it's hardware-based and much faster than the Sii 3132:

http://store.sansdigital-shop.com/ha4pcpomurac.html