Understanding RAID 0 Seek Times

[mikeblas]

I've been very curious about some of the statements held on this forum as "common knowlege" about RAID0.

In the past, I've tried asking about the specific topics when the issues were raised in other threads. That hasn't met with success; no matter how obivous or intuitive the original poster thinks the "fact" in question is, I have always fallen short of receiving a detailed description of why that specific fact might be true.

I've also noticed that asking follow-up questions on other threads isn't well-received in the culture of the forum. I'm sorry if I've offended or inconvenienced anyone by doing that.

Hopefully, by starting my own thread, I can avoid the issue with follow-up questions and get some good feedback on my own questions.

In at least a few other threads, I've read that RAID 0 arrays are slower than single drives for transferring smaller files. Hopefully, I'll someday have time to test this for myself, but I'm happy to stipulate that it is true.

Meanwhile, I'd like to learn exactly why it is the case. If RAID 0 is slower at transferring smaller files, at what exact (or even approximate) file size does transferring with a RAID 0 system become as fast as a single drive? Faster than a single drive?

Why is this opinion given in relation to file size, and not I/O transfer size? It turns out that copying even a large file happens by reading 64 kilobyte blocks.

One answer I've received in the past is that seeks are actually slower for a RAID 0 array than for a single drive. Maybe that's true, but I'm interested in understanding why it would be true. Can not the drives involved in a RAID 0 array operate independently? If so, then they should both be able to concurrently seek. The overhead of issuing a seek command is very low; the time taken to execute the command is a different matter, entirely, of course.

 

[unhappy_mage]

I have a matching pair of scratch disks I can do some tests on (bonnie++). I'll run them 3 ways: raid 0, raid 1, and single. It'll take me a while to set it up and test it all, but I'll report back. Bonnie++ tests linear read/write/rewrite speeds, and then seperately a file create/stat/delete on a large number of files.

As to the whys and wherefores, I'm not the one to ask.

 

[mikeblas]

As to the whys and wherefores, I'm not the one to ask.

Thanks for offering to run the tests. The fact is, the reasons, models, and limits are far more interesting to me than any empirical data.

I've run my own tests -- once using my own code, in fact. And read the results from plenty of others. Some of them don't agree with the claim that seeks on RAID 0 arrays are slower than on individual drives.

Whether the tests do or don't, they don't help me understand why or when they're expected to be slower, and therefore I'll never know when they're faster.

What I first found amusing, then found frustrating, was interacting with epopel who insist that it's "obvious" that seeking in two drives "should" be faster than seeking a single drive. Despite carefully explaining how I would think it works, and therefore why I don't think it's obvious, I've never been presented with a specific explanation of how seeking in a RAID 0 array does work, why it's implemented in this suboptimal way, and why it's faster.

Since little explanation is avaialble, and some evidence exists to the contrary of the assertions, I've begun to doubt the assertions.

 

[mikeblas]

In rereading my post, I'm not sure there's an obvious question. Let me try to do it this way, too.

If I wrote the driver or firmware that implemened RAID 0, I'd know I was talking with more than one drive and I'd want to take advantage of that fact. If I implemented the code to read some data fomr a particular location on the RAID0 volume, I would implement it so that as many drives were doing as much work as possible concurrently.

There are three parameters for a read request: the length of the read, the location of the read, and the dstination for the read data. If I'm implementing code for a RAID 0 array, I also know the number of drives and the stripe size of the drives in the set.

My job is to translate the requests for a length of data staring at a logical location into multiple reqeusts for different lengths of partial data at different physical locations on the drives involved in the array.

For the sake of simplicitly, let's assume I have two drives and a fixed stripe size of 128 kilobytes.

Someone asks me to read data starting at an offset from the beginning of the drive of 48,000 kilobytes. The requester wants me to get 384 kilobytes of data.

I first divide 48,000 kilobytes by two. That leaves 24,000 kilobytes and no remainder.

I would issue a seek command to disk#0 to move its head to a location 24,000 kilobytes from the start of the disk. (Sure, I know drives are broken down into clusters, heads, and cylinders. But to keep the description simple, I'll just use a single linear offset.)

There is no need to wait for a response to that seek command: I'd let the seek run asynchronously. I'd turn around and ask Drive#1 to seek to that same address. Issuing these two commands takes very little time. I can now wait for a drive to respond. One drive will respond first, and it's probably Drive#0 since it got its command first and the assumption that the drives in the array are equal (and were left in nearly identical previous states) doesn't seem too invalid.

Once the drive responds, I'll ask it to start reading at that offset; I want 128 kilobytes from that spot. That data is read into the target memory my caller specified.

By now, the other drive should have responded. I'll ask it to read 128 kilobytes. And then return to the first drive and ask it to read 128 kilobytes.

That's it: 384 kilobytes read. It's possible that I do a little math to begin with to learn that I could read 256 kilobytes from the first drive in one continuous read, but store the second block of 128 kilobytes at a different destination address.

I might even issue the read commands asynchronously, too, and let the drive and the DMA controller work out who writes what when.

Why would this approach be any slower than requesting the data from a single drive? I don't do any math when asking a single drive; no division to start things off. (On the other hand, in reality, some division to compute from a logical cluster to a pyhsical drive address might be involved depending on the interface I present. In that case, the algorithm is exactly the same: it's just that different denominators are used in the division to adjust for the sizing).

A seek request is issued. Since there's only one drive, I must block on the seek request before I can issue a read request. The data is then read, and I'm done.

The RAID 0 read algorithm I propose is a little more complicated, but I fail to see how it is any slower. Particularly because the work for the two drives can be concurrently completed, I don't see any reason why servicing a request for RAID 0 would be any slower. There might be some small fixed overhead (that division, for example), but this is infantessimal compared to the amount of time the physical I/O takes.

And so my questions are simple. If "reading small files" is really slower on RAID 0, then something must be wrong about my understanding.

While I've left out plenty of details, is my assumption of how a RAID 0 read is serviced fundamentally incorrect? In what way?

Is there some reason the low-level requests to the individual drives can't be made concurrent?

Is my understanding of how a single drive read is serviced not correct? In what way?

For now, I think my understanding and assumption are both correct. And that's why I don't understand why seeks are felt to be slower on a RAID 0 array. If seeks aren't more expensive, then why is "reading small files" more expensive?

Is the problem actually that I don't udnerstand the application? That "reading small files" isn't slower becasue of seeking, and is actually slower because of some other reason?

Please enlighten me.

 

[neo86]

I've heard seek times are slower because both of the drives have to be synchronized. Waiting for both drive heads to reach the proper position adds extra time. I suppose it might be slower for writing/reading a lot of small files since the drive have to be continously re-sychronized? Unless the small files are continguous... Just my guess from what I've heard.

 

[unhappy_mage]

Mike,
Have you looked at the linux software raid driver? It interfaces with two block devices in the way you're suggesting, albeit at a higher level. It doesn't request seeks and data, just "get me this block". I'm not enough of a kernel hacker to figure out whether it's blocking on read requests or not, but perhaps you could figure that out. If you don't have a copy of the kernel source available, I could mail you the relevant file. It's only 539 lines, it can't be all that hard to figure out.

I think that to implement a raid device at as low a level as you're talking would require direct access to the disk's firmware. In any case, the linux software raid makes a good example of what I would consider to be a normally capable raid0 implementation. It should make a fairly good dissection. Please post if you figure out what it's doing, or pm me if you'd like a copy of the file.

 

[mikeblas]

I've heard seek times are slower because both of the drives have to be synchronized.

Synchronized with what? Eachother? How? Spindle rotation? I don't think that's been the case since very early in the SCSI-1 days, when the controllers didn't have enough memory to buffer or queue anything. At all.

Waiting for both drive heads to reach the proper position adds extra time.

Why must we wait for both drive heads? Why ask both to move at the same time? The operation only takes longer if there is no concurrency; if we must wait for one head to finish moving before we can even initate the move required of the other head.

Is it required that the seeks be serialized? I can't understand why it would be.

 

[mikeblas]

Mike,
Have you looked at the linux software raid driver?

Unfortunately, not. I was playing around with gentoo and realized I could start dumping drivers and RAID support code. But I couldn't get gentoo to work; getting work done took priority over being 1337, so I installed Windows instead.

Thus, the code slipped through my fingers. I haven't been able to fetter out another source -- I'm not a Linux guy, obviously. If you can PM it to me, I'd be thrilled. Can you ZIP something up? I'd hate to have to keep asking for another directory or one more file.

It is very nice of you to offer.

It doesn't request seeks and data, just "get me this block".

"Get me block n" is the same as "Seek to n", then "read one block".

I think that to implement a raid device at as low a level as you're talking would require direct access to the disk's firmware.

Not at all.

Say you were working with me to write a program with tremendous data storage requirements. For whatever reason, we had to stick with the OS we had, and couldn't fool around with hardware. (Maybe we're working for the government, or something, and they're stupid about requirements changing.)

In fact, you want to open, then randomly read from and write to a file. Your file is a huge database, 240 gigs in size. All we have on the target system, though, is two 120 gig drives. They're mounted as drive D: and drive E.

Well, I'd run off and write some functions: SetBigFilePonter(), ReadBigFile(), and WriteBigFile(). They'd work exactly as I described, over D:\BIGFILE1.DAT and E:\BIGFILE2.DAT.

How is that different than RAID, really? I'm up a couple-three levels at the file and file system level. But the abstraction is the same. And I think the implementation that I describe, breaking your request into multiple overlapped I/Os queued concurrently to each device, is precisely the fastest way to implement the solution.

Do you see a better way? DO you think that translation doesn't work at a lower level, where the controller reads and writes to particular cylinder/head/track locations instead of offsets into a file? Abstractly, an C/H/T is just some math over an offset, expressed as a single integer.

At the drive firmware level, there's code that's much more involved. Listening to the head on the servo track, for example, and processing low-level IDE, SATA, or SCSI commands. Reporitng errors, responding to SMART requests, and so on.

 

[ashmedai]

Continued from original thread (here)

I don't mean to be writing it off as overhead without explaining it, and I may be using inappropriate terms.

My assumption is that it is slightly faster retrieve data from a drive, instead of a drive nested within an array. The drive still seeks just as fast, but the request for the data takes just a little longer before it's filled than it would otherwise. If the file is large enough to be striped across both drives, it more than compensates for this. If the file is too small, it only exists on one drive and is not striped, and thus is served up just a little slower than a single drive that isn't inside an array would be able to.

Hopefully I adequately answered the small file vs. large file issue over how small files won't end up being striped?

 

[mikeblas]

Hopefully I adequately answered the small file vs. large file issue over how small files won't end up being striped?

Well, you got as far as saying that the small file issue is a problem because of overhead. For a small file, you get "overhead but no benefit".

If this is true, why wouldn't it hold for small transfers, too, even in larger files? I can read a 20 gigabyte file going 256 bytes at a time. Or, I can read a 20 gigabyte file by requesting 128 kilobytes at a time. If I'm only asking for 256 bytes, you're saying there's no overhead problem because I'm still working with a large file.

Your explanation is based on the assumption that there's overhead. I know there is, but I really don't think it overwhelms any part of the transfer because it is so small. And in the case where the transfer request is small, the controller or the code doing software RAID 0 knows that the request doesn't map to two drives and can avoid talking to the other drive, anyway.

Again, I'm sorry I'm being a jerk about requiring a very detailed explanation. But if I don't have deep details, I really don't gain understanding. I'm not going to say "oh, there must be overhead" and leave it at that. If there's overhead, I want to know exactly what it is, what its characteristics are, and when it does and doesn't manifest itself.

Say we were interviewing, and you asked me to write malloc(). I scribble all over your white board, and my code works. You start asking me how it works, or how to change it to fit some special case.

If I can't answer you, then you know I just coughed-up some memorized answer and I don't really know what I'm doing. (But I'm pretty good at memorizing a couple dozen lines of code!) That's like saying "there's overhead".

On the other hand, if I can answer you, then I can describe what the code does, explain what changes I'd make, talk about worst-case and best-case behaviour, and so on. I really do understand the code and what it does, and can apply it to whatever I'd like, situationally. That's the kind of answer I'm looking for! It's a high standard for a hard question, but I don't think anyone who believes in the "overhead" answer but can't explain it really knows what's going on, and they're "NO HIRE".

I appreciate your efforts (and unhappy_mage, and everyone else, too) who's tried to help me figure it out. Think of it this way: if you provide enough information to give me an answer and satiate my curiosity, then you'll be a hero for the whole board, because I'll stop pecking at these questions!

 

[mikeblas]

Meanwhile, I poked around searching, and I found a random site that seems to have some of the lower-level Linux code. I don't know a damned thing about Linux, so I have to be very inductive. Plus I can only statically analyse the code, which is a world away from putting a breakpoint on it and figuring out which functions call each other and when.

With those caveats in mind, I looked at raid0.c, I find a function called raid0_make_request(). This function seems to take a request (in the form of a mddev_t structure) and translate it into a physical request that gets queued to a device, in the form of a buffer_read structure.

The translation is trivial. If you examine the code, after error checking, there's just a couple of lines of calculation which figure out what memory size is used, which offset into which partition (a Linux device, really, right?) and sector is involved.

This code has some issues that I really wonder about: dereferencing->lots->of->pointers every time a request is made isn't good for processor cache and memory latency; untangling those data structures and making the values handier would help. But that's a miniscule optimization: it would save dozens of clock cycles, not even tenths of milliseconds.

The stipulation was that RAID 0 causes many tenths of milliseconds of overhead. And this overhead is constant; it has nothing to do with the size of the file being accessed... though it might be repeated often if the request size is small and the number of request goes up because the requester is driven by the file size divided by the request size.

For comparison, let's look at linear.c for comparison. This code apparently works on single-unit devices; e.g., a standalone drive. linear_make_request() does the same work of translating an mddev_t request into a low-level buffer_read request. There's still work to do here: translate from an offset into a mounted "device" (which us Windows guys call "partitions") to an offset into a physical device. There's no division, but again, integer division and bit shifting takes a few clock cycles ... not a few tenths of milliseconds.

Comparing linear_make_request() to raid0_make_request() doesn't make me think there's any big overhead. Maybe something's hiding because I can't find some of the referenced headers at that site. (raid0.h, for example, isn't availaable.) And it's certainly possible I don't understand some important Linux-specific issue.

But the "overhead" everyone talks about just isn't jumping out at me.

 

[ashmedai]

I just don't know enough depth on hard drive controller theory to get much more specific as to the what & where. Also the overhead you're trying to pin down is a very small thing...RAID 0 is functionally equal to a single drive for non-striped files, but splits transfer for striped files.

I think it's something like:

Request - Controller - HD - Controller - Result

vs.

Request - RAID logic - Controller - HD - RAID logic - Controller - Result

that causes the very small overhead as compared to a single drive. AFAIK it's so small that it's only going to show up in benchies and not as a noticible performance difference, and even then maybe not.

On the other hand I could be completely off base as this is the way I'm imagining it in my head and not something I'm intimately familiar with the design principles of.

 

[ashmedai]

If that doesn't help, color me stumped.

 

[mikeblas]

Also the overhead you're trying to pin down is a very small thing...RAID 0 is functionally equal to a single drive for non-striped files, but splits transfer for striped files.

If the overhead is so small, then how can it be big enough to cause a gating factor in contraindicating RAID 0 for a particular application?

I think it's something like:
Request - Controller - HD - Controller - Result

Sure. An application makes a request and it goes to a device driver. The device driver has the controller do something, and the controller pases the requst to the HD. The HD starts sending back results (was the seek successful?) and the bytes, and there you have it.

Request - RAID logic - Controller - HD - RAID logic - Controller - Result

that causes the very small overhead as compared to a single drive. AFAIK it's so small that it's only going to show up in benchies and not as a noticible performance difference, and even then maybe not.

I guess I would call that path correct, sure. You're only thinking of the case where there's a single drive attached to the RAID controller; I guess that's because drawing the case wehre there are two drives isn't easy with ASCII art.

My questions remain: if the overhead is so small, then why isn't RAID 0 any good for "smaller files"? And why does it only affect small files, and not small transfer requests?

If we can measure this overhead, can we come up with a mathematical model that helps answer the questions about when the overhead is overcome by the file size? (And why it isn't actually the transfer size that causes the overhead?)

Any why does the overhead exist anyway? Why can't it be parallelized with the work in the other othe drive?

 

[Dually]

The benefit of RAID is parallelism … the question is under what condition(s) will you see the benefit. That will vary based up the configuration of the Array [software/hardware (manufacturer’s implementation varies) … ATA/SCSI (command queuing) … stripe width (increasing the number of drives increases parallelism) … the stripe size], and upon the I/O requests (number and size of files) … so, there is no simple answer. The vast majority of my experience has been with SCSI Arrays … and those are configured for the intended I/O.

My workstation is currently running three 15k 18GB SCSI drives in a RAID 0 Array, on an Adaptec 3210S with 256MB of cache, for its OS drive. I haven’t noticed a negative impact from the “overhead” associated with a RAID 0 Array, but that may be due to the fact that I’m running hardware based SCSI arrays w/ caching … (yes you can label me a SCSI F@nboy ). “Seat of the pants” performance is the same as a single drive for a light load of small files. Parallelism “kicks in” when you increase the load (either by the number of files and/or the size of the file). Given an adequate load, a properly configured RAID 0 Array will “thump” a single drive … even if the load is comprised of small files. As others have stated, the question becomes whether or not the average enthusiast will impose enough of an I/O load to justify the additional cost associated with the Array …or will they see roughly the same performance as a single drive.

I have spare drives and could do some testing as well, but the results would be specific to the configruation of my array.

 

[mikeblas]

so, there is no simple answer.

Well, that depends on the question.

The board has no shortage of simple answers about RAID: "don't use it", "myth", "gimmick", "no performance gain", and so on.

Parallelism “kicks in” when you increase the load (either by the number of files and/or the size of the file).

What is the work load? Are you copying these files, opening them for a database, or what?

As others have stated, the question becomes whether or not the average enthusiast will impose enough of an I/O load to justify the additional cost associated with the Array …or will they see roughly the same performance as a single drive.

Then, certainly, RAID 0 != myth. Just like anything else, it has applications where it's useful and applications where it isn't.

I have spare drives and could do some testing as well, but the results would be specific to the configruation of my array.

I guess I'm still not making myself clear. Empirical analysis doesn't really answer my question. There are posts all over this forum about RAID not being good for smaller files, being bad for seeking, and so on.

I'm trying to understand why those assertions are so widely held. I think they're false, and I think the reason they're held to be true is that someone has inductively applied the result of some empirical test when it wasn't appropriate to do so: RAID 0 didn't help game level load times, and game levels are comprised of lots of small files, and accessing small files seem to require lots of seeks, and therefore it must be true that RAID 0 has lots of overhead for seeking.

It can be completely true that some RAID 0 configurations aren't helpful for loading game levels while still be patently false that RAID 0 isn't any slower for seeking, or falls that "RAID 0 isn't good for small files", whatever that actually turns out to mean.

 

[uzor]

**Disclaimer: I cannot even begin to intelligently discuss this at your level of detail, so I won't even try.**


I have a few thoughts after reading this thread.

1 - Perhaps current RAID 0 implementations function contrary to your intuition and reasoning because, for whatever reason, developers could not or have not implemented it in this fashion. Is there a chance that you, by examining it from the outside, have thought of a solution so obvious that it has eluded the current mindset (the whole fresh brain idea)?

2 - It is said that we learn best by doing. You obviously have enough skill and knowledge to take up this project. Why not attempt to write a RAID 0 driver/implementation the way you think it ought to be, and see what happens and how it compares?

 

[Lazn_Work]

Well, that depends on the question.

The board has no shortage of simple answers about RAID: "don't use it", "myth", "gimmick", "no performance gain", and so on.

This is because from Empirical evidence, it doesn't provide enough of a benifit/risk for the most common useages (games) that people are trying to put them too.

What is the work load? Are you copying these files, opening them for a database, or what?

Well exactly, for some things it helps, for others it just increases the risks (it basically never hurts performance)

Then, certainly, RAID 0 != myth. Just like anything else, it has applications where it's useful and applications where it isn't.

So the responce given when people ask about using it, is usually it is not worth it, because WAY over 90% of the time they are looking to improve their gaming performance (either level loading times, or game startup times) and this is one instance that we know from experience that it doesn't help. We don't know why, but just because our knowledge is emperical it does not change the measured facts.

I guess I'm still not making myself clear. Empirical analysis doesn't really answer my question. There are posts all over this forum about RAID not being good for smaller files, being bad for seeking, and so on.

Because this is what we see, from empirical evidence. If you want to know the why behind it, you are asking the wrong people, call the RAID chipset MFG's and the Drive MFG's, don't ask a Soccer Mom why her minivan has a certain Piston size.

I'm trying to understand why those assertions are so widely held. I think they're false, and I think the reason they're held to be true is that someone has inductively applied the result of some empirical test when it wasn't appropriate to do so: RAID 0 didn't help game level load times, and game levels are comprised of lots of small files, and accessing small files seem to require lots of seeks, and therefore it must be true that RAID 0 has lots of overhead for seeking.

It can be completely true that some RAID 0 configurations aren't helpful for loading game levels while still be patently false that RAID 0 isn't any slower for seeking, or falls that "RAID 0 isn't good for small files", whatever that actually turns out to mean.

It all comes down to what we have experienced in using it ourselfs:

1) it helps Video encoding and Photoshop editing.
2) it does not help Game loading, and everyday comptuer use.
3) The risks are higher than just MTBF of the drives, because often the RAID array can fail without a Drive failing, and the data is still just as gone.

Why? ask the engineers who make the product, not the end users.

==>Lazn

 

[mikeblas]

1 - Perhaps current RAID 0 implementations function contrary to your intuition and reasoning because, for whatever reason, developers could not or have not implemented it in this fashion. Is there a chance that you, by examining it from the outside, have thought of a solution so obvious that it has eluded the current mindset (the whole fresh brain idea)?

Of course it's possible that my plan doesn't work. I wonder about that many times, and I've written about it. What am I missing? Why can't it be implemented so both heads are seeking at the same time? After all, if they're seeking at the same time (when necessary to move both) then there's no overhead for seeks.

It doesn't seem likely that a faster way to do things has eluded many people who do this for a living, though it seems the appropriate approach to me.

And those things, taken together, are why I keep asking: "Show me the overhead!" If it's necessary, I want to understand why it is necessary. If it's really there, I want to understand its characteristics.

2 - It is said that we learn best by doing. You obviously have enough skill and knowledge to take up this project. Why not attempt to write a RAID 0 driver/implementation the way you think it ought to be, and see what happens and how it compares?

Time, mainly. I race cars, so I'm very interested in learning how engines work. But I'm far more interested in driving than in trying to build my own engine. Knowing something about engines helps my racing, though.

Similarly, I write software. While I might not have time to build my own RAID 0 array, It would help me to write better software if I deeply understood how RAID 0 works (and all the other levels, too).

I have written my own programs to exercise and time both single-drive and RAID0 volumes, and I don't find that there's a problem with seeking. But the assertion is still made here, quite often. I'm trying to understand the assertion better, but when I ask about it I usually get more heat than light. It's touted as "obvious" and "everyone knows" and so on.

And I have implemented software that uses multiple single drives for combined storage. When I measure the performance of those products, I'm not tripping over any additional overhead in seeking: the benefit I get from multiple spindles far outweighs any code I have to write to decide which spindle to access, even on the smallest transfers.

My experience with writing that kind of software is why the claim that RAID 0 hurts seek performance was very surprising to me.

 

[mikeblas]

Because this is what we see, from empirical evidence. If you want to know the why behind it, you are asking the wrong people, call the RAID chipset MFG's and the Drive MFG's, don't ask a Soccer Mom why her minivan has a certain Piston size.

If the soccer mom in question repeatedly boasts about having a hemi, as several "end users" here expressed disdain about the "obvious" poor performance of RAID 0, then I'd probably start with the soccer mom.

It turns out it can be enlightning to understand why an opinion is held, aside from knowing the fact of the matter.

It all comes down to what we have experienced in using it ourselfs:

1) it helps Video encoding and Photoshop editing.
2) it does not help Game loading, and everyday comptuer use.
3) The risks are higher than just MTBF of the drives, because often the RAID array can fail without a Drive failing, and the data is still just as gone.

Why? ask the engineers who make the product, not the end users.

==>Lazn

I think you're missing the point of my questioning. I thought I spelled it out quite well in my first note from today, but I'll try another run at it.

In spirit, I don't dispute your assertions, numbered one through three, though the pedant in me knows they're poorly written and overly vague.

Even sticking with good faith, I do question the conclusions that other people have made and posted about. They're sometimes based on the observations you list. Because of your assertion #2, for example, some assert that RAID 0 has poor performance while seeking. That conclusion is just too inductive for me.

And I do doubt the assertions I've read that "RAID0 is a gimmick" based because of #3. There are many applications where transient data exists and total loss of the data isn't total at all, since it's just been copied from some other location anyway.

Further, I wonder about the details of #1. Isn't video encoding CPU-bound? How big does my video file or my Photoshop image have to be before RAID 0 is useful? How much performance gain will I notice? Proportional to the file size, or flatly?

There is lots of misinformation on this subject, such as posts that say RAID 0 is only useful for gigabit network users. There's plenty of posts that seem more vaild. If I know the details, then I have a very easy time separating the wheat from the chaffe.

Among other things, I'd like to definitively learn if the seak time issue really exists or not. If it does, then how big is it? Why can't a more efficient implementation (when one seems so straightforward to me) be used ... or more widely used? What parameters influence that overhead, and how much? And so on -- until I have enough knowledge and insight to change the way I'd approach writing software against RAID arrays.

Using empirical evidence to decide how to set up your own drive system is not a bad approach. Using it to recommend products to others probably isn't such a bad idea, either. But using evidence collected from limited observations to try and solve problems in the general case, for many users and diverse subsystem implementations? That a poor gambit.

 

[Lazn_Work]

http://www.google.com/search?source...GGLD,GGLD:2003-40,GGLD:en&q=RAID+0+seek+times

"Seek times on RAID-0 are generally slower than for a single drive so for small
amounts of data, a single drive will be faster"

"After setting up two hard drives in raid 0 my average seek times went UP to 11ms from 8ms on a single drive"

"In testing with my Raptors, my seek times actually increased with a RAID0 setup"

and digging deeper:
"The disk is spinning. The "average access" (not seek time) is comprised of
the time taken to move the heads to the correct track (strictly the real seek
time) and the time taken for the disk to spin to a position such that the
sector that is required is under the heads. For a 10,000rpm disk (not the
fastest we can get nowadays but still quite fast) the time taken for a
complete revolution is 6ms. This means that the average time to find a
sector will be 3ms once the heads are on the correct cylinder.

If there is only one disk being used for a small write then the average time
will be 3ms. If there are four disks and their spindles are not synchronised
(I think that synchronised disks have not been manufactured for some time)
then the average time for the sector to be found on all disks will be greater
than 3ms, maybe close to 6ms."

This is just from a little bit of googling.

==>Lazn

 

[Lazn_Work]

Some more info I found on storagereview (more in depth than their first RAID0 is bad article)

http://faq.storagereview.com/SingleDriveVsRaid0
and
http://storagereview.com/articles/200406/20040625TCQ_1.html

==>Lazn

 

[mikeblas]

This is just from a little bit of googling.

And it's not anything I haven't read before. It's only the third one that isn't an observation; at least one of these three things you've decided to post is an explanation.

I've questioned that explanation in this thread, I think -- or at least elsewhere on this board. But, I'm happy to play along one more time in the hopes that I actually get an answer.

If there is only one disk being used for a small write then the average time
will be 3ms. If there are four disks and their spindles are not synchronised
(I think that synchronised disks have not been manufactured for some time)
then the average time for the sector to be found on all disks will be greater
than 3ms, maybe close to 6ms."

This logic isn't valid to me. If I have four disks, why can't I have four outstanding I/O requests? That way, I issue the commands (almost) all at the same time. Issuing the command to seek or write doesn't take any time at all: it's just a few OUT I/O operations, say, for the processor to ask the controller to do some work.

The controller sees the request and starts acting on it. Part of acting on it includes moving the heads, waiting for the disk to spin to the right spot, and then performing the write. The physical I/O operations across those four disks in order to fulfil one single logical I/O operation can be performed asynchronously and in parallel, can't they?

If not, why not? At a much higher level, in application code, I kick off asynchronous requests to multiple physical devices all the time. It works great, and is substantially faster than using synchronous operations, one per device, waiting around.

NB: the Windows COPY command, the act of copying files by dragging and dropping in the shell Explorer, and even some "fancy" tools like ROBOCOPY don't do asynchronous I/O. They don't do overlapped operations, either. They just depend on the CopyFile() or CopyFileEx() APIs, which end up not (as of my most recent investigation) doing any overlapped or multithreaded I/O.

Do you think games, while loading levels, are doing multithreaded, overlapped I/O so they scale to more available spindles? Maybe game writers are more concerned with their drawing and shading algorithms to pay attention to these simple optimizations.

But certainly someone implementing firmware for a RAID controller card would do so, right? Someone implementing a software RAID implementation would, wouldn't they?

And, sure, I might be off base: maybe it's not possible at that level. But if not, why not?

I don't see anything relevant to this conversation about on the first page you link to StorageReview.

The second link has been proffered in response to my queries at least three times. I'll shoot it down the same way this time. Here's the text:

Some may say that access time can be increased (worse) due to a specific read (or write) requiring both heads to seek to their portion of the data, creating the result that average access time will be slower. However, this only really applies when accessing large numbers of small files as larger files will benefit from the improved STR. Unless the configured stripe size is too small, most of the small reads will access only one drive.

The first sentence is great. No insight yet, but I can't disagree with it -- many people say that, particularly around here.

Accessing large numbers of small files does not necessarily involve a lot of seeking as most of the table lookups actually go against the system cache (if not against the same sectors already in memory from previously reading the directory and allocation tables). Iit's the size of the transfer, not the size of the file, that I'd regard with more importance.

This does suggest that "small file" means "a file smaller than the stripe size of the drive", which I suppose I can understand. OTOH, if you know you're working two drives, why wouldn't you make the operations concurrent and asychronous? Then, the overhead of the second drive is largely absorbed while you're working with the first drive.

Let's try it the other way around. Say both those articles are completely accurate. I seek (9ms), wait for the sector (3ms), read it, then seek again (9ms), and wait again (3ms). To overcome the extra seek, how much data do I have to read before I break even? If I'm reading at 50 megabytes/second, then I guess it's about 150,000 bytes, right? Is that the break-even point?

Does that number make sense? It means I have to read 292 sectors at 512 bytes per sector. Which seems like a lot; are there that many sectors on a single track in a modern drive? (Because of all the translation going on, and that I live at the application level, I just don't know.) Would I move to the next sector, or the next head, first?

I appreciate your taking the time to try and answer my questions, but these quotes and links aren't anything I haven't seen before and don't really deal with my follow-on questions. I'm sorry to drive so hard at it, but I really want to have a very deep understanding of what's really happening and all the articles I've seen fall short of scratching my itch. Oversimplifications and vague generalizations aren't gonig to cut it.

 

[Lazn_Work]

And once again you are asking Soccer Moms about what oil to put in your minivan's differential.

You want more in depth information than is easy to find, your best bet is to get into contact with the MFGs of drives and RAID chipsets to get your questions answered.

I mean I can find stuff like this: http://www.pcguide.com/ref/hdd/op/over.htm

But not info on how the RAID chipset sends requests when certain situations occur.

Edit: but from that, part of the problem might be that the HD it self doesn't know exactly where a sector is, see:
"6. When the heads are in the correct position, the controller activates the head specified in the correct read location. The head begins reading the track looking for the sector that was asked for. It waits for the disk to rotate the correct sector number under itself, and then reads the contents of the sector."

So how much control can you expect from a HD that doesn't exactly know where the data is. It can go right to the track, but then it is spinning, so it has to read the track untill the correct sector is found, throw away the useless data and send the correct data down the bus (be it IDE or SCSI or SATA)

==>Lazn

 

[mikeblas]

And once again you are asking Soccer Moms about what oil to put in your minivan's differential.

Then why are all the soccer moms 'round here so sure that seek time is a problem?

 

[Lazn_Work]

Then why are all the soccer moms 'round here so sure that seek time is a problem?

Because when you RAID 0 two drives, and run HD Tach (or other disk benchmarking tools) on them the reported seek time goes up, as seen in my quote from a google search: "After setting up two hard drives in raid 0 my average seek times went UP to 11ms from 8ms on a single drive"

See my edit above too, just some thinking on my part, but it might help explain some of the problems associated with HD's.

==>Lazn

 

[mikeblas]

Because when you RAID 0 two drives, and run HD Tach (or other disk benchmarking tools) on them the reported seek time goes up, as seen in my quote from a google search: "After setting up two hard drives in raid 0 my average seek times went UP to 11ms from 8ms on a single drive"

... which contradicts the StorageTech article you linked, where the two-drive RAID 0 system had an I/O service time about three-quarters of one percent slower than the single-drive system they tested. Service time includes both the seek time and the rotational latency, plus whatever overhead happened to be walking across the highway at that moment.

And it doesn't match the results I've meaured myself, either. Service time was faster for me with RAID 0 on small size-I/O requests after a seek.

See my edit above too, just some thinking on my part, but it might help explain some of the problems associated with HD's.

Sorry; which HD-associated problem are you trying to explain? The rotational latency issue?

 

[Lazn_Work]

... which contradicts the StorageTech article you linked, where the two-drive RAID 0 system had an I/O service time about three-quarters of one percent slower than the single-drive system they tested. Service time includes both the seek time and the rotational latency, plus whatever overhead happened to be walking across the highway at that moment.

And it doesn't match the results I've meaured myself, either. Service time was faster for me with RAID 0 on small size-I/O requests after a seek.

It's a crazy world we live in but, three quarters of one percent is not much of an improvement, and some people see an increase.. thus seek (really access) time is not improved by RAID 0 and sometimes seems to be made worse. So when doing operations when access time is more important than throughput, RAID 0 is not an improvement, and it increases the risks of data loss.

Sorry; which HD-associated problem are you trying to explain? The rotational latency issue?

No rather why things are more complicated than just sending a request and expecting a result in a known amount of time, thus making RAID firmwares a little more complicated to make than say writing a C program that opens a text window that pronounces "Hello World" But I am not a programmer (by choise, I hate getting into that kind of detail) so I could be wrong.

==>Lazn

 

[mikeblas]

But I am not a programmer (by choise, I hate getting into that kind of detail) so I could be wrong.

I see.

 

[Lazn_Work]

I see.

That is a difference between us right there, you are a programmer. Me, I did just barely enough of it to get my degree from college.. That was torture enough.

I loved the labs, and the soldering and building hardware that interfaced with the various buses in a computer, but programming was just too tedious for me. Staring at a screen, keeping track of variables, making calls to dll's yuck. Give me a soldering iron and some hardware or even a wrench and an engine first.

==>Lazn

 

[mikeblas]

That is a difference between us right there, you are a programmer. Me, I did just barely enough of it to get my degree from college.. That was torture enough.

I guess I'm a programmer first, yeah. It's paying the bills. I don't have a degree, I do as much work as I can on my own street and race cars, and I've done dozens of electronics projects.

If you've made non-trivial hardware that interfaces to USB or PCI busses, or done your own internal engine work, then you know there's lots of detail in everything -- not just software.

And so I'm not sure what your point is.

 

[Lazn_Work]

I guess I'm a programmer first, yeah. It's paying the bills. I don't have a degree, I do as much work as I can on my own street and race cars, and I've done dozens of electronics projects.

If you've made non-trivial hardware that interfaces to USB or PCI busses, or done your own internal engine work, then you know there's lots of detail in everything -- not just software.

And so I'm not sure what your point is.

No point just rambling, and I don't mind the detail of hardware for some reason, partly because the things are at hand and visable. But programming for whatever reason I find mind numbing. (heck I even enjoyed desigining stuff in OrCAD running traces and etc. though I have not done that since college either)

Back to the subject at hand. How does a IDE or RAID driver handle queued requests to multiple drives (or even to the same drive)? Can mutiple requests be put out on the bus without waiting for each one to complete first? I don't really know. I know you can set up DMA block transfers, but is that used for smaller data requests?

==>Lazn

 

[mikeblas]

Can mutiple requests be put out on the bus without waiting for each one to complete first?

Yes. Obviously, the results can't come back at the same time.

I know you can set up DMA block transfers, but is that used for smaller data requests?

Yes. If you request only a few bytes, the OS reads a whole sector into a private buffer. (Or, if the sector has already been cached, it just goes to the private buffer and gets the data it already has.)

Then, copies the few bytes you asked for into the buffer you've supplied.

 

[mikeblas]

A guy I know finally helped me understand this.

The problem is more accurately latency, not seek time. Specifically, rotational latency.

At the instant a request is issued, the drive seeks to the cylinder where the data it wants is found. The request then has to wait for the correct sector on that cylinder to be under the read/write head.

On a drive spinning at 7200 rpm, one rotation takes 1/120th of a second; about 8.3 ms. The drive might have to wait almost a full rotation for the right sector to show up. Even on a 10000 rpm drive, the rotation can take 6ms; on a 15000 rpm drive, 4ms.

While this wait can be made concurrent as more drives become invovled in the transfer, it's overwhelmingly large compared to the time it takes the data to be read, and also greater than the track-to-track seek time you read in drive specs.

Since drives in RAID 0 arrays aren't synchronized for spindle rotation, they won't be in the same rotational position at the same time (unless you're lucky on that particular boot). Doing a small transfer, then, invovles waiting for both drives to make their rotation.

It's the rotational time that adds up. It's actually actually different between drives, and is far more likely to be a substantial contributor to the request overhead than the seek time.

 

[Lazn_Work]

A guy I know finally helped me understand this.

The problem is more accurately latency, not seek time. Specifically, rotational latency.

At the instant a request is issued, the drive seeks to the cylinder where the data it wants is found. The request then has to wait for the correct sector on that cylinder to be under the read/write head.

On a drive spinning at 7200 rpm, one rotation takes 1/120th of a second; about 8.3 ms. The drive might have to wait almost a full rotation for the right sector to show up. Even on a 10000 rpm drive, the rotation can take 6ms; on a 15000 rpm drive, 4ms.

While this wait can be made concurrent as more drives become invovled in the transfer, it's overwhelmingly large compared to the time it takes the data to be read, and also greater than the track-to-track seek time you read in drive specs.

Since drives in RAID 0 arrays aren't synchronized for spindle rotation, they won't be in the same rotational position at the same time (unless you're lucky on that particular boot). Doing a small transfer, then, invovles waiting for both drives to make their rotation.

It's the rotational time that adds up. It's actually actually different between drives, and is far more likely to be a substantial contributor to the request overhead than the seek time.

Yay, an answer, and one that makes sence.

So what we have is (usually) is increased seek times, and increased throughput. So if what you are doing is affected more by throughput, RAID 0 is a good thing, and if it is more affected by seek time, RAID 0 is not as much of a benifit. Right?

==>Lazn

 

[DougLite]

As I sit here and think about this, there is probably an impact on buffer performance as well - when you are running RAID-0, there is an added arbitration and layer to requests that might be ordinarily serviced very quickly by the buffer on a SLED setup. Largely thinking out loud here, but I would be curious to learn how read ahead mechanisms, write caching, etc and RAID-0 work together.

 

[unhappy_mage]

Damn, that would explain it quite well. Nice find, thanks for posting it!

 

[mikeblas]

As I sit here and think about this, there is probably an impact on buffer performance as well - when you are running RAID-0, there is an added arbitration and layer to requests that might be ordinarily serviced very quickly by the buffer on a SLED setup. Largely thinking out loud here, but I would be curious to learn how read ahead mechanisms, write caching, etc and RAID-0 work together.

I'd speculate the opposite: that the buffer positively impacts RAID0 efficiency, not that RAID0 efficiency impacts buffer performance. Why do you think that RAID0 slows down on-drive cache hits? Do you think they're less likely?

I'm pretty frustrated with the "time to answer" on this one. because I really don't know where else to go with similar questions. Funny thing was, I was very close to getting it: I knew there was something that must have been missing from my model of how the drives would work together. I was aware of rotational latency, but never understood the magnitude of the time involved. After all, 10,000 revolutions per minute seems quite fast -- but 6ms is kind of crappy.

Lazn_Work's quote from some other site above is very close: but it stops short of doing the statistical analysis over the likely locations of the heads. I guess it's not tempting to do that analysis or think of the overhead cumulatively because we're all used to thinking strictly of seek time and not the times required to service the complete request.

 

[DougLite]

Why do I think that? Because any request larger than the stripe size, even if it is available in cache, must be retrieved from both drives. Also, if it is smaller than the stripe size, the controller still has to ask both drives for it, even if only one has it in the buffer. Additionally, the required data, even if it is smaller than the stripe size, may still be spanned across both drives. These scenarios, will probably take longer (not necessarily noticeably longer) on a RAID-0 setup, because of the additional solid state logic in the RAID BIOS that is involved.

Also, as for Lazn's comments about the spindles not necessarily being in sync, that makes sense - let's say you're rolling dice, and a roll of 3-4 (in the middle, or short rotational latency) is good while a 1 or 6 (an outlier, or high rotational latency) is bad - the more dice you throw, ie the more drives in your array, the more likely you are to get a 1 or 6 that tacks on ~3ms (presuming 10000RPM drives) to your read service time.

 

[mikeblas]

These scenarios, will probably take longer (not necessarily noticeably longer) on a RAID-0 setup, because of the additional solid state logic in the RAID BIOS that is involved.

I don't think there's a chance of it being noticably longer because of the firmware. The actual solid state logic response in nanoseconds. The firmware can respond in thousandths of milliseconds, and the logic can respond in nanoseconds. I still don't believe they're not adding much overhead to the equation.

If both drives have the requested sector in cache, the response is very fast. If only one drive does, then the response time is probably a little better than if neither drive does.

Thing is, when would the drive's cache actually be populated with a required sector? If you have a 64 kilobyte stripe and a 8 meg cache, the drive has 128 stripes ready to go, assuming the drive pre-fetches as much as it can... which is optomistic. I'd expect that it just reads as much as it can from the current cylinder.

There are now two caches, realizing twice the logical distance ahead of the original request.

Also, as for Lazn's comments about the spindles not necessarily being in sync, that makes sense - let's say you're rolling dice, and a roll of 3-4 (in the middle, or short rotational latency) is good while a 1 or 6

Right; the probability of a fast response from one drive of the set ends up being Gaussian.