Stripesize influence on random I/O performance

[sub.mesa]

Introduction
As i love to discover the theory behind how technology works, i often do benchmarks of my own rather than those available on the web, as i've found that many reviews are actually misleading or simply incorrect.

A recent discussion in a related thread about optimal stripesize has sparked my interests to run some RAID0 benchmarks again, especially regarding random I/O performance. For that i utilized a FreeBSD platform which i understand well with regard to benchmarking. The gstat utility also allows me to monitor the I/O activity in real time.


Test setup
As i had to run alot of benchmarks to make a meaningful article, i created a script that runs the benchmarks in consecutive order. The host system is a quad-core AMD with 8GB RAM, with the disks being connected to the chipset-powered SATA ports utilizing AHCI and also NCQ. All tests were run at least three times, which the average number taken.
I've added my own comments in the article, but i'd love to discuss the interpretation of the results. It appears to confirm my theories that a lower stripesize would hurt random I/O performance instead of help. Interestingly, a stripesize of 1 megabyte gave me higher results than 128KiB stripe. I would like to know why.


Benchmark results with nice graphs
Time to checkout the benchmark results yourself:
===> http://submesa.com/data/raid/geom_stripe <===


One-line conclusion
Sequential I/O prefers smaller stripesizes, random I/O prefers larger stripesizes.


Rationale about stripesizes
In RAID0, if you let all physical disks in the array work on ONE I/O request at a time, you would limit or completely disable the opportunity to process I/O requests in parallel - leading to much lower performance. Thus, a high stripesize is preferable to the FreeBSD RAID0-engine.

Instead of all disks working on ONE request, you would want them working on different requests simultaniously instead. So while disk1 is processing request A, disk2 is processing request B. When reading, you need a queue depth of at least the number of disks in the RAID0 to get a decent benefit from striping. When doing writes it is different, as buffering in the HDDs causes both disks being busy even with a queue depth of 1; essentially this is write-back performance. The buffer size is limited though, but extremely necessary to get good write performance. Without it, no harddrive can achieve 1MB/s speeds.

So the only pitfall of a properly configured RAID0, is that it requires multiple queued I/O's for read-heavy workloads. If your application only reads or writes in one I/O stream that is not sequential, it may not be enough to let both disks work at the same time and you may get close to single-disk performance. Examples of this are blocking I/O like application launch and booting. However, when we're talking database applications there will be lots of queued I/O's on a busy server; thus allowing the RAID engine saturate its disks.


Windows limitations
Windows-based engines employed in "FakeRAID" drivers may have "quick-and-dirty" optimizations to always read blocks as large as the stripesize; thus reading data which is not requested and though it may help remote synthetic sequential I/O bottlenecks it hurts random I/O performance by forcing the disks to do unnecessary I/O. For these RAID drivers, a lower stripesize is necessary to prevent extremely low random I/O performance.


How to get proper performance
So to make RAID0 improve random read/write IOps:

• you need a modern RAID0 engine
• you need proper alignment for both RAID-engine and filesystem
• you need a large stripesize to ensure each I/O is handled by one physical disk
• you need proper bandwidth so each disk is not waiting on the other like with Parallel ATA/PCI.
• you need multiple queued I/Os to get a decent random read performance boost on RAID0.
• (random) writes can be buffered by the HDDs themselves, already enjoying higher performance at queue depth=1.
• sequential transfers need buffering/read-ahead; don't test the raw RAID instead test on filesystem level that uses these optimizations.

Feedback
I'm open to discussion! I'm especially interested on feedback on my benchmark results. And any clues of why 1MiB stripesize continues to give higher performance? Just look at the 64-queue results and you'll see it is very close to the theoretical 100% performance increase! Extremely nice scaling.

I haven't tested with 4 disks yet - just 2 Seagate 320GB (2x160GB) disks. I can add more results later. If any want me to run any different benchmarks available under FreeBSD i can do that too.

 

[sub.mesa]

No one interested?

 

[john4200]

I guess you may be assuming that the blocks of random data you are reading and writing to the RAID are aligned with the stripes of the RAID. But I do not think that is the case. If you write a 128KB block to a random LBA on a 2 disk RAID 0 with a 128KB chunk size, the 128KB block will usually be split across both disks, since it is unlikely that it will start exactly at the beginning of a 128KB stripe on one disk.

That explains why chunk sizes bigger than 128KB increase your random IOPS. With a 1MB chunk size and 128KB block read, it is likely that the 128KB block will be entirely on one disk. So if you have several 128KB blocks queued up to read, the schedulers should be able to find 2 on different disks to read in parallel.

That also explains why you get a slight increase with chunk sizes over 1MB. There is still about a 12% probability that a random 128KB block will be split with 1MB chunks. That probability goes down with larger chunks.

 

[sub.mesa]

Yes i think you're absolutely correct in that. I will start playing with the sector size; if i can turn that 128KiB i should have an aligned test, with scores not increasing beyond 128KiB.

Still, it does confirm random I/O prefers larger stripesizes, rather than small stripesize and letting each I/O cover all RAID disks.