ZFS Performance Numbers

[Tau]

Just wanted to start a thread to publish some performance numbers here with a new test setup that I have been playign with and get some input.

Config:
Q6600
P5Q-E
8GB RAM
AOC-USAS-L8i
HP Expander
750Watt PC&P
14x 1TB 7200.12's

OpenSolaris 2009.06 & latest SVN (134 i think)

These results are on 2009.06 with the 14 drives configured in different ways using dd for some simple write tests to compare speeds.

14 drive Raid10 - 316MB/s Write
7 drive Raidz2 - 318MB/s Write
14 drive Raid60 (two 7 disk Raidz2) - 376MB/s Write
13 Drive Stripe - 382MB/s Write

the 7 drive raidz2 writes look about right to me as it is close ot the same as the speeds i have seen on my other test box (330MB/s write to a 7 disk raidz2) though the scaling on the other tests (14 drive raid60, and the stripe) do not seem even remotly proper to me.

Based on 50MB/s write per disk the stripe should be pulling ~700MB/s

I must be hitting a bottleneck of a wall here somewere and would like to track it down.


Watching the box access LEDS during a long write is interesting as well... the data access/write is in little patches, drive access for 2-3 seconds, 1 second no acces, 2-3 seconds of access, etc. Now one of the drives is accessing wile the others are not wich is odd.... iostat doesent report any odd writes on that drive either... wich is kinda odd...

Curious if anyone can help shed some light on these puzzling numbers... I'm going to see if I can get filebench installed and do some testing with it.

 

[sub.mesa]

Hm that is indeed curious. Could you begin by checking the interface bandwidth? You can do this by starting a dd thread for each physical disk, and then looking at disk bandwidth via some monitor program (i use gstat on FreeBSD; opensolaris should have something similar).

The dd-commands would look like:
dd if=/dev/da0 of=/dev/null bs=1m count=5000 &
dd if=/dev/da1 of=/dev/null bs=1m count=5000 &
dd if=/dev/da2 of=/dev/null bs=1m count=5000 &
dd if=/dev/da3 of=/dev/null bs=1m count=5000 &
...
The trailing & will make it a thread. That means the dd commands will start at the same time and read from all disks at the same time. So you should be seeing at least 70MB/s per disk. If it doesn't, you have hardware bottlenecks somewhere.

For example, can you omit the usage of your expander, and use all onboard SATA ports and the ones on your Supermicro controller? If you make a RAID0 of them, how is your write speed then?

Also, can you set the sysctl option "vfs.zfs.cache_flush_disable" to 1? It may be called differently on OpenSolaris. This would lead to different behavior when writing, though it would be dangerous to use it could identify whether the LED access pattern you saw is because of ZIL journaling.

 

[Tau]

Hm that is indeed curious. Could you begin by checking the interface bandwidth? You can do this by starting a dd thread for each physical disk, and then looking at disk bandwidth via some monitor program (i use gstat on FreeBSD; opensolaris should have something similar).

The dd-commands would look like:
dd if=/dev/da0 of=/dev/null bs=1m count=5000 &
dd if=/dev/da1 of=/dev/null bs=1m count=5000 &
dd if=/dev/da2 of=/dev/null bs=1m count=5000 &
dd if=/dev/da3 of=/dev/null bs=1m count=5000 &
...
The trailing & will make it a thread. That means the dd commands will start at the same time and read from all disks at the same time. So you should be seeing at least 70MB/s per disk. If it doesn't, you have hardware bottlenecks somewhere.

For example, can you omit the usage of your expander, and use all onboard SATA ports and the ones on your Supermicro controller? If you make a RAID0 of them, how is your write speed then?

Also, can you set the sysctl option "vfs.zfs.cache_flush_disable" to 1? It may be called differently on OpenSolaris. This would lead to different behavior when writing, though it would be dangerous to use it could identify whether the LED access pattern you saw is because of ZIL journaling.

Will try that here in a couple minutes, I just tested the 14 disk stripe @ 335MB/s, destroyed it, created a new one with 13 drives (left out the disk in question) dropped it to 312MB/s write. Wich is in line with ~24MB/s per disk performance that the 14 disk stripe shows... So i guess the disk is fine... even though according to the access LED its accessing weird... lol

I was assuming that the access I was seeing was due to the ZIL flushes, iostat (im pretty sure the equiv to your gstat) also showed the drives writing at ~25MB/s then dropping to 8, then back up to 24, etc.

I will try the DD thread thing here in a moment and post back results.

 

[Tau]

Got around to doing some more indept testing using Filebench for single stream read/write of 10/20GB files (just after some raw numbers here to make sure the disks are performing properly) And I am a bit mistifyed at my results... Hoping someone has some info to lend, or gently prod me in the right direction...

7 drive Raidz2 - 10Gb Read

IO Summary:      34297 ops, 566.0 ops/s, (566/0 r/w) 565.9mb/s,   2256us cpu/op,   1.7ms latency
  732: 620.620: Shutting down processes

20GB Write

IO Summary:      15246 ops, 245.3 ops/s, (0/245 r/w) 245.3mb/s,   5445us cpu/op,   4.1ms latency
  761: 104.731: Shutting down processes

Added another 7 drive Raidz2 to the same pool for basically a Raid60. (Logic would dictate that I/O performance should double, since you are doubling the total spindle count)

2x 7 drive Raid60 - 10GB Read

IO Summary:      40600 ops, 670.0 ops/s, (670/0 r/w) 669.9mb/s,   3124us cpu/op,   1.5ms latency

20GB Write

IO Summary:      19860 ops, 308.0 ops/s, (0/308 r/w) 307.9mb/s,   5709us cpu/op,   3.2ms latency

IOStat showes ALL drives being written to, and read from.

After seeing these results I blew the pool away and setup a 14 drive stripe, results as follow;

14 drive strip 10GB read

IO Summary:      48335 ops, 797.6 ops/s, (798/0 r/w) 797.5mb/s,   1239us cpu/op,   1.2ms latency
 1176: 133.002: Shutting down processes

14 drive stripe 20GB write

IO Summary:      26573 ops, 432.2 ops/s, (0/432 r/w) 432.2mb/s,   2988us cpu/op,   2.3ms latency
 1183: 81.070: Shutting down processes

The read speeds are correct since the HP Expander is hardware limited to 800MB/s so I will settle for 797. But the write speeds are retardedly low... I mean each of these drives are capable of ATLEAST 50MB/s (LOW estimate) so with 14 it should have maxed the expander again... but limped along at 432MB/s.

I did some testing with disabling ZIL completly (DO NOT DO THIS ON A SERVER WITH DATA ON IT!!!!) and noticed no improvement... wich also confused me.

With a huge 14 drive stripe maxing the expander @ 800MB/s for reads, and writes should be a no brainer.... I cant seem to figure out were the bottleneck is.

I will be doing some more testing tommorow and monitoring CPU/Kernal usage to see if something is leeching CPU clocks... but other than that I am at a complete loss...

If anyone is interested I did some more read/write testing with some other configurations (14 drive raid10, 4x3 raidz2, 3x4 raidz2, 2x6 raidz2, 1x12 raidz2, etc.) I can post the results if someone is interested... But they showed the same kinda numbers as the setups listed.

I am kinda fryed troubleshooting to many issues this week already >.<

 

[sub.mesa]

Howmany disks can you connect without using that expander, but using only 'normal' and full-bandwidth Serial ATA - on your motherbaord and any controllers that has excessive bandwidth.

Back a long time where i had one or two disks on PCI controller in a large array; it really lowered my results, even though the disks only did 13MB/s x 2 = 26MB/s. So the array performance of i guess ~150MB/s was bottlenecked by PCI even though the bandwidth of PCI wasn't reached. After putting the two disks on a PCIe controller; the array performance doubled (~350MB/s). This was all with old disks and hardware.

My point is: you should try to isolate any performance issue to a specific setup; for example if not using your HP Expander leads to much higher results, it could be it is still bottlenecking something even though it is capable of 800MB/s. If it increases latency i can imagine it would lower performance beyond that 800MB/s.

 

[Tau]

Howmany disks can you connect without using that expander, but using only 'normal' and full-bandwidth Serial ATA - on your motherbaord and any controllers that has excessive bandwidth.

Back a long time where i had one or two disks on PCI controller in a large array; it really lowered my results, even though the disks only did 13MB/s x 2 = 26MB/s. So the array performance of i guess ~150MB/s was bottlenecked by PCI even though the bandwidth of PCI wasn't reached. After putting the two disks on a PCIe controller; the array performance doubled (~350MB/s). This was all with old disks and hardware.

My point is: you should try to isolate any performance issue to a specific setup; for example if not using your HP Expander leads to much higher results, it could be it is still bottlenecking something even though it is capable of 800MB/s. If it increases latency i can imagine it would lower performance beyond that 800MB/s.

Hmmm good idea there, I have 8 ports on the HBA available (I was using them prior to the expander and getting 330MB/s writes with DD to a 7 disk Raidz2) I think the motherboard has 4 or 5 available as well. I will swap some disks over when I get home and see what the speed diffrence is.

 

[Krobar]

Hmmm good idea there, I have 8 ports on the HBA available (I was using them prior to the expander and getting 330MB/s writes with DD to a 7 disk Raidz2) I think the motherboard has 4 or 5 available as well. I will swap some disks over when I get home and see what the speed diffrence is.

Did you manage this? (I'm not using ZFS but this is an interesting thread)

 

[Tau]

I will be tonight, Got called out to a job the other day and dident have time. I will be swapping 8 drives to just the HBA, and another 2-4 onto the motherboard and will do a couple tests on that. to see whats going on.

 

[Stanza33]

According to this

http://constantin.glez.de/blog/2010/01/home-server-raid-greed-and-why-mirroring-still-best

snip

Performance: We'll assume a random read usage pattern. The mirror will write at about the same performance as a single disk. So does the RAID-Z pool. But for reads, the mirror will be up to twice as fast, because data blocks can be fetched in parallel from both disks. The RAID-Z configuration needs to access all disks for every single block of data, so not much to gain here. There is a corner case where streaming large amounts of data (if you're lucky) can take advantage of all the disks in parallel, but this doesn't apply to the regular usage pattern of a home server.
The difference gets worse if you increase the number of disks: A 2 x 2 mirror will be roughly twice as fast for writes and 4 times as fast for reads, while a 2+1 RAID-Z is still stuck at the speed of a single disk for both writes and reads.
Does performance matter for a home server? Maybe, maybe not: Average performance for a single drive of this class is around 60-70 MB/s (and I've seen that on my current home server, too), but this is only about 50% of what Gigabit Ethernet offers, so you won't be able to run a backup at full network speed. But I agree that 60 MB/s may be enough for most use cases.

 

[Tau]

Well I have some odd numbers again... what a surprise


12 disk strip with no expander, HBA and 4 onboard.

10Gb Read

IO Summary:      23308 ops, 384.6 ops/s, (385/0 r/w) 384.6mb/s,   1206us cpu/op,   2.6ms latency

IO Summary:      30464 ops, 502.7 ops/s, (503/0 r/w) 502.7mb/s,   1236us cpu/op,   2.0ms latency

IO Summary:      42361 ops, 699.0 ops/s, (699/0 r/w) 698.9mb/s,   1237us cpu/op,   1.4ms latency

20Gb write

IO Summary:      22536 ops, 367.4 ops/s, (0/367 r/w) 367.4mb/s,   2964us cpu/op,   2.7ms latency

IO Summary:      23541 ops, 383.4 ops/s, (0/383 r/w) 383.4mb/s,   2933us cpu/op,   2.6ms latency

IO Summary:      22517 ops, 368.2 ops/s, (0/368 r/w) 368.2mb/s,   2899us cpu/op,   2.7ms latency

8 disk stripe HBA Only

10GB Read

IO Summary:      38464 ops, 634.7 ops/s, (635/0 r/w) 634.7mb/s,   1262us cpu/op,   1.6ms latency

IO Summary:      42610 ops, 703.1 ops/s, (703/0 r/w) 703.1mb/s,   1235us cpu/op,   1.4ms latency

IO Summary:      44781 ops, 739.0 ops/s, (739/0 r/w) 738.9mb/s,   1226us cpu/op,   1.3ms latency

20Gb write

IO Summary:      20135 ops, 328.1 ops/s, (0/328 r/w) 328.1mb/s,   4060us cpu/op,   2.9ms latency

IO Summary:      19452 ops, 318.6 ops/s, (0/319 r/w) 318.6mb/s,   4281us cpu/op,   3.0ms latency

IO Summary:      19449 ops, 317.9 ops/s, (0/318 r/w) 317.9mb/s,   3751us cpu/op,   3.0ms latency

The first read/wright were done one after the other. Then the second two reads (for both sets of tests) were done back to back after destroyign the test file.

When running the 12 disk test with the 4 extra drives running off the onboard ports I noticed some very odd LED activity... the 4 drives that were onboard would stop accessing every few seconds for a couple seconds... than about a second after the HBA controlled drives would stop... Im going to assume the onboard controller just sucks.

Looking at the 8 drives off just the HBA the read speeds are acceptable... ~80-90MB/s per drive.... wich is the numbers I was expecting.

The write speeds on the other hand have me baffled

I think im going to post a copy of this thread over on the opensolaris forums because I must be missing/forgetting something major here

 

[Tau]

According to this

http://constantin.glez.de/blog/2010/01/home-server-raid-greed-and-why-mirroring-still-best

snip

Performance: We'll assume a random read usage pattern. The mirror will write at about the same performance as a single disk. So does the RAID-Z pool. But for reads, the mirror will be up to twice as fast, because data blocks can be fetched in parallel from both disks. The RAID-Z configuration needs to access all disks for every single block of data, so not much to gain here. There is a corner case where streaming large amounts of data (if you're lucky) can take advantage of all the disks in parallel, but this doesn't apply to the regular usage pattern of a home server.
The difference gets worse if you increase the number of disks: A 2 x 2 mirror will be roughly twice as fast for writes and 4 times as fast for reads, while a 2+1 RAID-Z is still stuck at the speed of a single disk for both writes and reads.
Does performance matter for a home server? Maybe, maybe not: Average performance for a single drive of this class is around 60-70 MB/s (and I've seen that on my current home server, too), but this is only about 50% of what Gigabit Ethernet offers, so you won't be able to run a backup at full network speed. But I agree that 60 MB/s may be enough for most use cases.

Not sure were he is getting his facts but i get much better then single disk speeds for Raidz1/2... as per DD, and filebench. Both in the read and write departments. Even testing raid10 (mirrored stripes in zfs) the numbers are not as they should be.

^^ I have posted some more results.

 

[Stanza33]

Not sure were he is getting his facts but i get much better then single disk speeds for Raidz1/2... as per DD, and filebench. Both in the read and write departments. Even testing raid10 (mirrored stripes in zfs) the numbers are not as they should be.

^^ I have posted some more results.

Not got an IRQ clash happening have you?

ie system has both onboard and expander both assigned to same IRQ... so it has to issue commands to one IRQ assigned board...then pauses as it flips and writes to other controller on same IRQ ??

 

[Eschertias]

How did you get filebench to compile on opensolaris? Did you build from source, or use the old 1.0 pkg file? I'm having odd issues with make failing.

 

[sub.mesa]

Wouldn't be a bad idea to involve the OpenSolaris (or should i say: Oracle?) guys as they may have much more detailed insights into these performance issues.

One other question: did you check the 'top' output during benchmarking? What was the CPU usage, what was the interrupt usage, what was the average load? i.e. is there a CPU bottleneck or is the bottleneck 100% to disk?

Can you see the average latency per I/O request on OpenSolaris like you can with BSD (gstat) ? In that case compare the latency of your HBA card with the onboard SATA ports.

 

[Tau]

How did you get filebench to compile on opensolaris? Did you build from source, or use the old 1.0 pkg file? I'm having odd issues with make failing.

Mine was not compiling either for the longest time, then I fixed the bundled install of it and im using w/e version ships with 2009.06

Wouldn't be a bad idea to involve the OpenSolaris (or should i say: Oracle?) guys as they may have much more detailed insights into these performance issues.

One other question: did you check the 'top' output during benchmarking? What was the CPU usage, what was the interrupt usage, what was the average load? i.e. is there a CPU bottleneck or is the bottleneck 100% to disk?

Can you see the average latency per I/O request on OpenSolaris like you can with BSD (gstat) ? In that case compare the latency of your HBA card with the onboard SATA ports.

Yeah I am going to post a copy of the thread over there here over the next couple hours, I got to fed up with it last night and decided to go to bed before I did anymore testing, though I have the next couple days free so I would like to get it nailed down.

I will be watching CPU usages here tonight or tommorow when doing some more testing, I kind of have a feeling there is some kinda leak somewere thats dominating my Kernel time or something equally retarded.

I will also be putting Server 2008 R2 on the spare OS disk and setting up a software stripe in there to rule out the expander/HBA as the bottlneck... should tell me for sure if its hardware or Opensolaris being a PITA.

I will keep this thread updated

 

[Tau]

Got Server 2008 R2 (man i hate windows 7 based things so much >.<) loaded on there after some dicking around with it not liking the partitions >.<. updated it all and setup a 14 disk stripe to do a bit of testing on. Results are in the attached screenshot. The numbers seem to be in line with my OpenSolaris testing... So i guess the OS is not to blame, was just expecting some higher numbers from these drives I guess =/

I still need to determine why the performance doesent seem to scale when configured in raid60 in OpenSolaris now.

I think I may actually be hitting a performance wall on this SuperMicro controller....

 

[monstertruck]

Looked at the firmware on the SM controller?
Tried it in a different PCI-E slot?

 

[Tau]

Looked at the firmware on the SM controller?
Tried it in a different PCI-E slot?

I updated all the firmwares when I was first installing it, slot is operating correctly.


I think I cracked it! I should have put windows on my other disk ages ago i would have this issue solved last week >.> lol...

Did some individual disk testing under Server 2008 R2 and have some pretty crazy results.

All 14 disks hooked up to the expander in one massive stripe the results can be seen here;

Very dissapointing but on par with my OpenSolaris numbers.

After testing each disk individually I identifyed 2 problem drives, and 3 others that could potentially be causing the issues. Unhooked those 5 drives, setup a new 9 disk stripe and ran the same benchmarks again.... I GAINED performacne over the 14 disk array.... Bam problem identifyed and isolated.

Here is the pic of the numbers from the 9 disk stripel

You know after comparing my results as well as the individual disk results it is pretty surprising how much variance in performance there are on a disk to disk level... Something I may have to write a paper on

Of the 9 disks I left connected there were some that had very good statistics, and others that I deemed acceptable... So I may be able to glean a bit more performacne out of it yet If I was a bit more strict on the drives.

Now I just need to try and get these problem drives swapped even though they are being reported as 100% health... LOL

 

[Dangman]

How exactly did you determine that five of those drives were problematic?

 

[mikesm]

Are you mixing drives of various types? Not a good idea for RAID work.

 

[sub.mesa]

He said "14x 1TB 7200.12's" - but i assume he found some 'duds' or partially broken drives in the array, causing lower performance?

 

[mikesm]

Interesting. I guess things haven't gotten better in the 7200.12 series... After the 7200.11 fiascos, I have pretty much written off Seagate.

 

[Tau]

How exactly did you determine that five of those drives were problematic?

Primarily with HDTach, and some Atto, i charted all the outputs and compared values and based my decission on that.

Are you mixing drives of various types? Not a good idea for RAID work.

No they are 14 brand new 7200.12 1TB drives

He said "14x 1TB 7200.12's" - but i assume he found some 'duds' or partially broken drives in the array, causing lower performance?

Thats the odd part they all pass SMART and Seatools.... yet there is some pretty wild diffrences between them all. Thats what I am assuming here as well, the shoddy performance out of some of them must have been bringing my RAID totals down.

Interesting. I guess things haven't gotten better in the 7200.12 series... After the 7200.11 fiascos, I have pretty much written off Seagate.

I hear you with the 7200.11s.... we had over 60 of them go south wich was NOT a good thing >.<

For the most part we have been very happy with these .12's i dont really think anything is "wrong" with the drive in the tranditional sence of the word, its just some of these drives dont have very linear performance numbers.... Now the drives still check out with 100% health, and no issues as per Seatools, so they meet the spec, its just they are not doing it as well as others....

I need to talk to my RMA guy now and see what i can do about getting these "ok" drives swapped :S


I may end up playing with some WD's or Hitachis for the SAN box here as well to see what their consistency betwen drives on the consumer level is.

 

[Eschertias]

Good to know. I finally got filebench to work, my RAID6 array pushes a fairly nice 440MB/sec on 8 drives using the mediaserver preset. And god damn does it take forever to pre-allocate the files.