Hi guys,
I've been trying to figure out if my CPU really is my bottleneck here... hopefully someone with some experience can weigh in or point out what I am missing.
I am setting up a new ZFS storage box.
Basic specs:
Supermicro 16x 2.5" bay chassis - CSE-213A-R740WB
Memory - 64gb
CPU - 1x E5-2603 V2
HBA - 2x AOC-S2308L-L8i (9207-8i equivalent)
Drives - 7x 840 Pro 512GB - intended to be used as a 6disk RaidZ2 with 1 spare.
OS - OmniOS (latest) with Napp-It on top
I have everything installed and running and I created my RaidZ2 array with 6 drives and I was doing a bunch of benchmarks but I was only seeing just over 500mB/sec write... I started doing a bunch of tests with different pool configurations and I seem to be hitting some kind of limit on writing as soon as the drives are striped. I am using Bonnie++ for my benchmarking. (NOTE: I have the drives separated across the HBA's as well, 4 on one, 3 on the other)
This is what I get for a single drive:
Bonnie command used for large pools:
Bonnie command used when testing multiple in parallel:
(The -r 10000 is just to force Bonnie to run a 50GB test even though the system had 64gb of memory - because I was running 3-7 at the same time)
Basically - 500mB/sec write, and 520mB/sec read. Great!
But as soon as I stripe two drives together (not RaidZ, just stripe - so Raid0) I get:
So the read doubled, but the write only goes up to 673mB/sec.
Then I tried the same for 3x in a stripe:
So read tripled, but write is still at that same place.. As the stripe increases the read also gets capped around 1.6gB/sec:
I did some more testing with multiple single-drive zpools, running Bonnie++ in parallel across them. When I do this - I see what I'd expect for performance:
For 6 drives at once:
Write: 1588
Read: 2768
For 7:
Write: 1576
Read: 2874
So everything in the system seems to be able to handle the speed - but something to do with striping limits it..
I (maybe stupidly?) figured that any modern Xeon is going to be able to handle something like being able to stripe @ 1gB/sec.. but this is the cheapest E5 V2 cpu and only clocks @ 1.8GHz.
So - does anyone have any suggestions on other things I could test to try and work out if it really is the CPU or not?
Thanks
I've been trying to figure out if my CPU really is my bottleneck here... hopefully someone with some experience can weigh in or point out what I am missing.
I am setting up a new ZFS storage box.
Basic specs:
Supermicro 16x 2.5" bay chassis - CSE-213A-R740WB
Memory - 64gb
CPU - 1x E5-2603 V2
HBA - 2x AOC-S2308L-L8i (9207-8i equivalent)
Drives - 7x 840 Pro 512GB - intended to be used as a 6disk RaidZ2 with 1 spare.
OS - OmniOS (latest) with Napp-It on top
I have everything installed and running and I created my RaidZ2 array with 6 drives and I was doing a bunch of benchmarks but I was only seeing just over 500mB/sec write... I started doing a bunch of tests with different pool configurations and I seem to be hitting some kind of limit on writing as soon as the drives are striped. I am using Bonnie++ for my benchmarking. (NOTE: I have the drives separated across the HBA's as well, 4 on one, 3 on the other)
This is what I get for a single drive:
Bonnie command used for large pools:
Code:
bonnie++ -d /Testpool/bm -u root -f -s 225000
Code:
bonnie++ -d /Testpool/bm -u root -f -s 50000 -r 10000
Code:
Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
2015.01.05 225000M [B][COLOR="red"]500686[/COLOR][/B] 76 163859 38 [B][COLOR="red"]520021 [/COLOR][/B]37 12283 35
But as soon as I stripe two drives together (not RaidZ, just stripe - so Raid0) I get:
Code:
Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
2015.01.06 100000M [B][COLOR="Red"]673257[/COLOR][/B] 99 336536 77 [B][COLOR="red"]998914 [/COLOR][/B]63 15904 44
So the read doubled, but the write only goes up to 673mB/sec.
Then I tried the same for 3x in a stripe:
Code:
Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
2015.01.06 100000M [B][COLOR="red"]675508 [/COLOR][/B]98 469291 93 [B][COLOR="red"]1515632 [/COLOR][/B]86 +++++ +++
So read tripled, but write is still at that same place.. As the stripe increases the read also gets capped around 1.6gB/sec:
Code:
Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
4X Stripe 100000M 666164 96 464853 93 1614523 91 +++++ +++
5X Stripe 100000M 664409 96 467964 93 1609386 92 +++++ +++
6X Stripe 100000M 660486 95 465797 92 1600067 91 +++++ +++
I did some more testing with multiple single-drive zpools, running Bonnie++ in parallel across them. When I do this - I see what I'd expect for performance:
For 6 drives at once:
Code:
Drive1 100000M 264298 43 158171 35 463943 33 11236 23
Drive2 100000M 267198 44 156709 35 465954 33 11554 24
Drive3 100000M 264293 43 156424 35 464482 33 10407 19
Drive4 100000M 264858 43 157579 35 458790 32 11728 23
Drive5 100000M 263174 43 156470 35 458161 32 10533 20
Drive6 100000M 266659 43 156296 35 460471 32 11214 23
Write: 1588
Read: 2768
For 7:
Code:
Drive1 100000M 223623 36 148292 33 407406 29 10308 22
Drive2 100000M 222744 36 149911 34 414207 29 12017 24
Drive3 100000M 224019 36 148816 33 416678 29 12855 25
Drive4 100000M 229917 37 146537 33 411769 29 11229 22
Drive5 100000M 231382 38 149004 34 408070 29 10735 23
Drive6 100000M 221885 36 149957 34 413474 29 11656 23
Drive7 100000M 226960 37 149537 34 405300 28 12302 25
Write: 1576
Read: 2874
So everything in the system seems to be able to handle the speed - but something to do with striping limits it..
I (maybe stupidly?) figured that any modern Xeon is going to be able to handle something like being able to stripe @ 1gB/sec.. but this is the cheapest E5 V2 cpu and only clocks @ 1.8GHz.
So - does anyone have any suggestions on other things I could test to try and work out if it really is the CPU or not?
Thanks
Last edited: