Project: ZFS Monster - Phase I (testbed)

Nice results =). I am curious what you get when you have all 40 something disks hooked up and what you get when your using a raidzX raid level. You thought about using raidz3 at all?

I was able to get around 900 MB/sec and 770 megabytes/sec write when I wasn't using raidz2 but I think I am being limited by the fact I am using only one SFF-8087 port to my drives. My controller is the LSI SAS2008 chipset. Instead of upgrading to snv134 have you tried booting a snv134 ISO?

I have heard from a coworker that even the 1068E based cards have some driver issues and he highly recommended I run snv134. You can download some snv134 images from:

http://www.genunix.org/

http://www.genunix.org/distributions/indiana/osol-dev-134-x86.iso
 
5 more on SASp2:

Code:
zpool add monster c9t50d0 c9t51d0 c9t52d0 c9t53d0 c9t54d0

Write:

definitely hit a brick wall on writes here:

zfs01:~# ./dd  bs=1000000 count=100000 if=/dev/zero of=/monster/100gb.
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 96.7475 s, 1.0 GB/s

Read:

Read is peaking as well:

zfs01:~# ./dd  bs=1000000 if=/monster/100gb. of=/dev/null
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 76.0751 s, 1.3 GB/s


last pid: 14142;  load avg:  1.44,  1.67,  1.20;  up 0+00:45:36                                                                                              04:23:50
54 processes: 52 sleeping, 2 on cpu
CPU states: 23.0% idle,  0.1% user, 76.9% kernel,  0.0% iowait,  0.0% swap
Kernel: 13744 ctxsw, 13 trap, 475089 intr, 2825 syscall
Memory: 8190M phys mem, 1178M free mem, 4095M total swap, 4095M free swap

Not too shabby...

Each of the sas ports should  *theoretically* be good for:

3Gbps * 4 = 12Gbps * 80% (20% overhead for encoding) = 9.6 Gbps / 8b/B = 1.2GB/s

* 2 (ports) = 2.4GB/s total

We are achieving 1.4GB/s / 2.4GBs = 58% of the theoretical.

The card is a 8x PCI-e..according to specs to means it PCI bandwidth is 8*.25GB/s = 2.5GB/s...just a tad over what we need, so that doesn't look like the bottleneck.

We still have some CPU left, so I'm assuming the card itself is the bottleneck.

I REALLY want to get the SAS2 card working...hope someone has some ideas on that.
 
Here's raidZ (5 drives) * 9:

Code:
zfs01:~# zpool status monster
  pool: monster
 state: ONLINE
 scrub: none requested
config:

 NAME         STATE     READ WRITE CKSUM
 monster      ONLINE       0     0     0
   raidz1     ONLINE       0     0     0
     c9t9d0   ONLINE       0     0     0
     c9t25d0  ONLINE       0     0     0
     c9t27d0  ONLINE       0     0     0
     c9t28d0  ONLINE       0     0     0
     c9t29d0  ONLINE       0     0     0
   raidz1     ONLINE       0     0     0
     c9t30d0  ONLINE       0     0     0
     c9t31d0  ONLINE       0     0     0
     c9t32d0  ONLINE       0     0     0
     c9t33d0  ONLINE       0     0     0
     c9t34d0  ONLINE       0     0     0
   raidz1     ONLINE       0     0     0
     c9t35d0  ONLINE       0     0     0
     c9t36d0  ONLINE       0     0     0
     c9t37d0  ONLINE       0     0     0
     c9t38d0  ONLINE       0     0     0
     c9t39d0  ONLINE       0     0     0
   raidz1     ONLINE       0     0     0
     c9t40d0  ONLINE       0     0     0
     c9t41d0  ONLINE       0     0     0
     c9t42d0  ONLINE       0     0     0
     c9t43d0  ONLINE       0     0     0
     c9t44d0  ONLINE       0     0     0
   raidz1     ONLINE       0     0     0
     c9t45d0  ONLINE       0     0     0
     c9t46d0  ONLINE       0     0     0
     c9t47d0  ONLINE       0     0     0
     c9t48d0  ONLINE       0     0     0
     c9t49d0  ONLINE       0     0     0
   raidz1     ONLINE       0     0     0
     c9t50d0  ONLINE       0     0     0
     c9t51d0  ONLINE       0     0     0
     c9t52d0  ONLINE       0     0     0
     c9t53d0  ONLINE       0     0     0
     c9t54d0  ONLINE       0     0     0
   raidz1     ONLINE       0     0     0
     c9t55d0  ONLINE       0     0     0
     c9t56d0  ONLINE       0     0     0
     c9t57d0  ONLINE       0     0     0
     c9t58d0  ONLINE       0     0     0
     c9t59d0  ONLINE       0     0     0
   raidz1     ONLINE       0     0     0
     c9t60d0  ONLINE       0     0     0
     c9t61d0  ONLINE       0     0     0
     c9t62d0  ONLINE       0     0     0
     c9t63d0  ONLINE       0     0     0
     c9t64d0  ONLINE       0     0     0
   raidz1     ONLINE       0     0     0
     c9t65d0  ONLINE       0     0     0
     c9t66d0  ONLINE       0     0     0
     c9t67d0  ONLINE       0     0     0
     c9t68d0  ONLINE       0     0     0
     c9t69d0  ONLINE       0     0     0

Write:

zfs01:~# ./dd  bs=1000000 count=100000 if=/dev/zero of=/monster/100gb.
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 174.106 s, 574 MB/s

Read:
zfs01:~# ./dd  bs=1000000 if=/monster/100gb. of=/dev/null
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 108.177 s, 924 MB/s


Running out of steam:

last pid: 15020;  load avg:  3.25,  1.15,  0.62;  up 0+01:15:41                                                                                              04:53:55
54 processes: 50 sleeping, 3 running, 1 on cpu
CPU states: 17.9% idle,  0.0% user, 82.1% kernel,  0.0% iowait,  0.0% swap
Kernel: 1515 ctxsw, 2 trap, 578178 intr, 1237 syscall
Memory: 8190M phys mem, 1146M free mem, 4095M total swap, 4095M free swap
 
Raidz2 7 drives * 5 (35 drives total)

I could do 42 drives as a couple of my Hitachi 2TB drives are at another location so I've got 5 1.5TB drives in the case as stand-ins (BTW: I can barely hear myself thing as there has been a jet engine next to me for the last 6 hours of testing).

Code:
zfs01:~# zpool status monster
  pool: monster
 state: ONLINE
 scrub: none requested
config:

 NAME         STATE     READ WRITE CKSUM
 monster      ONLINE       0     0     0
   raidz2     ONLINE       0     0     0
     c9t9d0   ONLINE       0     0     0
     c9t25d0  ONLINE       0     0     0
     c9t27d0  ONLINE       0     0     0
     c9t28d0  ONLINE       0     0     0
     c9t29d0  ONLINE       0     0     0
     c9t30d0  ONLINE       0     0     0
     c9t31d0  ONLINE       0     0     0
   raidz2     ONLINE       0     0     0
     c9t32d0  ONLINE       0     0     0
     c9t33d0  ONLINE       0     0     0
     c9t34d0  ONLINE       0     0     0
     c9t35d0  ONLINE       0     0     0
     c9t36d0  ONLINE       0     0     0
     c9t37d0  ONLINE       0     0     0
     c9t38d0  ONLINE       0     0     0
   raidz2     ONLINE       0     0     0
     c9t39d0  ONLINE       0     0     0
     c9t40d0  ONLINE       0     0     0
     c9t41d0  ONLINE       0     0     0
     c9t42d0  ONLINE       0     0     0
     c9t43d0  ONLINE       0     0     0
     c9t44d0  ONLINE       0     0     0
     c9t45d0  ONLINE       0     0     0
   raidz2     ONLINE       0     0     0
     c9t46d0  ONLINE       0     0     0
     c9t47d0  ONLINE       0     0     0
     c9t48d0  ONLINE       0     0     0
     c9t49d0  ONLINE       0     0     0
     c9t50d0  ONLINE       0     0     0
     c9t51d0  ONLINE       0     0     0
     c9t52d0  ONLINE       0     0     0
   raidz2     ONLINE       0     0     0
     c9t53d0  ONLINE       0     0     0
     c9t54d0  ONLINE       0     0     0
     c9t55d0  ONLINE       0     0     0
     c9t56d0  ONLINE       0     0     0
     c9t57d0  ONLINE       0     0     0
     c9t58d0  ONLINE       0     0     0
     c9t59d0  ONLINE       0     0     0

Write:

zfs01:~# ./dd  bs=1000000 count=100000 if=/dev/zero of=/monster/100gb.
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 171.766 s, 582 MB/s

Read:

zfs01:~# ./dd  bs=1000000 if=/monster/100gb. of=/dev/null
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 119.486 s, 837 MB/s
 
5 more on SASp2:
I REALLY want to get the SAS2 card working...hope someone has some ideas on that.

You could try the ISO I linked to. If your having problems with the nvidia driver maybe try some cheap video card just for testing? I am using the SAS2 driver (mpt_sas driver instead of mpt) and so far its working great for me. I think I am semi limiting myself either by the SAS expander, CPU, or miniSAS port. Because multipath is enabled by default on snv_134 the drive names are really weird (see my previous zpool output). I had to pull the drives one by one to figure out the order but if you have the serials dumped already you could use smartctl and do it by the serial.
 
(BTW: I can barely hear myself thing as there has been a jet engine next to me for the last 6 hours of testing).

I know how that goes with supermicro equipment. We have almost pure supermicro stuff at work.

I know my newest 2U machine that has a supermicro case actually let me control the fan speeds low enough that it wasn't that bad (Compared to previous models) but I am guessing that you can't control fan speed at all on those JBOD cases =(
 
OK..FINALLY got the LSI 9200-8e working by using your liveCD suggestion.

Here's all 45 drives in a strip:

Code:
Write: 

root@opensolaris:~# dd  bs=1000000 count=100000 if=/dev/zero of=/monster/100gb.
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 213.78 s, 468 MB/s

Read:

root@opensolaris:~# dd  bs=1000000 if=/monster/100gb. of=/dev/null
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 79.0292 s, 1.3 GB/s

Wierd...read is exactly the same, but write is actually WORSE.

CPU is WAY lower though:

Code:
load averages:  0.18,  0.33,  0.19;               up 0+00:33:43                                                                                                                        23:51:36
80 processes: 79 sleeping, 1 on cpu
CPU states: 78.3% idle,  0.1% user, 21.6% kernel,  0.0% iowait,  0.0% swap
Kernel: 19868 ctxsw, 17389 intr, 2855 syscall
Memory: 8190M phys mem, 6970M free mem

I'm going to retest stepping through adding 5 drives at a time like before.
 
Were those writes to parity or was it just adding a bunch of drives to a zpool creating a raid0 type of dealio? I was atleast able to see 700 MB/sec write speeds when not dealing with parity.
 
OK...figured it out...the Asus M4A79T mb I'm using does have 4 16x PCI-e...there's only one problem, you only have 26 lanes to share across the 4 PCI-e slots. By default, if you stick a 16x display adapter in slot 1, it assumes you'll also use slot3 for 16x and thus it drops slot 2 and 4 down to 1x (or 500MB/s for Pci-e 2.0).

I had the LSI 1068E in slot3 for my testing and the LSI 2008 in slot 4...which is why the 1068E (even at PCI-e 1.0 and 250MB/lane) was doing MUCH better than the SAS2 card.

Modified MB bios to force slots3 and 4 to get 8x each.

Code:
45 drives in one big stripe again, LSI 2008 controller


Write:

zfs01:~# ./dd  bs=1000000 count=100000 if=/dev/zero of=/monster/100gb.
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 79.2527 s, 1.3 GB/s

Still not what I'd expect to see, but at least 30 percent better than the 1068E.

zfs01:~# ./dd  bs=1000000  of=/dev/null if=/monster/100gb.
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 72.1374 s, 1.4 GB/s

Grrrr...at least it's back up to a reasonable level, but I'd think I should be getting about 2GB/s.

There seems to be something now bottlenecking this as 1.4GB/s...either the drives are negoitiating at Sata/150 instead of Sata/300, or the SAS link to the controller is negotiating 3Gbps SAS instead of 6Gbps SAS. ... or maybe that's a mis-understanding on my part...can the drives be talking Sata 3gbps and yet the aggregating link from the expander to the controller be at 6Gbps??
 
Have you ever climbed Mt. Hitachi...the highest peak on Hardforum?

mounthitachi.jpg


52 2TB drives...most Hitachi with a few Seagate and Samsungs thrown in for good measure.

You know what the remnants of a drunken weekend bender looks like...here's what a storage bender looks like:

storagebender.jpg
 
Last edited:
Storage controllers are in.

The controller is designed to cascade to one or more external SC847-JBOD (45 drives) enclosures. I decided to keep things clean, have a separate 2U SC-213 to run Solaris and separate 4U SC847s to hold drives.

The nice thing about the SC-213 is that it has it's own backplane with 16 SAS (unfortunately just SAS1) 2.5 inch drives. This provides a great place to place the OS drives and load up with SSDs for L2 cache and/or the ZIL.

We built the controller with the Supermicro X8DTH-6 motherboard which also has on-board SFF-8087 connectors...making it very easy to connect at least 8 ports on the backplane to the motherboard without tying up any of the PCI-e slots. Speaking of PCI-e, this beast has 7 (seven!) PCI-e 2.0 x8 slots.

FTM we're using just one PCI-e slot to connect this box to an external SC847 via LSI 9211-8e SAS2 HBA. For now I'm connecting the front and back SC847 backplanes using separate SAS2 ports...one to the front and one to the back. If I wanted to conserve ports I could connect the two back planes together inside the SC847 and then I'd only need one SAS HBA port. However, going with the two ports I *theoretically* have 12Gbps of bandwidth between the controller and the disk cases.

Each port on the LSI 9211-8e supports 256 drives...thus with just a single 9211 controller this configuration could support 512 * 2TB or about 1PB. With 6 additional PCI-e slots available, a single storage controller could scale just under 7PB.

Controller Front:
zfscontrollerfront.jpg


PCI-e 2.0 up the ...:

zfscontrollerpcie20time.jpg


Controller top view:
zfscontrollertop.jpg


Controller internal SAS backplane:
zfscontrollersasbackpla.jpg


Controller rear:
zfscontrollerrear.jpg
 
Here we go...

100TB:

Controller plus SC847*1 front:
singlefront.jpg


Controller plus SC847*1 rear:
singlerear.jpg


Oh...you need another 100TB...no problem mon:

Controller plus SC847*2 front:
doublefront.jpg


..and a party in the back...

Controller plus SC847*2 rear:
doublerear.jpg


Seriously, the cabling does not get any easier and cleaner than this!

Cabling closeup:
sascabling.jpg


Note: I only have 52 drives between the two cases, so this will give me 100TB with the ability to trivially expand to 200TB.

Installing Solaris this weekend so should have numbers by Monday...I'm hoping to pull 4000MB/s, though I'm afraid the SAS2 HBA may top out at half that....we'll see.
 
I will say that we are building a monster VM environment and plan to host cloned drive images on one of the two SC847 storage servers we'll have...with dedup of course. Our goal is 1000 VMs.

What is your transport link going to be between the zfs server and the vm servers?
 
What is your transport link going to be between the zfs server and the vm servers?

We'll start at GigE, but if need be we can easily add 10Gbe.

The VMs should not be doing a lot of IO so I think GigE should be fine.
 
Solari
Code:
# uname -sr
SunOS 5.10


# format
       0. c0t5000C500248921F6d0 <DEFAULT cyl 30398 alt 2 hd 255 sec 63>
       1. c0t5000CCA221CD7B2Fd0 <ATA-Hitachi HDS72202-A20N-1.82TB>
       2. c0t5000CCA221D6F680d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
       3. c0t5000CCA221D33A00d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
       4. c0t5000CCA221D40FB0d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
       5. c0t5000CCA221D86A2Dd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
       6. c0t5000CCA221D87AEEd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
       7. c0t5000CCA221D87B16d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
       8. c0t5000CCA221D419C9d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
       9. c0t5000CCA221D420A3d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      10. c0t5000CCA221D420E5d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      11. c0t5000CCA221D420EAd0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      12. c0t5000CCA221D420F9d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      13. c0t5000CCA221D420FBd0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      14. c0t5000CCA221D1050Fd0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      15. c0t5000CCA221D3121Cd0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      16. c0t5000CCA221D4212Ed0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      17. c0t5000CCA221D42042d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      18. c0t5000CCA221D42049d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      19. c0t5000CCA221D42054d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      20. c0t5000CCA221D42102d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      21. c0t5000CCA221D42116d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      22. c0t5000CCA221D87517d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
      23. c0t5000CCA221DA8BD0d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
      24. c0t5000CCA221DA9542d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      25. c0t5000CCA221DAB8EBd0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      26. c0t5000CCA221DABAA2d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      27. c0t5000CCA221DAD356d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      28. c0t5000CCA221DB27EDd0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      29. c0t5000CCA221DB083Cd0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      30. c0t5000CCA221DC3FA6d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      31. c0t5000CCA221DF7FBBd0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      32. c0t5000CCA221DF73D1d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      33. c0t5000CCA221DFA2EBd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
      34. c0t5000CCA221DFD2F8d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      35. c0t5000CCA221DFD37Ad0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      36. c0t5000CCA221DFD300d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      37. c0t5000CCA221DFD311d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      38. c0t5000CCA221DFD398d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      39. c0t5000CCA221DFD971d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      40. c0t5000CCA221DFD972d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      41. c0t5000CCA221DFDA8Fd0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      42. c0t5000CCA221DFDA97d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
      43. c0t50024E900368D105d0 <ATA-SAMSUNG HD203WI-0002-1.82TB>
      44. c0t50024E900368D163d0 <ATA-SAMSUNG HD203WI-0002-1.82TB>

# zfs create pool monster blah...
# zfs add monster blah blah blah blah etc

Then literally 30 seconds later, you have a 64TB array:
# df -h
Filesystem             size   used  avail capacity  Mounted on
monster                 64T    11G    64T     1%    /monster

I'm only added 36 drives and I have a few 1TB and 1.5TB in the mix...I'll get those out and rebuild tommorrow, but for now we get:

# zpool status
  pool: monster
 state: ONLINE
 scrub: none requested
config:

        NAME                     STATE     READ WRITE CKSUM
        monster                  ONLINE       0     0     0
          c0t5000CCA221CD7B2Fd0  ONLINE       0     0     0
          c0t5000CCA221D86A2Dd0  ONLINE       0     0     0
          c0t5000CCA221D87AEEd0  ONLINE       0     0     0
          c0t5000CCA221D87B16d0  ONLINE       0     0     0
          c0t5000CCA221D87517d0  ONLINE       0     0     0
          c0t5000CCA221DFA2EBd0  ONLINE       0     0     0
          c0t5000CCA221D6F680d0  ONLINE       0     0     0
          c0t5000CCA221D33A00d0  ONLINE       0     0     0
          c0t5000CCA221D40FB0d0  ONLINE       0     0     0
          c0t5000CCA221D419C9d0  ONLINE       0     0     0
          c0t5000CCA221D420A3d0  ONLINE       0     0     0
          c0t5000CCA221D420E5d0  ONLINE       0     0     0
          c0t5000CCA221D420EAd0  ONLINE       0     0     0
          c0t5000CCA221D420F9d0  ONLINE       0     0     0
          c0t5000CCA221D420FBd0  ONLINE       0     0     0
          c0t5000CCA221D1050Fd0  ONLINE       0     0     0
          c0t5000CCA221D3121Cd0  ONLINE       0     0     0
          c0t5000CCA221D4212Ed0  ONLINE       0     0     0
          c0t5000CCA221D42042d0  ONLINE       0     0     0
          c0t5000CCA221D42049d0  ONLINE       0     0     0
          c0t5000CCA221D42054d0  ONLINE       0     0     0
          c0t5000CCA221D42102d0  ONLINE       0     0     0
          c0t5000CCA221D42116d0  ONLINE       0     0     0
          c0t5000CCA221DA8BD0d0  ONLINE       0     0     0
          c0t5000CCA221DA9542d0  ONLINE       0     0     0
          c0t5000CCA221DAB8EBd0  ONLINE       0     0     0
          c0t5000CCA221DABAA2d0  ONLINE       0     0     0
          c0t5000CCA221DAD356d0  ONLINE       0     0     0
          c0t5000CCA221DB27EDd0  ONLINE       0     0     0
          c0t5000CCA221DB083Cd0  ONLINE       0     0     0
          c0t5000CCA221DC3FA6d0  ONLINE       0     0     0
          c0t5000CCA221DF7FBBd0  ONLINE       0     0     0
          c0t5000CCA221DF73D1d0  ONLINE       0     0     0
          c0t5000CCA221DFD2F8d0  ONLINE       0     0     0
          c0t5000CCA221DFD37Ad0  ONLINE       0     0     0
          c0t5000CCA221DFD300d0  ONLINE       0     0     0


I'm NOT going to trust dd for benchmarking, but it looks MUCH better than when I was testing with the desktop class ASUS MB:


WRITE:
# time dd if=/dev/zero of=/monster/10GB bs=1024000 count=10240
10240+0 records in
10240+0 records out

real        3.6
user        0.0
sys         3.2

2700MB/s

READ:

# time dd if=/monster/10GB of=/dev/null bs=1024000
10240+0 records in
10240+0 records out

real        1.7
user        0.0
sys         1.7

5882MB/s

Oh yeah...THATS what I'm talking about.
 
I know nothing of Solaris, but if you did that dd test under linux, the read would come partially or fully out of the buffer-cache (depending on how much RAM your system has).

Maybe try it again with 100GB? (unless you have a bunch of RAM, in which case you might want to go even larger)
 
Last edited:
I would be more worried about a scub speed than thruput speeds...

That many disks and that much space 100-200tb... and you are in for a few days scrub time.

Whilst a scrub is running, thern try a speed thruput test concurrently.

See what speeds you get and if that will support your celerons fine
 
Does the scrub speed scale with the number of disks? In theory it could stay almost constant due to the increased parallelism.
 
Doing full blow testing this weekend on this beast...trying to find tools that enable me to interrogate the SAS controllers to verify what speed all the SAS2 links are running at. I'm expecting the links between the LSI-9200-8e controllers to be running at 6Gbps per channel and all the drives to be at 3Gbps (sata).

Does this sound right?

Please tell me that the interconnects between the controller and the SC847 expander can run at 6Gbps even though all the drives in the case are SATA at 3Gbps?

Code:
# ./sas2ircu LIST
LSI Corporation SAS2 IR Configuration Utility.
Version 5.00.00.00 (2010.02.09) 
Copyright (c) 2009 LSI Corporation. All rights reserved. 


         Adapter      Vendor  Device                       SubSys  SubSys 
 Index    Type          ID      ID    Pci Address          Ven ID  Dev ID 
 -----  ------------  ------  ------  -----------------    ------  ------ 
   0     SAS2008     1000h    72h   00h:05h:00h:00h      1000h   0072h 

         Adapter      Vendor  Device                       SubSys  SubSys 
 Index    Type          ID      ID    Pci Address          Ven ID  Dev ID 
 -----  ------------  ------  ------  -----------------    ------  ------ 
   1     SAS2008     1000h    72h   00h:83h:00h:00h      1000h   30b0h 

         Adapter      Vendor  Device                       SubSys  SubSys 
 Index    Type          ID      ID    Pci Address          Ven ID  Dev ID 
 -----  ------------  ------  ------  -----------------    ------  ------ 
   2     SAS2008     1000h    72h   00h:84h:00h:00h      1000h   30b0h 
SAS2IRCU: Utility Completed Successfully.

SAS2IRCU: Utility Completed Successfully.
# ./sas2ircu 1 display
LSI Corporation SAS2 IR Configuration Utility.
Version 5.00.00.00 (2010.02.09) 
Copyright (c) 2009 LSI Corporation. All rights reserved. 

Read configuration has been initiated for controller 1
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS2008
  BIOS version                            : 48.46.48.53
  Firmware version                        : 2.00.50.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 125
  Concurrent commands supported           : 4015
  Slot                                    : 7
  Segment                                 : 0
  Bus                                     : 131
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Enclosure services device
  Enclosure #                             : 2
  Slot #                                  : 24
  State                                   : Standby (SBY)
  Manufacturer                            : LSI CORP
  Model Number                            : SAS2X36         
  Firmware Revision                       : 0414
  Serial No                               : x36554200
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Enclosure services device
  Enclosure #                             : 3
  Slot #                                  : 21
  State                                   : Standby (SBY)
  Manufacturer                            : LSI CORP
  Model Number                            : SAS2X36         
  Firmware Revision                       : 0414
  Serial No                               : x36554200
  Protocol                                : SAS
  Drive Type                              : SAS_HDD
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#                              : 1
  Logical ID                              : 500605b0:0201de50
  Numslots                                : 8
  StartSlot                               : 0
  Enclosure#                              : 2
  Logical ID                              : 50030480:007bb17f
  Numslots                                : 25
  StartSlot                               : 0
  Enclosure#                              : 3
  Logical ID                              : 50030480:007baf7f
  Numslots                                : 22
  StartSlot                               : 0
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.
 
Let's see what we can crank this up to.

Test config is as follows:

External 'controller' 2U supermicro server with (2) LSI 9200-8e HBAs (thus a total of 4 WIDE 6Gbps ports) or 24Gbps of total potential storage bandwidth. Each LSI 9200-8e is connected to an 8x PCI-e 2.0 slot. I then have TWO SC847 JBOD SAS2 storage cases (each with a possible of 21 drives in the rear and 24 drives in the front). Thus, I have the potential to connect 90! drives to this beast...however I only have about 60 Hitachi 2TBs so my testing will max out at that for the moment.

I have the SAS cabling setup as follows (to prefer max bandwidth):

SASp1 -> SC847 #1 - Front backplane
SASp2 -> SC847 #1 - Rear backplane
SASp3 -> SC847 #2 - Front backplane
SASp4 -> SC847 #2 - Rear backplane

For the initial test I have just 6 drives in a stripe connected to SC847#1/Front...I'm going to keep adding one drive at a time to the strip and re-running filebench to see when the hit a saturation point...this should enable me to infer what speed that SAS2 link is syncing to (as for the life of me I can't see how else I can tell as this is no status light nor anything in the Solaris logs).

Code:
# /opt/filebench/bin/go_filebench
FileBench Version 1.3.4
filebench> load multistreamread
 2098: 18.957: Multi Stream Read Version 2.0 personality successfully loaded
 2098: 18.957: Usage: set $dir=<dir>
 2098: 18.957:        set $filesize=<size>    defaults to 107374182400
 2098: 18.957:        set $nthreads=<value>   defaults to 1
 2098: 18.957:        set $iosize=<value> defaults to 1048576
 2098: 18.957:        set $directio=<bool> defaults to 0
 2098: 18.957:  
 2098: 18.957:        run runtime (e.g. run 60)
filebench> set $dir=/monster
filebench> run 60
 2098: 35.966: Creating/pre-allocating files and filesets
 2098: 35.966: File largefile4: mbytes=102400
 2098: 35.966: Re-using file largefile4.
 2098: 35.966: Creating file largefile4...
 2098: 73.201: Preallocated 1 of 1 of file largefile4 in 38 seconds
 2098: 73.201: File largefile3: mbytes=102400
 2098: 73.201: Re-using file largefile3.
 2098: 73.201: Creating file largefile3...
 2098: 110.444: Preallocated 1 of 1 of file largefile3 in 38 seconds
 2098: 110.444: File largefile2: mbytes=102400
 2098: 110.444: Re-using file largefile2.
 2098: 110.444: Creating file largefile2...
 2098: 147.843: Preallocated 1 of 1 of file largefile2 in 38 seconds
 2098: 147.843: File largefile1: mbytes=102400
 2098: 147.843: Re-using file largefile1.
 2098: 147.843: Creating file largefile1...
 2098: 184.938: Preallocated 1 of 1 of file largefile1 in 38 seconds
 2098: 184.938: waiting for fileset pre-allocation to finish
 2098: 184.939: Starting 1 seqread instances
 2099: 185.940: Starting 1 seqread4 threads
 2099: 185.941: Starting 1 seqread3 threads
 2099: 185.941: Starting 1 seqread2 threads
 2099: 185.941: Starting 1 seqread1 threads
 2098: 188.940: Running...
 2098: 248.942: Run took 60 seconds...
 2098: 248.943: Per-Operation Breakdown
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread4                  124ops/s 123.6mb/s      8.1ms/op      308us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread3                  136ops/s 135.8mb/s      7.3ms/op      319us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread2                  117ops/s 116.6mb/s      8.6ms/op      305us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread1                  111ops/s 110.8mb/s      9.0ms/op      306us/op-cpu

 2098: 248.943: 
IO Summary:      29216 ops 486.9 ops/s, (487/0 r/w) 486.8mb/s,   1148us cpu/op,   8.2ms latency
 
Let's see what we can crank this up to.

Test config is as follows:

External 'controller' 2U supermicro server with (2) LSI 9200-8e HBAs (thus a total of 4 WIDE 6Gbps ports) or 24Gbps of total potential storage bandwidth.

Shouldn't that be 2 * 8 * 6 or 4 * 4 * 6 = 96 Gbps (12 GB/s) of potential bandwidth less overheads. Though you are limited by the PCI interfaces to 8 * 2 * 5 = 80 Gbps (10 GB/s).
 
Shouldn't that be 2 * 8 * 6 or 4 * 4 * 6 = 96 Gbps (12 GB/s) of potential bandwidth less overheads. Though you are limited by the PCI interfaces to 8 * 2 * 5 = 80 Gbps (10 GB/s).

You're right, my math was way off (so much for the math degree)...as you said it should be: 4 wide ports * 4 ports each = 16 ports * 6Gbps/port = 96Gbps / 8b/B = 12GB/s

On the PCI-e (2.0) side, it should be 2 slots * 8 lanes * .5GBs/lane = 8GB/s

Let's look at it from a drive perspective. 60 drives * .1GBs/drive = 6GB/s

So it looks like the drives will ultimately be the limiting factor. Of course it's mostly a moot point anyway as most of the access to this thing will be across a .1GB/s Ethernet until I can afford to spring for some 10Gbe.
 
You're right, my math was way off (so much for the math degree)...as you said it should be: 4 wide ports * 4 ports each = 16 ports * 6Gbps/port = 96Gbps / 8b/B = 12GB/s

On the PCI-e (2.0) side, it should be 2 slots * 8 lanes * .5GBs/lane = 8GB/s

Let's look at it from a drive perspective. 60 drives * .1GBs/drive = 6GB/s

So it looks like the drives will ultimately be the limiting factor. Of course it's mostly a moot point anyway as most of the access to this thing will be across a .1GB/s Ethernet until I can afford to spring for some 10Gbe.

Doh, looks like I messed up my PCI-e calculation by using the gen 3 encoding efficiency vs the gen 2.

And yeah, it is kinda scary how low 10g has been to roll out. The actual NICs are starting to come done and get reasonable (supermicro duel port for <$500) but the switches are still insane. In the meantime, you could always grab a couple of these http://www.provantage.com/intel-e1g44htblk~7ITEN0FE.htm and stick them in.

Have the same issue at home, even a basic 4 drive raid 5 can easily saturate 1G ethernet.
 
Lsiutil will tell you the link speed of each link on the controller, if it detects the controller.
 
THANKS!

Now where can I find a version of it that works?

Code:
# ./lsiutil

LSI Logic MPT Configuration Utility, Version 1.62, January 14, 2009

0 MPT Ports found
 
Benchmarking is real tricky.

I went for a hardware setup for max bandwidth.

StorageMasterCtrl
|
-PCI-e#1
---SasWide#1
----Expander#1
------Drive#1
...
------Drive#12
-SasWide#2
-Expander#2
-Drive#13
...
-Drive#24
-PCI-e#2
-SasWide#3
-Expander#3
-Drive#25
...
-Drive#36
-SasWide#4
-Expander#4
-Drive#37
...
-Drive#48

When I hang only 6 drives off a SAS wide port I easily get 100MB/s per drive...it drops to half that with 12 drives per wide port...I should be able to get 4 ports * 6Gbps = 24Gbps / 8b/B = 3GB/s = 3000MB/s

Instead I'm only getting 500MB/s...not sure what's wrong.

Also, note that filebench reports throughput at 3x what is actually happening.

Code:
                            capacity     operations    bandwidth
pool                      used  avail   read  write   read  write
-----------------------  -----  -----  -----  -----  -----  -----
monster                   654G  86.4T      1  18.7K  88.2K  2.29G
  c0t5000CCA221D87B16d0  31.3G  1.78T      0    340  4.20K  41.7M
  c0t5000CCA221D419C9d0  31.4G  1.78T      0    339      0  41.8M
  c0t5000CCA221D420FBd0  31.4G  1.78T      0    345  25.2K  42.4M
  c0t5000CCA221D42054d0  31.4G  1.78T      0    349  4.20K  41.9M
  c0t5000CCA221D87517d0  31.2G  1.78T      0    343      0  41.5M
  c0t50024E900368D118d0  31.2G  1.78T      0    340      0  41.2M
  c0t5000CCA221D6F680d0  31.3G  1.78T      0    342      0  41.8M
  c0t5000CCA221D33A00d0  31.3G  1.78T      0    336      0  41.3M
  c0t5000CCA221D86A2Dd0  31.3G  1.78T      0    340      0  41.9M
  c0t5000CCA221D420E5d0  31.3G  1.78T      0    335      0  41.2M
  c0t5000CCA221D4212Ed0  31.3G  1.78T      0    339      0  41.8M
  c0t5000CCA221D42042d0  31.2G  1.78T      0    335      0  41.4M
  c0t5000CCA221D87AEEd0  7.87G  1.80T      0    425      0  51.7M
  c0t5000CCA221D420A3d0  7.78G  1.80T      0    420      0  51.1M
  c0t5000CCA221D420EAd0  7.77G  1.80T      0    424      0  51.8M
  c0t5000CCA221D420F9d0  7.77G  1.80T      0    417  4.20K  51.1M
  c0t5000CCA221D1050Fd0  7.73G  1.80T      0    415  4.20K  50.8M
  c0t5000CCA221D3121Cd0  7.73G  1.80T      0    414      0  50.8M
  c0t5000CCA221D42049d0  7.79G  1.80T      0    420      0  51.7M
  c0t5000CCA221D42102d0  7.76G  1.80T      0    420  4.20K  51.3M
  c0t5000CCA221D42116d0  7.76G  1.80T      0    417      0  51.2M
  c0t5000CCA221DA8BD0d0  7.79G  1.80T      0    415      0  51.3M
  c0t5000CCA221DB27EDd0  7.72G  1.80T      0    418  4.20K  51.1M
  c0t50024E900368D11Dd0  7.76G  1.80T      0    417      0  51.0M
  c0t5000CCA221D332FCd0  7.72G  1.80T      0    413      0  50.9M
  c0t5000CCA221D33315d0  7.81G  1.80T      0    419      0  51.3M
  c0t5000CCA221D40FB0d0  7.72G  1.80T      0    416      0  51.3M
  c0t5000CCA221DA9542d0  7.67G  1.81T      0    415      0  51.1M
  c0t5000CCA221DAB8EBd0  7.79G  1.80T      0    419      0  51.2M
  c0t5000CCA221DABAA2d0  7.73G  1.80T      0    413      0  50.7M
  c0t5000CCA221DAD356d0  7.74G  1.80T      0    418  4.20K  51.0M
  c0t5000CCA221DB083Cd0  7.76G  1.80T      0    415      0  51.0M
  c0t5000CCA221DC3FA6d0  7.77G  1.80T      0    420  8.40K  51.4M
  c0t5000CCA221DF73D1d0  7.78G  1.80T      0    417      0  51.3M
  c0t5000CCA221DF7FBBd0  7.78G  1.80T      0    419      0  51.8M
  c0t5000CCA221DFA2EBd0  7.70G  1.80T      0    418  4.20K  50.6M
  c0t5000CCA221DFD2F8d0  7.73G  1.80T      0    420      0  51.3M
  c0t5000CCA221DFD300d0  7.74G  1.80T      0    416      0  50.9M
  c0t5000CCA221DFD311d0  7.74G  1.80T      0    417      0  51.1M
  c0t5000CCA221DFD37Ad0  7.75G  1.80T      0    418  4.20K  51.0M
  c0t5000CCA221DFD398d0  7.73G  1.80T      0    418      0  51.3M
  c0t5000CCA221DFD971d0  7.78G  1.80T      0    420      0  51.5M
  c0t5000CCA221DFD972d0  7.67G  1.81T      0    408      0  50.2M
  c0t5000CCA221DFDA8Fd0  7.72G  1.80T      0    415      0  50.9M
  c0t5000CCA221DFDA97d0  7.72G  1.80T      0    415      0  51.2M
  c0t50024E900368D105d0  7.73G  1.80T      0    416  8.40K  50.6M
  c0t50024E900368D10Ad0  7.77G  1.80T      0    416      0  51.2M
  c0t50024E900368D163d0  7.68G  1.80T      0    415  8.40K  50.6M

ash-3.00# sh fb.sh
parsing profile for config: seqread100g
Running tmpfs-seqwrite1-Jul_11_2010-21h_47m_48s/seqread100g/thisrun.f
FileBench Version 1.3.4
 1780: 0.004: Multi Stream Write Version 2.0 personality successfully loaded
 1780: 0.005: Creating/pre-allocating files and filesets
 1780: 0.005: File largefile1: mbytes=512000
 1780: 0.005: Creating file largefile1...
 1780: 426.008: Preallocated 1 of 1 of file largefile1 in 427 seconds
 1780: 426.009: waiting for fileset pre-allocation to finish
 1780: 426.009: Running '/opt/filebench/scripts/fs_flush tmpfs /monster'
filesystem type is: tmpfs, no action required, so exiting
 1780: 426.015: Change dir to -tmpfs-seqwrite1-Jul_11_2010-21h_47m_48s/seqread100g
 1780: 426.015: Starting 1 seqwrite instances
 1795: 427.016: Starting 10 seqwrite1 threads
 1780: 430.016: Running...
 1780: 460.053: Run took 30 seconds...
 1780: 460.054: Per-Operation Breakdown
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqwrite1                6980ops/s 6979.2mb/s      1.4ms/op      577us/op-cpu

 1780: 460.054: 
IO Summary:      209646 ops 6979.5 ops/s, (0/6980 r/w) 6979.2mb/s,    692us cpu/op,   1.4ms latency
 1780: 460.054: Stats dump to file 'stats.seqread100g.out'
 1780: 460.054: in statsdump stats.seqread100g.out
 1780: 460.619: Shutting down processes
 
I could not get lsiutil to work with SAS2008 based controllers, but this could be a problem with the linux driver not providing a control device. You can still check the link speed in the controller BIOS.
 
I could not get lsiutil to work with SAS2008 based controllers, but this could be a problem with the linux driver not providing a control device. You can still check the link speed in the controller BIOS.

Based on other posts, it seems that it hasn't been updated to work with SAS2008 yet....very frustrating. I'm going to call LSI today and find out what the deal is.
 
I used two LSI 9211-8i HBAs in a system (should be nearly the same controllers as 9200-8e, just internal) and the system would hang every time I tried to enter the controller BIOS. I had to remove one of the controllers to change the BIOS settings.

A nice feature of the Supermicro X8SIA-F board I used is, that one can disable option ROMs per slot. This incredibly speeds up the boot process and to my observations does not change anything for the OS (Linux) as long as it does not have to boot from one of the attached drives.
 
A nice feature of the Supermicro X8SIA-F board I used is, that one can disable option ROMs per slot. This incredibly speeds up the boot process and to my observations does not change anything for the OS (Linux) as long as it does not have to boot from one of the attached drives.

I'm trying not to buy anymore LGA1156 hardware, but you are making it difficult for me with posts like that. :)
 
THANKS!

Now where can I find a version of it that works?

Code:
# ./lsiutil

LSI Logic MPT Configuration Utility, Version 1.62, January 14, 2009

0 MPT Ports found

Not much help for you.. But version 1.62 works in w2k8 r2, and does detect the sas2008 based controller in that system. Options #16 and #68 show port details.

There are different builds for 32-bit and 64-bit win/linux/solaris. Make sure you are using the correct build.
 
When I hang only 6 drives off a SAS wide port I easily get 100MB/s per drive...it drops to half that with 12 drives per wide port...I should be able to get 4 ports * 6Gbps = 24Gbps / 8b/B = 3GB/s = 3000MB/s

Instead I'm only getting 500MB/s...not sure what's wrong.

Depending on how the sas ports on the hba are configured, this thread may apply to you.
The author writes about single wide port vs dual wide port multipathing vs other multipath configs.
 
Hi,

I'm currently building a similar system and I'm wondering which model SC847 you are using, is it the E16 or E26 ?
 
Back
Top