Project: ZFS Monster - Phase I (testbed)

Nice results there Packetboy, I had played with filebench a few months ago and could never get it installed properly... I might need to look back at it.

For the life of me I couldent get iSCSI working with any kind of speed... read a bunch of results saying it was related to the 2009.06 build So I am now updating the box to the latest svn as we speak 751MB/888 So I should have some more input later tonight about it.

Do you have more than 8 of those drives? I would love to see a bench with wx 8 drve Raidz2's, to see how the striping scales on Sata drives.
 
Kick off a tcpdump while throughput is sucking wind...eg.:

tcpdump -U -s 0 -w cifs_test.pcap port *insert_iscsi_tcp_port_number*

Let capture run for 15-30 seconds while tput low.

IM me and we'll figure out how to arrange time to analyze Pcap together.

Do you have a managed switch? e.g. so you can check ports for Ethernet errors?

The most common Ethernet problem I run into is Duplex mis-match issues (e.g. where host and switch port it's connected to do NOT negotiate full-duplex...or someone has forced something to half). That will create a situation where if you are only using 10% of your network capacity you won't even notice a problem, but as soon as you try to burst traffic the whole thing just collapses due to massive packet errors, *late* collisions, and packets generally dropped all over the floor.

I like how i missed 3-4 posts wile i was replying... lol

I need to reload my box that has Intel NICs in it since when i Vlited it long ago i musta taken out iScsi... so I was doing the test on a box that i THINK has a realtek in it (onboard)

I can do a packet log here once the server is done upgrading (removal phase right now) Unfortunatly the setup here has an unmanaged switch untill I move into the new house (most of my gear is in storage).

I dont think its a negotiation issue since cifs is pretty speedy, i think it was a writeback cache issue... which apparently you cannot set on the build i was running with the COMSTAR target...

Will post some more information here shortly.
 
Do you have more than 8 of those drives? I would love to see a bench with wx 8 drve Raidz2's, to see how the striping scales on Sata drives.

I have 26 of them...but I only have the 8 drive Sans digital SAS enclosure to test with...that is until 4/16 when my Supermicro SC847 enclosures are supposed to arrive...after that I'll be able to do much more interesting testing with some really monster arrays.
 
I have 26 of them...but I only have the 8 drive Sans digital SAS enclosure to test with...that is until 4/16 when my Supermicro SC847 enclosures are supposed to arrive...after that I'll be able to do much more interesting testing with some really monster arrays.

Ah alright.

Update - I installed the latest SVN (134) seems the crazy iSCSI slowdown has gone away now. Copying to the server (writing) is faster than cifs now, except reading from it is slower.. go figure...

Im a little fed up with fiddling with it for now so im going to let it be untill the expande and drives get here.
 
Ah alright.

Update - I installed the latest SVN (134) seems the crazy iSCSI slowdown has gone away now. Copying to the server (writing) is faster than cifs now, except reading from it is slower.. go figure...

Im a little fed up with fiddling with it for now so im going to let it be untill the expande and drives get here.

Nice, I just finished throwing my big server together, and now I'm updating to the latest SVN, 600MB to go before it starts doing it's thing.

One thing I found with CIFS is if you make sure to enable duplexing on all ports involved, and team 2 NICs on the server together, you can end up getting the CIFS share to push nearly 95mb/sec over gigabit ethernet, which is near the design limit when you factor in overhead.
 
True, I want that kind of performance out of a single port though ;) I will be playing with it a bunch more and might actualy optomize it a bit and see what I can come up with.
 
I've got a iscsi box (same board as 7000$ dlink rig) 3u, 16 hot swaps, mini sas cables, 8 give ports and supports link ag. Supposedly good for 850MB/s and 80k iops Getting 105MB/s over single gigE port without jumbo frames with 4 500gb es.2's in raid5. Don't have link ag working yet. I know iscsi can perform, so if the embedded raid on chip ppc CPU in my box can do those speeds with 4 drives over a single port, yours should fly. I don't know if the box is actually sas or if it's Sata but I can try running mini sas cable into my supermicro sc836e1 backplane with it's built in expander and see. I think your really being limited by crap nic. Try direct connection with several different cables on an intel nic maybe even a cheap 82574L pcie x1 nic and see how that changes things. Best I can do with a realtek 8168 on supermicro x7sla-h (atom330) is about 80mb/s even from ramdisk to ramdisk. Realtek nics suck bad. Even the crap marvel nics blow them away. Depending on OS choices you might try ebaying for some pcie x4 dual port 82576's as they support sr-iov as well as tcp offload and basically every other feature any nic can have. The up version of the card can often be found for under $100. Too bad there are no osx drivers for that nic yet. (although the 82571 dual port nic works great!) other ideas for mobo might be something like my x8dth-6f. Onboard 82576, 7 pcie x16 slots(x8 gen2 electrical) for nics etc and dual ioh for tons of native pcie bandwidth without nf200 chips and their added latency and notorious epic fail in many cases for raid cards or high bandwidth IO cards.

Also, I'm not sure how efficient the cellerons really are compared to a newer solution. What about 1156 with l3426 xeons for compute nodes? Vt-d and vt-x, 1.86 ghz plus crazy turbo, 45w tdp for 4 cores/8 threads, and that includes the northbridge. Clock per clock should totally destroy the celerons, and if you can replace 8 celerons with one that would be some huge power savings. Especially in something like a 2utwin^2 setup with 90+% efficient psu's. Still might be 2-4 year power payback rate, but would give you a few high power nodes to use for other tasks as well. Might be easier to manage also. (plus older pci nics aren't terribly power efficient compared to newer stuff and your switch port count requirements would certainly drop.)

On the switch subject, if the cellerons are just doing 10/100 and thats enough, I have a cisco catalyst 5006 switch with 144 10/100 ports and 3 open slots I'm not using that I would love to trade for something I could actually use. LOL (shipping would suck though)
 
I've got a iscsi box (same board as 7000$ dlink rig) 3u, 16 hot swaps, mini sas cables, 8 give ports and supports link ag. Supposedly good for 850MB/s and 80k iops Getting 105MB/s over single gigE port without jumbo frames with 4 500gb es.2's in raid5. Don't have link ag working yet. I know iscsi can perform, so if the embedded raid on chip ppc CPU in my box can do those speeds with 4 drives over a single port, yours should fly. I don't know if the box is actually sas or if it's Sata but I can try running mini sas cable into my supermicro sc836e1 backplane with it's built in expander and see. I think your really being limited by crap nic. Try direct connection with several different cables on an intel nic maybe even a cheap 82574L pcie x1 nic and see how that changes things. Best I can do with a realtek 8168 on supermicro x7sla-h (atom330) is about 80mb/s even from ramdisk to ramdisk. Realtek nics suck bad. Even the crap marvel nics blow them away. Depending on OS choices you might try ebaying for some pcie x4 dual port 82576's as they support sr-iov as well as tcp offload and basically every other feature any nic can have. The up version of the card can often be found for under $100. Too bad there are no osx drivers for that nic yet. (although the 82571 dual port nic works great!) other ideas for mobo might be something like my x8dth-6f. Onboard 82576, 7 pcie x16 slots(x8 gen2 electrical) for nics etc and dual ioh for tons of native pcie bandwidth without nf200 chips and their added latency and notorious epic fail in many cases for raid cards or high bandwidth IO cards.

Also, I'm not sure how efficient the cellerons really are compared to a newer solution. What about 1156 with l3426 xeons for compute nodes? Vt-d and vt-x, 1.86 ghz plus crazy turbo, 45w tdp for 4 cores/8 threads, and that includes the northbridge. Clock per clock should totally destroy the celerons, and if you can replace 8 celerons with one that would be some huge power savings. Especially in something like a 2utwin^2 setup with 90+% efficient psu's. Still might be 2-4 year power payback rate, but would give you a few high power nodes to use for other tasks as well. Might be easier to manage also. (plus older pci nics aren't terribly power efficient compared to newer stuff and your switch port count requirements would certainly drop.)

On the switch subject, if the cellerons are just doing 10/100 and thats enough, I have a cisco catalyst 5006 switch with 144 10/100 ports and 3 open slots I'm not using that I would love to trade for something I could actually use. LOL (shipping would suck though)


Are you using software raid? and what OS?

I am going to be building a little (lol... little) SAN box here shortly and am exploring some diffrent avenues in the event that Solaris iSCSI doesent do what i need.
 
I'm doing raid5 on the iscsi box, but it runs a vendor build of Linux so far as I know. It has two sets of ram, one for os and one for cache. Client os is osx on a couple hackintoshes, iscsi initiator is the free globalsan one for osx. I've also done quite a bit of software raid and some fairly basic hardware raid in osx, but nothing requiring parity. The ability to lun out my storage the way I want with the iscsi box is nice. I originally bought it just for the 3u chassis and hot swap backplanes and redundant psu's. It was going to house my atom board for storage server but when I got the iscsi board working(just needed ram lol) my ideas and plans changed a bit, heh. Supposedly with some badass drives it's rated at 850mb/s and 80k iops. That'll be more than I'll ever need that's for sure. Once I get link ag switch and a few more drives in it I'll have to see what the realistic limits are. The fact that they make a version with 10gigE is promising though. Mine just has the 8 gigabit ports on it.
 
Update: Supermicro says they are doing burn-in on my enclosures this weekend and will ship them out on Monday...eta 4/19. woot!
 
nice :D - Thats about the time my expander and drives should be here too! (also going to dig out my Intel NICs and stop being lazy and using onboard... LOL)
 
:(:(:( Supermicro now says they are encountering some kind of bug with the LSI expander in their backplane and availability of this case is delayed for another 2-3 weeks.

I'm disappointed but was expecting delays given the bleeding edge nature of this product.
 
:(:(:( Supermicro now says they are encountering some kind of bug with the LSI expander in their backplane and availability of this case is delayed for another 2-3 weeks.

I'm disappointed but was expecting delays given the bleeding edge nature of this product.

:( that sucks.

My expander got here 2 days ago, just need to wrangle up the drives to play with it now :) Results will be inc.
 
Update:

Supermicro has been struggling with some kind of LSI bug in the backplane for the 45-drive version of their chassis...they still haven't gotten it fully worked out but it doesn't afftect the scenario we are building so I finally convinced them to ship me the two SC847 enclosures we have on order with them....I'm told they are definitely shipping today show should have them by the end of next week.

Additionally, with the recent Fry's 25th aniversary sale (and other sources) I was able to snarf a total of 46 Hitachi 2TB drives for this build...have a few more to purchase to reach 100TB...sorry treadstone but this ZFS monster will be much prettier and spry than your BEAST.
 
Update: Two SC847's (45-drive backplanes) shipped from factory on Friday...200 HUNDRED pounds...tracking indicates they will arrive Friday.

We plan on building a separate 2U controller as when you use the 45 drive backplane you can't put a motherboard in the case...

Based on this project:

http://www.natecarlson.com/2010/05/07/review-supermicros-sc847a-4u-chassis-with-36-drive-bays/

Note: We are using expander based versions of the SC847...don't really understand why Nate went with non-expander...though admittedly you could get better performance depending on what the application is.


we are leaning towards using the same motherboard as Nate:

SuperMicro X8DTH-6F

http://www.supermicro.com/products/motherboard/QPI/5500/X8DTH-6F.cfm

And SAS HBA LSI 9200-8e:

http://www.lsi.com/channel/products/hba/sas_sata_hbas/external/lsisas92008e/index.html

And this case:

http://www.supermicro.com/products/chassis/2U/213/SC213A-R900LP.cfm

The X8DTH-6f has an 8 port SAS LSI 2008 controller on the motherboard (same chipset as the 9200-8e), we figure we'll connect those to the two SAS backplanes in the SM case...then use 2.5" drives in case for O/S, ZIL (SSD*2) and L2ARC (SSD).


The LSI 9200-8e will then connect to the external SC847A...I'm hoping I can connect it with TWO SAS2 wide ports for double the bandwidth, but unclear whether the SM backplanes will support this.

Each 9200-8e wide port supports 256 drives...so even this initial config allow cascading of TWELVE SC847 chassis, each with 45 drives...or 512TB.

What's even more sick is this motherboard has 7 PCI-E 2.0(!) slots...so you could in theory repeat this by adding say 5 more LSI 9200-8e with 512TB hanging off each for a total of 3.0PB...and STILL have a PCI-E slot left over for 10Gbe Ethernet.
 
Very nice. I really can't wait to see this start coming together. A model for all. :)
 
Whats the price of those cases? And I want detailed high-ress pics of them! :D
 
Whats the price of those cases? And I want detailed high-ress pics of them! :D

We're technically getting *samples* (I believe we will be getting the very first consumer delivery)...so not sure what the ultimately price will be, but we paid $1600/each.

Will definitely post photos on Friday.
 
...

Each 9200-8e wide port supports 256 drives...so even this initial config allow cascading of TWELVE SC847 chassis, each with 45 drives...or 512TB.

What's even more sick is this motherboard has 7 PCI-E 2.0(!) slots...so you could in theory repeat this by adding say 5 more LSI 9200-8e with 512TB hanging off each for a total of 3.0PB...and STILL have a PCI-E slot left over for 10Gbe Ethernet.

Theeeeyyyy'rrreeee heeerrreeeee !!!

http://img269.imageshack.us/img269/4373/dsc0519w.jpg

The tires on the handtruck are busted so had to literally throw them down the side of the hill to get them to the basement.

Proceeding with uncrating.


Fap fap fap.... :eek:
 
Hrrmmm..I'm waiting for my LSI 9200-8e SAS2 controllers to come in so I figured I'd try to connect this chassis to the Areca 1680 (NOT ix).

If I boot the machine w/o the SC847 connected the Areca comes up...if I try to boot it with the Areca connected to J0 on the fron SC847 backplane it stalls on the:

Waiting for Raid F/w to initialize

message....forever!

If I boot up w/o the SC847 connected, enter the Areca menu and then plug in the SC847 it hangs immediately and any keystroke has no effect. Areca stays hung even after disconnecting the chassis.

I'm running Areca FW 1.48 (Jan 2010).

I've tried the Areca in both JBOD and RAid mode...same results.

I'm not too concerned as I don't plan on using this controller and I've seen others use the 1680 with the SC846 backplane with is also LSI based, so rather confusing.

Thoughts anyone?
 
SWEET!!!!

Code:
AVAILABLE DISK SELECTIONS:
       0. c7d0 <DEFAULT cyl 7780 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@11/ide@0/cmdk@0,0
       1. c9t9d0 <ATA-Hitachi HDS72202-A20N-1.82TB>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@9,0
       2. c9t25d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@19,0
       3. c9t27d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@1b,0
       4. c9t28d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@1c,0
       5. c9t29d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@1d,0
       6. c9t30d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@1e,0
       7. c9t31d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@1f,0
       8. c9t32d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@20,0
       9. c9t33d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@21,0
      10. c9t34d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@22,0
      11. c9t35d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@23,0
      12. c9t36d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@24,0
      13. c9t37d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@25,0
      14. c9t38d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@26,0
      15. c9t39d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@27,0
      16. c9t40d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@28,0
      17. c9t41d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@29,0
      18. c9t42d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@2a,0
      19. c9t43d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@2b,0
      20. c9t44d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@2c,0
      21. c9t45d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@2d,0
      22. c9t46d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@2e,0
      23. c9t47d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@2f,0
      24. c9t48d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@30,0
      25. c9t49d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@31,0
      26. c9t50d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@32,0
      27. c9t51d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@33,0
      28. c9t52d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@34,0
      29. c9t53d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@35,0
      30. c9t54d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@36,0
      31. c9t55d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@37,0
      32. c9t56d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@38,0
      33. c9t57d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@39,0
      34. c9t58d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@3a,0
      35. c9t59d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@3b,0
      36. c9t60d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@3c,0
      37. c9t61d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@3d,0
      38. c9t62d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@3e,0
      39. c9t63d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@3f,0
      40. c9t64d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@40,0
      41. c9t65d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 189>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@41,0
      42. c9t66d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 189>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@42,0
      43. c9t67d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 189>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@43,0
      44. c9t68d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 189>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@44,0
      45. c9t69d0 <ATA-ST31500341AS-CC1H cyl 60798 alt 2 hd 255 sec 189>
          /pci@0,0/pci1002,5980@b/pci1000,30a0@0/sd@45,0
 
Code:
# zpool list
NAME      SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
monster  79.3T  1.24M  79.3T     0%  ONLINE  -
 
the same as any other guy
run botnet and store porn!


hows the connectivity with lsi card? no previously mentioned areca type issues?
 
FYI op you can compile/install coreutils:

ftp://mirrors.kernel.org/gnu/coreutils/

and the dd included will give you speeds like your used to seeing on linux. Here is what I get on my 20 disk raidz2 pool with my LSI SAS2 controller (AOC-USAS2-L8E). Writes are for sure being CPU bottlenecked as idle goes near 0% on my 2.66 Ghz core 2 duo:

writes:
Code:
root@opensolaris: 11:36 AM :/data# dd bs=1M count=100000 if=/dev/zero of=./100gb.bin
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 233.257 s, 450 MB/s

reads:
Code:
root@opensolaris: 11:44 AM :/data# dd bs=1M if=./100gb.bin of=/dev/null
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 131.051 s, 800 MB/s

Not all the utilities on coreutils compile so you after running configure and stuff you can cd src/ and compile individual utilities IE: 'make dd' and copy it to somewhere in your path.

Here is my layout as I am only dealing with sequential reading/writing and no need to have multiple vdevs:

Code:
root@opensolaris: 12:19 AM :~# zpool status data
  pool: data
 state: ONLINE
 scrub: scrub completed after 8h40m with 0 errors on Wed Jun  2 02:06:54 2010
config:

        NAME                         STATE     READ WRITE CKSUM
        data                         ONLINE       0     0     0
          raidz2-0                   ONLINE       0     0     0
            c4t5000C500028BD5FCd0p0  ONLINE       0     0     0
            c4t5000C50009A4D727d0p0  ONLINE       0     0     0  7K repaired
            c4t5000C50009A46AF5d0p0  ONLINE       0     0     0
            c4t5000C50009A515B0d0p0  ONLINE       0     0     0
            c4t5000C500028A81BEd0p0  ONLINE       0     0     0
            c4t5000C500028B44A1d0p0  ONLINE       0     0     0
            c4t5000C500028B415Bd0p0  ONLINE       0     0     0
            c4t5000C500028B23D2d0p0  ONLINE       0     0     0  7.50K repaired
            c4t5000C5000CC3338Dd0p0  ONLINE       0     0     0  7K repaired
            c4t5000C500027F59C8d0p0  ONLINE       0     0     0
            c4t5000C50009DBF8D4d0p0  ONLINE       0     0     0
            c4t5000C500027F3C1Fd0p0  ONLINE       0     0     0
            c4t5000C5000DAF02F3d0p0  ONLINE       0     0     0  7K repaired
            c4t5000C5000DA7ED4Ed0p0  ONLINE       0     0     0
            c4t5000C5000DAEF990d0p0  ONLINE       0     0     0
            c4t5000C5000DAEEF8Ed0p0  ONLINE       0     0     0
            c4t5000C5000DAEB881d0p0  ONLINE       0     0     0
            c4t5000C5000A121581d0p0  ONLINE       0     0     0
            c4t5000C5000DAC848Fd0p0  ONLINE       0     0     0  7K repaired
            c4t5000C50002770EE6d0p0  ONLINE       0     0     0  7K repaired

The disks:

Code:
root@opensolaris: 12:21 AM :~# iostat -E
cmdk0     Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Model: ST3400632A      Revision:  Serial No:             4NF Size: 400.09GB <400085876736 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0
sd1       Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD15 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd2       Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD14 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd3       Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD15 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd4       Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD1A Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd5       Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD04 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd6       Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD04 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd7       Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD15 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd8       Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD1A Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd9       Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD14 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd10      Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD04 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd11      Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD14 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd12      Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD14 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd13      Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD04 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd14      Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD04 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd15      Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD15 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd16      Soft Errors: 6 Hard Errors: 10 Transport Errors: 50
Vendor: ATA      Product: ST31000340AS     Revision: SD15 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 2 Device Not Ready: 0 No Device: 8 Recoverable: 6
Illegal Request: 0 Predictive Failure Analysis: 0
sd17      Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD15 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd18      Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD04 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd19      Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD15 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0
sd20      Soft Errors: 5 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000340AS     Revision: SD14 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 5
Illegal Request: 0 Predictive Failure Analysis: 0

I might get another SFF-8087 cable so I can do multipath to the SAS expander since the controller should have more headroom on it.

Whats kind of layout or you using or can you show me what zpool status monster shows?
 
What do you plan to do with 150 celerons and 100TB of space ???

Sorry that's classified. ...Ironically does actually involve botnets, but not in the way you'd think.

I will say that we are building a monster VM environment and plan to host cloned drive images on one of the two SC847 storage servers we'll have...with dedup of course. Our goal is 1000 VMs.
 
the same as any other guy
run botnet and store porn!
hows the connectivity with lsi card? no previously mentioned areca type issues?

I could NOT get the Areca 1680 to work at all (would just hang with 'waiting for RAID f/w' unless I disconnected the SC847 from it)...I gave up on that as I have no plans to use that card anyway.

I *tried* to get the LSI-9200-8e working...the card IS recognized and the LSI bios shows attached drives. However, I'm currently running 2009.06 which lacks the LSI SAS2008 chipset support this card uses. I *did* successfully update to snv_134 but that seems to introduce massive Nvidia problems due to the introduction of Xorg 1.7...I can't even get the GUI to load under snv_134. From the command prompt I see no drives, but I'm unclear whether the SAS2008 driver should be loaded automatically...I presumed I had to run the device driver tool to do that, but of course I can't do that as I have no GUI...so FRUSTRATING. Anyone know how I can force loading of the SAS2008 driver from the commmand line? With this I can test the 9200-8e.

Today, I fell back to the LSI 1068E (SAS1) card so that I had something to try that was compatible with 2009.06...that worked perfectly. Hot plugging worked great..I just keep plugging in drives, waiting about 15 seconds, and then checking if the drives were visible with format...they all came up with none of the usual hassles I've had with port multiplying...really loving SAS right now.

One gotcha with the SC847 as shipped is the 'front' and 'rear' backplane are NOT cabled together at the factory...so you need your own SFF-8087 to SFF-8087 cable (which I don't have)...not a big deal as I'm just running each backplane off it's own dedicacted SAS port for now.
 
Code:
Starting small:
zfs01:~/Downloads/coreutils-8.5/src# zpool status monster
  pool: monster
 state: ONLINE
 scrub: none requested
config:

 NAME        STATE     READ WRITE CKSUM
 monster     ONLINE       0     0     0
   c9t9d0    ONLINE       0     0     0
   c9t25d0   ONLINE       0     0     0
   c9t27d0   ONLINE       0     0     0
   c9t28d0   ONLINE       0     0     0
   c9t29d0   ONLINE       0     0     0

errors: No known data errors

(note 1 drive is on SAS port 1 and the other 4 are on SAS port 2)


Write:

zfs01:~# ./dd  bs=1000000 count=100000 if=/dev/zero of=/monster/100gb.
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 195.591 s, 511 MB/s




Read:

zfs01:~# ./dd  bs=1000000 if=/monster/100gb. of=/dev/null
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 161.829 s, 618 MB/s

CPU 77% idle btw.
 
Add 5 more:

Code:
zpool add monster c9t30d0 c9t31d0 c9t32d0 c9t33d0 c9t34d0

write:

zfs01:~# ./dd  bs=1000000 count=100000 if=/dev/zero of=/monster/100gb.
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 117.138 s, 854 MB/s

read:

zfs01:~# ./dd  bs=1000000 if=/monster/100gb. of=/dev/null
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 89.4373 s, 1.1 GB/s

CPU 50% idle.
 
15 drives - 14 on SASp1, 1 on SASp2

Code:
zpool add monster c9t35d0 c9t36d0 c9t37d0 c9t38d0 c9t39d0

Write:
zfs01:~# ./dd  bs=1000000 count=100000 if=/dev/zero of=/monster/100gb.
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 103.425 s, 967 MB/s

Read:
zfs01:~# ./dd  bs=1000000 if=/monster/100gb. of=/dev/null
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 88.0584 s, 1.1 GB/s

(uggh ...running out of steam...no improvement on the reads)

40% idle
 
Add five more drives on SASp2, so now:

20 drives, 14on SASp2, 6 on SASp2

Code:
 zpool add monster c9t55d0 c9t56d0 c9t57d0 c9t58d0 c9t59d0

write:

*barely* an improvement...broke 1GB/s though.

zfs01:~# ./dd  bs=1000000 count=100000 if=/dev/zero of=/monster/100gb.
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 95.9503 s, 1.0 GB/s

read:

zfs01:~# ./dd  bs=1000000 if=/monster/100gb. of=/dev/null
100000+0 records in
100000+0 records out
100000000000 bytes (100 GB) copied, 76.1256 s, 1.3 GB/s

32% idle...running out of CPU...it's ALL kernel time.

Code:
last pid: 14084;  load avg:  1.86,  1.13,  0.80;  up 0+00:38:40                                                                                              04:16:54
54 processes: 52 sleeping, 2 on cpu
CPU states: 31.8% idle,  0.0% user, 68.2% kernel,  0.0% iowait,  0.0% swap
Kernel: 11329 ctxsw, 7 trap, 380994 intr, 2195 syscall
Memory: 8190M phys mem, 1230M free mem, 4095M total swap, 4095M free swap
 
Back
Top