40TB buildout

L

loosh

Weaksauce
Joined
Sep 4, 2010
Messages
85
just want a thumbs up or thumbs down my selection, been doing some compatibility checks and reading around.. seems this will all go together, if i missed something please let me know.

psu: http://www.newegg.com/Product/Product.aspx?Item=N82E16817139016
mobo: http://www.newegg.com/Product/Product.aspx?Item=N82E16813182211
chassis: http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038
ram: 2x http://www.newegg.com/Product/Product.aspx?Item=N82E16820139040
cpu: http://www.provantage.com/YITEP33T.htm
ssds: 2x http://www.provantage.com/YCSMC0CH.htm
hba: http://www.provantage.com/intel-sasuc8i~7ITSP0NQ.htm
hdds: 24xTBD (probably hitachis)

i want to do 3 partitions on the ssds, one for os, l2arc, zil. gonna throw them in raid1.
i was going to go with the nova 32GBs but these wern't much more expensive for twice the amount of space, so it'll give me a considerable amount more for cache.
as far as the software side goes i want to stick to what i know and run freebsd8 on top of it all.
wanted to order most of this tonight/tomorrow, just thought i'd get a final opinion.
 
Last edited:
Don't buy the Indilinx SSDs, they are slow for L2ARC/SLOG. You would want plenty of spare space too, so a 50GB model (64GiB physical) would be your best bet i think. Most Sandforce SF1200 models are 60GB now.

These SSDs are not suitable as ZIL; you need a supercapacitor to use ZIL on SSD; the upcoming Intel G3, Marvell C400 and select Sandforce SF2500 models have a supercap onboard. So either wait with the SSDs or buy other type SSD for L2ARC only.

Your memory is the weakest point of your server. 4GB memory is too few, 4GB memory would be the minimum for an Atom-class ZFS server. For your build i would not use less than 16GiB DDR3 memory (4x4GiB).

Consider also the SuperMicro X8ST3-F (Socket 1366 - Core i7/Xeon) motherboard:

X8ST3-F_spec.jpg


Big advantage is able to install 24GiB unbuffered DDR3 memory. Or start with 4x4GiB = 16GiB and upgrade to 24GiB later.

What kind of drives did you have in mind? My own preference would be Samsung F4 and have two RAID-Z2 vdevs of 10 disks each. That would be very fast and give you 20 * 2TB raw storage, the remaining 4 ports could be used for SSD or system disk instead.
 
i had 2x on the ram for 8GB starting, i was going to start with getting 12x2TB from the start, and create a zpool on that, along with double (or even triple) my ram later on when i needed more space. i thought about mounting the ssds/os disks in the other section with the motherboard since it was a microatx i would have the space to stick them in there.

i see what you mean on the SSDs though, any drives you'd specifically recommend? i just peaked over at processors on the 1366 socket and i know nahalems are pretty good. but whats the diff between nehalem and nehalem-ep? for 1366 socket i'd probably just go with http://www.newegg.com/Product/Product.aspx?Item=N82E16819115225 as a processor.

these were the hitachis i had in mind http://www.newegg.com/Product/Product.aspx?Item=N82E16822145298 mostly because everybody has recommended them to me.

if the ssds are going to hold me back i'll just go with raptors or something for now, or just not run ZIL on the ssds.
 
Last edited:
You had 8GB ram starting but as 4 x 2GB DIMM's which would fill all your slots on the X8SIL-F. You would have to scrap or sell it to upgrade ram at all. If you want to stay with that board at the least start with 2 x 4GB dimm's so you can add to more later for 16GB total. Or just start with 16. Also...If you want to stay with X8SIL-F then perhaps save a few bucks on the CPU and drop down to the X3430 chip. The 3440 you choose is only one clock speed notch higher and adds hyper-threading....but since you won't use hyper threading anyway for a ZFS setup you might as well save a few bucks and get the 3430.

If you decide to go the X8ST3-F route and you want to have ECC ram you cannot use the core i7, you need a Xeon (X35xx series). Or rather you can run ECC ram with the Core i7 chip but the ECC will be disabled. Remember both board AND CPU need to support ECC for it to work.

I'm sure some Hitachi drives will be fine, but there is really no reason for you to use 7200 rpm drives, especially if you are aiming at having both L2ARC and ZIL sometime. Unless you know that you will be really hammering this thing so hard with synchronous i/o that the faster seek times of the spindles could actually come into play. They certainly won't make a difference for anything sequential and will only waste power over 5400 rpm drives if you can't make use of the extra rpm.

Ps- I have personally used the Samsung F4 2TB (5400rpm, HD204UI) drives that sub.mesa mentioned and so far they have been running great. I'm running a modest 9 drive RAIDz2 with hot spare. No L2ARC, no ZIL. It's generally just a mass storage NAS box for storing backup images of the companies workstations. Even so I have a small iSCSI share on the same box that I use for hosting virtual drives for test VM's from our XenServer host. It has a gigabit NIC dedicated to the iSCSI traffic and is easily fast enough to run a couple lightly loaded VM's from. And on sequential read/write from the NAS it's fast enough to saturate gigabit a few times over.
 
Last edited:
Silenus said:
You had 8GB ram starting but as 4 x 2GB DIMM's which would fill all your slots on the X8SIL-F. You would have to scrap or sell it to upgrade ram at all. If you want to stay with that board at the least start with 2 x 4GB dimm's so you can add to more later for 16GB total.
Click to expand...

doh! can't believe i missed that. must have not updated the link i had saved in the text file. http://www.newegg.com/Product/Product.aspx?Item=N82E16820134978

could i upgrade to something like this http://www.newegg.com/Product/Product.aspx?Item=N82E16820139273 later on? (didn't check compatibility)

Silenus said:
If you want to stay with X8SIL-F then perhaps save a few bucks on the CPU and drop down to the X3430 chip. The 3440 you choose is only one clock speed notch higher and adds hyper-threading....but since you won't use hyper threading anyway for a ZFS setup you might as well save a few bucks and get the 3430.
Click to expand...

i thought about this, but i will be doing some cpu intensive things at times as well, so the extra $40 doesn't really bother me.

Silenus said:
If you decide to go the X8ST3-F route and you want to have ECC ram you cannot use the core i7, you need a Xeon (X35xx series). Or rather you can run ECC ram with the Core i7 chip but the ECC will be disabled. Remember both board AND CPU need to support ECC for it to work.
Click to expand...
yeah, i know. problem is ECC ram is a bit more expensive, too. except for unbuffered.

Silenus said:
Ps- I have personally used the Samsung F4 2TB (5400rpm, HD204UI) drives that sub.mesa mentioned and so far they have been running great. I'm running a modest 9 drive RAIDz2 with hot spare. No L2ARC, no ZIL. It's generally just a mass storage NAS box for storing backup images of the companies workstations. Even so I have a small iSCSI share on the same box that I use for hosting virtual drives for test VM's from our XenServer host. It has a gigabit NIC dedicated to the iSCSI traffic and is easily fast enough to run a couple lightly loaded VM's from. And on sequential read/write from the NAS it's fast enough to saturate gigabit a few times over.
Click to expand...

if you're maxing out a gbit with no l2arc and no zil on the 5400rpms then that's good enough for me. i'm still waiting for my gbit hardware for the rest of my network, and the lines to my building will never reach those speeds ( in the near future anyway .. ) so perhaps i shall follow your advice, then. you've answered a few unanswered (but not asked) questions, thanks.
 
Last edited:
Hey I'm in the process of doing a 32TB build and here are my thoughts:

loosh said:
psu: http://www.newegg.com/Product/Product.aspx?Item=N82E16817139016
Click to expand...

I ended up going with a Seasonic X660 which is an 80Plus Gold PSU with 90+% efficiency. I heard at one point that Seasonic actually made Corsair PSU's but I dont know if thats true anymore. The efficiency matters to me because I'll be dropping this box into a datacenter where power is EXPENSIVE ($350/month per 20A circuit).



loosh said:
mobo: http://www.newegg.com/Product/Product.aspx?Item=N82E16813182211
Click to expand...

Good call, I chose the same one! The IPMI/KVM over IP is amazing!



loosh said:
chassis: http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038
Click to expand...

I went with a Norco 3216 (3U) because I only needed 16 data drives. The 4224 is a solid choice too and I would have gone with that had I needed more drives.



loosh said:
ram: 2x http://www.newegg.com/Product/Product.aspx?Item=N82E16820139040
Click to expand...

Why dont you use the Kingston 8GB (2x4GB) memory instead so that you have the option to add much more ram later? I'm using this RAM which is from the same family that you have but its 2x4GB instead of 2x2GB: http://www.newegg.com/Product/Product.aspx?Item=N82E16820139262&Tpk=KVR1333D3E9SK2/8G



loosh said:
cpu: http://www.provantage.com/YITEP33T.htm
Click to expand...

I don't need much CPU power so I went with an Intel Core i3 540



loosh said:
ssds: 2x http://www.provantage.com/YCSMC0CH.htm
Click to expand...

I'm also using 2xSSD's Raid1 for my OS drive. I went with 2xOCZ Solid 2's (60GB each). Will run the RAID1 for that from the Motherboard.



loosh said:
hba: http://www.provantage.com/intel-sasuc8i~7ITSP0NQ.htm
Click to expand...

I went with an Areca 1880i that I picked up on the forums here slightly used.



loosh said:
hdds: 24xTBD (probably hitachis)
Click to expand...

I'm going with 16x2TB Hitachi 5K3000's.


HTH

Cheers
 
Last edited:
SoN][c;1036865678 said:
Why dont you use the Kingston 8GB (2x4GB) memory instead so that you have the option to add much more ram later? I'm using this RAM which is from the same family that you have but its 2x4GB instead of 2x2GB: http://www.newegg.com/Product/Product.aspx?Item=N82E16820139262&Tpk=KVR1333D3E9SK2/8G
Click to expand...

perhaps!

SoN][c;1036865678 said:
I'm also using 2xSSD's Raid1 for my OS drive. I went with 2xOCZ Solid 2's (60GB each). Will run the RAID1 for that from the Motherboard.
Click to expand...

i've read a bit about some ocz ssds being kinda faulty, low performance, etc. how are these doing for you?

SoN][c;1036865678 said:
I went with an Areca 1880i that I picked up on the forums here slightly used.
Click to expand...

i would as well, but it's too expensive to just use as an HBA. :D


thanks for your input! wanted to start ordering some tonight and thought i'd get some other peoples opinions first.
 
looks like something I'd like to build... subbed. Please keep us updated on how it turns out
 
Careful about Areca; you would want a real HBA not a RAID controller! Doing so could give you problems with disconnecting disks as soon as they encounter bad sectors. This could create a very unreliable ZFS storage system as disks are disconnected outside of ZFS' control.

I would definitely use at least 8GiB RAM; Putting 4x2GiB in there could be very expensive if you want to upgrade. Especially on FreeBSD platform you would want 8GiB RAM at least.

As for SSDs, the best you could do is wait with these and buy a new 25nm SSD with supercapacitor, like the upcoming Intel G3 or Marvell C400. Those would be suitable for both L2ARC and SLOG feature, and thus provide the biggest benefit to you for the money invested. You can add L2ARC later so i would wait a month or two with the SSDs.
 
omniscence said:
Don't buy this! It is registered (will not work on a X58 platform like the X8ST3-F) and it has x4 chips, which will not work with ANY LGA1156 processors.
Click to expand...

been busy with work and never had a chance to check the compatibility, didn't get this but went with the 2x4GB sticks instead. finally was able to sit down and check this thread again - thanks for the heads up!

ordered all gear last night. :D


edit: sub.mesa, i decided not to use l2arc/zil in the end, perhaps in the future when price comes down on the newer ssds and it wouldn't be too costly to do it i will approach the idea again, but for right now it's not really a requirement. appreciate the info though. i can always just pick up the new ssds later on and completely dedicate one to l2arc. :)
 
got all the cables + remaining hardware to run 12 drives for now on order, seems it will all show up around wednesday/thursday of this week (some on friday too)
 
loosh said:
edit: sub.mesa, i decided not to use l2arc/zil in the end, perhaps in the future when price comes down on the newer ssds and it wouldn't be too costly to do it i will approach the idea again, but for right now it's not really a requirement. appreciate the info though. i can always just pick up the new ssds later on and completely dedicate one to l2arc. :)
Click to expand...

I bought a 60G Vertex 2 for ZIL/L2ARC purposes in November, finally got around to setting it up (bought it far far too early). For my workload (and most home NASes, really), loss of the ZIL will generally be non-fatal. Pool versions before 19 (IIRC) are completely hosed with ZIL death, but newer versions, I think you just lose whatever was in the ZIL and in the middle of getting committed to disk.

Code: 
  pool: tank
 state: ONLINE
 scan: scrub repaired 0 in 79838h15m with 0 errors on Mon Feb 14 05:08:11 2011
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            c1t0d0  ONLINE       0     0     0
            c1t1d0  ONLINE       0     0     0
            c1t2d0  ONLINE       0     0     0
            c1t3d0  ONLINE       0     0     0
            c1t4d0  ONLINE       0     0     0
            c1t5d0  ONLINE       0     0     0
          raidz2-1  ONLINE       0     0     0
            c2t4d0  ONLINE       0     0     0
            c2t5d0  ONLINE       0     0     0
            c2t0d0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0
            c2t3d0  ONLINE       0     0     0
          raidz2-2  ONLINE       0     0     0
            c3t0d0  ONLINE       0     0     0
            c3t1d0  ONLINE       0     0     0
            c3t2d0  ONLINE       0     0     0
            c3t3d0  ONLINE       0     0     0
            c3t4d0  ONLINE       0     0     0
            c3t5d0  ONLINE       0     0     0
        logs
          c9t1d0p1  ONLINE       0     0     0
        cache
          c9t1d0p2  ONLINE       0     0     0
 
Whats up with your scrub time? I've seen it say that whilst 'calculating' but not after it had finished Oo How long do your scrubs take?
 
i sure hope you didn't scrub for 79838 hours.. and thats good to hear about ZIL death in newer revisions of ZFS. i was kind of curious when/if they were going to make it non-fatal. i'm not too concerned with data being lost as it's being written so much as after it's been written to disk.
 
Ah, scrub time says that because I had cleared CMOS, and noticed a week into machine uptime that I had forgotten to set time. Set up ntp to pull time, and updated system time whilst the scrub was running (there's a reason they tell you that services can break if you do such a huge jump in time, heh).

So the scrub thinks it started in January 2002 and finished in February 2010. Only 6 years! :downs:
 
got 10 hitachis sitting in front of me, waiting on my last 2 for the first 12 disk pool.

i'd just like to thank newegg for making sure they put enough tape on my packaging.

2011-02-22%2021.42.49.jpg


this is how it was sitting at my front door when i grabbed it. i hadn't opened the package.

the other box of drives, which a friend received - was in perfect condition. doing full sector tests on all drives right now.
 
normally i would say UPS is at fault but newegg should had known better than to use only 1 strip of tape for the box and should had wrapped the foam holding the drives better than just throwing them on the top of all the brown paper.. only 1 layer of bubble wrap, which is better than nothing - but they were pretty much all popped so it wasn't really protecting them.
 
amazingly all the disks passed sector scans.. i expect one to fail soon. just installed freebsd in zfs mirror on the 2 ssds, setting up mpt driver and such and will be creating the first 12 disk pool :)
 
done! now to just create a jail for actual usage
Code: 
mnemosyne# uptime ; uname -a ; zfs list
 2:28PM  up 4 mins, 2 users, load averages: 0.02, 0.17, 0.10
FreeBSD mnemosyne.xax.li 8.2-RELEASE FreeBSD 8.2-RELEASE #3: Sun Feb 27 11:19:37 CST 2011     [email protected]:/usr/obj/usr/src/sys/MNEMOSYNE  amd64
NAME                        USED  AVAIL  REFER  MOUNTPOINT
tank                        179K  17.7T  52.2K  /tank
zroot                      3.18G  51.5G   779M  legacy
zroot/tmp                    26K  51.5G    26K  /tmp
zroot/usr                  2.41G  51.5G  2.17G  /usr
zroot/usr/home               31K  51.5G    31K  /usr/home
zroot/usr/ports             244M  51.5G   237M  /usr/ports
zroot/usr/ports/distfiles  7.28M  51.5G  7.28M  /usr/ports/distfiles
zroot/usr/ports/packages     21K  51.5G    21K  /usr/ports/packages
zroot/var                  3.68M  51.5G   115K  /var
zroot/var/crash            22.5K  51.5G  22.5K  /var/crash
zroot/var/db               3.35M  51.5G   153K  /var/db
zroot/var/db/pkg           3.20M  51.5G  3.20M  /var/db/pkg
zroot/var/empty              21K  51.5G    21K  /var/empty
zroot/var/log              86.5K  51.5G  86.5K  /var/log
zroot/var/mail               22K  51.5G    22K  /var/mail
zroot/var/run                55K  51.5G    55K  /var/run
zroot/var/tmp                23K  51.5G    23K  /var/tmp
mnemosyne#
Code: 
mnemosyne# zpool status -v tank
  pool: tank
 state: ONLINE
 scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  raidz2    ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da3     ONLINE       0     0     0
	    da4     ONLINE       0     0     0
	    da5     ONLINE       0     0     0
	    da6     ONLINE       0     0     0
	    da7     ONLINE       0     0     0
	    da8     ONLINE       0     0     0
	    da9     ONLINE       0     0     0
	    da10    ONLINE       0     0     0
	    da11    ONLINE       0     0     0

errors: No known data errors
mnemosyne#
 
how did you setup your zpools using the 24 drives? how much space do you end up with? what's the logic/strategy with using zfs zpools? like what are the recommended number of drives in a raidz2 pool? and what do you take into consideration when planning this out?
still trying to figure out how to implement zfs with 24 disks...
 
hi, sorry for the late reply. been busy with work+school

Bookmage said:
how did you setup your zpools using the 24 drives? how much space do you end up with? what's the logic/strategy with using zfs zpools? like what are the recommended number of drives in a raidz2 pool? and what do you take into consideration when planning this out?
still trying to figure out how to implement zfs with 24 disks...
Click to expand...

as far as space goes;

Code: 
[jason@mnemosyne ~]$ zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
tank1  21.8T  1.36T  20.4T     6%  ONLINE  -
zroot  55.5G  6.61G  48.9G    11%  ONLINE  -

i only have 12 drives (+2 ssds (zroot)) in it now. i still have 6~7TB of stuff from older drives to migrate over (i actually started one earlier). as far as using 24 drives, i'm going to do it in 2 pools of 12 drives. 2 raid-z2 pools of 12 drives each. honestly you could do a single raid-z2 pool with all 24 drives, but i think you'd still only have 2 parity drives.. unless you want to go raid-z3, and have 3 parity drives (i don't think raid-z3 is in freebsd yet? maybe in 9.0-RELEASE). regardless, i would really say 12 drives per raid-z2 pool is more than enough.

]|[ Mar']['in ]|[;1036940687 said:
what was the final component list?
Click to expand...

processor - Xeon Lynnfield X3440
os disks - 2 of these in z-mirror Crucial C300 64GB MLC SSD
memory - 8GB kingston, soon to be 16GB
psu - CORSAIR AX750 750W
motherboard - MBD-X8SIL-F-O
chassis - NORCO RPC-4224
storage drives - 12x Hitachi Deskstar 5K3000 (for now, can expand to 24 drives)
HBA cards - 2x LSI 1068e (Intel SASUC8I) these are flashed to IT mode for direct passthrough to disable raid mode in order to let ZFS fully handle the drives.

i didn't list cables cause thats a given.

on another note; if anybody has a spare 6pin->4 molex cable from that psu ^ i'd be happy buy it off you for a few bucks or buy you a beer :p
 
loosh said:
on another note; if anybody has a spare 6pin->4 molex cable from that psu ^ i'd be happy buy it off you for a few bucks or buy you a beer :p
Click to expand...

I've got a bunch of spare AX750/AX850 cables, and would be happy to send you one, but...


For my first AX750 build, I needed some extra cables as well. Read in a review on Newegg that you could e-mail Corsair and they'd send you free cables. Did as told, didn't receive a response back for a few days. Started grumbling, then I received this big envelope with something like 6 cables inside SCORE!

I'll put their info up here in just a second...
 
To: '[email protected]'
Subject: Additional cables for AX750 PSU

Hello,

I am building a batch of direct attached storage devices, and am using Corsair AX750 PSU’s for this project. My backplanes require 10 molex connections, and I only have 8 currently. As I am not using SATA power connections, I would like to add an additional molex cable (called peripheral cables in Corsair literature).

In the FAQ’s on this page, http://www.corsair.com/power-supplies/modular-psus/professional-series-gold-2/ax750w.html, I was told that contacting this address would allow me to obtain additional cables.


-----

If they don't send you any cables loosh (but I'm pretty sure they will) PM me and I will send you one of mine.
 
Had the same experience with that email address; no reply or anything but two weeks later the cables showed up in the mail. Good stuff :)
 
awesome! thanks for the heads up. will be emailing them very soon :)

as far as performance goes on this setup.. (note, i have not tuned zfs at *all*)

Code: 
[jason@mnemosyne /tank1/storage]$ rm test2.bin 
[jason@mnemosyne /tank1/storage]$ dd if=/dev/zero of=./test.bin bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 1.546072 secs (678219382 bytes/sec)
[jason@mnemosyne /tank1/storage]$ mv test.bin test2.bin
[jason@mnemosyne /tank1/storage]$ dd if=./test2.bin of=/dev/null bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.213057 secs (4921574027 bytes/sec)
[jason@mnemosyne /tank1/storage]$
 
You must log in or register to reply here.
Back
Top