Hardware for ZFS expandable storage?

J

jaw4322

n00b
Joined
May 26, 2010
Messages
58
I am looking into building a ZFS expandable storage system. I have read online docs and example specs, but still have some questions. I would also like recommendations on the setup. What I'm looking at doing is using a Norco 4220 with SAS back-plane, and separating the unit into to pools. Both being Raidz2 (6, 2parity times 2 for the chassis, and each pool has a hot-swap drive). I also want to be able to expand beyond one unit by SAS expanders (same Raidz2 setup in the expansions). I've never done this and would like some detailed info on what hardware would be needed in the main unit and the expansion chassis's. Also, what is maximum number of expansion chassis's for this setup?

Here is what i was thinking:

Main Chassis:
Chassis Norco 4220
SUPERMICRO MBD-X8SAX-O LGA 1366
Kingston ValueRAM 6GB (3 x 2GB) x2
Western Digital Caviar Black WD1501FASS 1.5TB x16 for the 2 Raidz2 storage pools (only 8 to start)
32GB Intel X25-E x2 Mirror for ZIL
40GB Intel X-25V x2 Mirror for Boot
HP SAS Expander? Would this work? Im guessing not.

Expansion Chassis's:
Chassis Norco 4220
Chenbro CK13601 24 internal, 1 External in, 1 External out

Thats where I'm stuck, i dont know what to put into the main chassis for the SAS back-plane connection, and for the expansion chassis connections. Thanks for the help.

Also, I understand the risks of using ZFS, and don't want to go OEM.
 
Might want to check out this link:
http://www.servethehome.com/sas-expanders-build-jbod-das-enclosure-save-iteration-2/

For FreeBSD, ZFS and SAS, this card gets recommended often:
$120 - Supermicro AOC-USAS-L8i PCI-Ex8 8 Port SATA Controller Card

You will need to modify them to get them to fit onto ATX specifications correctly. You'll need about three of those cards to use all 20 of the drive bays in the Norco. You will have spare SFF-8087 port on one of those cards as result.

As for the expansion portion, this is still a bit uncharted territory for me so I recommend getting a second opinion on my recommendation: Connect that spare SFF-8087 port to this SFF-8087 to SFF-8088 adapter that would be installed in the main chasis:
http://www.pc-pitstop.com/sas_cables_adapters/AD8788-1.asp

Then set up the HP SAS Expander as shown in the first link I provided. After you do that, connect the HP SAS Expander to the SFF-8088 adapter on the main chasis via SFF-8088 cable and you should be set.
 
Some notes:

- SUPERMICRO MBD-X8SAX-O LGA 1366
Seems an expensive board but it misses all two things other SuperMicro boards have to offer:
- lots of PCI-express (only 3 ports)
- lots of SAS/SATA (only 6 onboard SATA)

This board instead have extra's:
http://www.supermicro.com/products/motherboard/Xeon3000/X58/X8ST3-F.cfm
- total of 14 SATA/SAS ports
- 4x PCI-express (last one is x4 electrically; others are x8)

The onboard SAS ports are the same chip as SuperMicro USAS-L8i; the LSI 1068E which is a greatly supported non-RAID SAS HBA for use with ZFS. I have two of those; and this board has basically one card already integrated.


- HDDs:
Western Digital Caviar Black WD1501FASS 1.5TB x16 for the 2 Raidz2 storage pools (only 8 to start)

Are you sure you want 7200rpm? A modern 5400rpm with 666GB-platter would seem to be as fast for your purpose but only half the power consumption. Definately worth a look since 7200rpm disks especially on ZFS are becoming depricated. By combining SSDs and HDDs you get the high sequential speeds from the HDDs and the small random reads from the SSD; the perfect combination. So why need high rpm disks? 5400rpm disks from the same series may also be more reliable.

- 32GB Intel X25-E x2 Mirror for ZIL.
A ZIL needs safe writes, and mirroring is kind of a waste; you won't need mirroring ZIL anymore if you got ZFS version 19 (remove SLOG devices). Also, Intel writes unsafely so isn't really suitable as ZIL device. This has to do with the HPA mapping table in DRAM memory; it will be changed whenever your SSD maps virtual space to physical space; this change is in the DRAM chip but when the power goes down the contents of DRAM are not backed up on NAND; it's too late for that. So you can lose quite a bit not just the last write. Still have to do more investigating, but i've considered a RAMdisk for this purpose with Acard (cheap RAMdisk via SATA with BBU). And SSDs with supercapacitor (some sandforce models?) would also be very suitable to this task.
The X25-E series are likely not worth their money at the moment; why don't you wait with the SSDs when the next wave of SSDs is coming around Christmas? You can always add ZIL (SLOG) and Cache (L2ARC) disks to your pool later. You don't have to do this all at once.

- 40GB Intel X-25V x2 Mirror for Boot
I have these SSDs and they are great for random reads (cache device) but you may not need a mirror and you may not need a separate boot disk; you can choose to boot from your main HDD array instead; every HDD would be bootable. Such a setup is not hard at all. Having a DVD-ROM in it for allowing to boot an .iso is recommended though; for disaster recovery.

Instead, dedicate the SSDs to L2ARC or SLOG. You can have one (good) SSD that does both. Currently i don't know any good SSDs for SLOG device.

- Expander or HBA? If you pick the mobo i suggested, you would only need one additional controller. If that's true then you can avoid an expander. It may work, it may be a little bit cheaper, but it is sharing your bandwidth. If possible look for dedicated bandwidth per drive; to avoid any performance problems as you expand you pool.
Then again, an expander can chealy give you lots more ports, but if the difference in price is very small then i would prefer an expander/port multiplier-free solution.


Hope any of this is helpful. Good luck on your setup! Feel free to ask any more questions.
 
Last edited:
jaw4322 said:
Also, what is maximum number of expansion chassis's for this setup?
Click to expand...
Usually limited to 122 disks, AIUI.
jaw4322 said:
Here is what i was thinking:

Main Chassis:
Chassis Norco 4220
SUPERMICRO MBD-X8SAX-O LGA 1366
Kingston ValueRAM 6GB (3 x 2GB) x2
Western Digital Caviar Black WD1501FASS 1.5TB x16 for the 2 Raidz2 storage pools (only 8 to start)
32GB Intel X25-E x2 Mirror for ZIL
40GB Intel X-25V x2 Mirror for Boot
Click to expand...
Looks like a reasonable setup. Which SAS controller are you planning on using? Why WD in particular?
jaw4322 said:
HP SAS Expander? Would this work? Im guessing not.
Click to expand...
I have an HP sas expander connected to my LSI 3442E-R, and it works fine.
jaw4322 said:
Expansion Chassis's:
Chassis Norco 4220
Chenbro CK13601 24 internal, 1 External in, 1 External out

Thats where I'm stuck, i dont know what to put into the main chassis for the SAS back-plane connection, and for the expansion chassis connections. Thanks for the help.
Click to expand...
The HP sas expander has an external port on it, which can be used to daisy-chain additional units.
jaw4322 said:
Also, I understand the risks of using ZFS, and don't want to go OEM.
Click to expand...
Which operating system will you run on this machine?
 
Thank you guys for all the info. This is great stuff. I appreciate the detailed information.

Mesa, the motherboard suggestion is awesome. To have the on board SAS will definitely help. Then like you said i can add a LSISAS1068E for the other eight drives if/when needed. As far as expanding beyond the main chassis, can i use another LSISAS1068E which then connects to Chenbro CK13601 in the expansion chassis (using a SFF-8087 to SFF-8088 adapter, as suggested by Danny) ? Is this the proper way to do this? Lets say i want to add another chassis on top of that and go from the 1st expansion chassis' s CK13601 to the 2nd expansion chassis's CK13601. Will i run into any performance problems, or am i not thinking about linking/expansion correctly?

I will also take your advice on the L2ARC and Slog, and will wait to add this at a later time. Dont truly understand that side yet anyway. As far as hard drive, i will look into your suggestion on using a 5400 rpm drive, but do like WD's, and have never had problems with them. I'm also leaning towards using FreeBDS w/ Mesa's web-gui addon. That project is very interesting and I would like to thank Mesa for the time and effort he has put into it. For noobs like me, that will be very helpful. ;)
 
can i use another LSISAS1068E which then connects to Chenbro CK13601 in the expansion chassis
Click to expand...
There might be adapters with LSI 1068E chip but with external connectors. Looked at SuperMicro, but they only have 1 Mini-SAS internal + 1 mini-SAS external. You would want 2 mini-SAS external (= 8 drives).

If you connect this way, you connect at full bandwidth; so no issues here. As long as the hardware all works together; but that shouldn't be an issue with plain HBA.

So you can expand the system by using external casings for the HDD and use Mini-SAS cables; one cable feeds 4 drives. That quite cool, but i found such hardware generally expensive compared to cheap HDDs. So i just built myself a large full-tower casing for my main ZFS server. But external might be much cleaner with cables and easier to move your disks around.

SLOG i would wait until really suitable SSDs pop up (note that you only need a few gigs for this; a very good 10GB SLC SSD with safe writing would be excellent. Virtually NO reads from SLOG! Only reads when recovering from power down/crash.

L2ARC is very cool and losing L2ARC disks (well SSDs) is also no problem at all; this acts much like RAM filecache but only your SSD is much bigger than your RAM. With an 80GB SSD or so, you could accelerate reading small files from your array; this is a really cool feature of ZFS. It also doesn't require extremely fast SSDs, though the Intels would be great here due to their excellent random reads (which is what you need) and super low latency. X25-V 40GB is best value here; as they reach up to 200MB/s *RANDOM* reads which is about the same as the 80GB with double the channels. Thus getting two 40GBs and teaming them together is great.

You can perform a secure erase on the SSDs to reset them to factory condition; L2ARC does not issue TRIM and due to its function this might not be desirable even. Instead, once per 6 months or so just detach the SSD from your ZFS pool and nuke it with this command:
newfs -E -b 65536 /dev/<SSD>

That would issue TRIM command on the entire surface, effectively the same as a secure erase. Then add the SSD to your pool again as L2ARC and it's fresh and ready to rock! So you wouldn't even need to reboot your system or ever leave FreeBSD OS to perform 'secure erase'. This also works across RAIDs. So if you got your two SSDs in casual geom_stripe you could just newfs the stripe (RAID0) and both SSDs get TRIM-ed across their surface. TRIM on RAID already works on GEOM I/O framework; but ZFS filesystem likely won't support it anytime soon due to its design. It's also less required as with copy-on-write the SSDs won't have to do much remapping anyway.

So i do think SSDs can play a significant role in adding random read performance to your pool; which is quite important for a NAS if you store alot on your fileserver. For example, i run all my linux workstations on the ZFS pool; none of the workstations have a local disk; they boot via the network and get their system disk through iSCSI from the ZFS fileserver. Since this has L2ARC; it's as if all my workstations are on SSDs as the read latency stays very low; the network doesn't add much latency compared to the huge latencies that HDDs have when they have to seek to a location. 0.4ms versus 5-20ms is just great. And the best thing: all this goes automatically; your most accessed files will be cached on the SSD.

Nice to hear you're interested in my project. Please know that it's not yet advisable to use for real data. My first targeted stable release is 0.2.0; 0.1.6 is about to release next week. So i would hope to be able to get to 0.2.0 around november/december. But perhaps that's too conservative; right now i'm doing all the hard stuff; once that works it's just extending the web-interface and things should go rather smooth. I really enjoy working on this project, and all the great feedback is really nice.

But whatever you run, (Open)Solaris, FreeBSD, FreeNAS or Mesa; you should have the ability to switch to any alternative. The only limitation here is supported ZFS versions. FreeBSD getting ZFS v28 is really cool though; this is great news for my project too!
 
Mesa, thanks for the info. I will definitely be keeping an eye on your project. In case anyone ever needs it, this is what i will probably be doing, hardware wise. If anyone sees anything wrong, please let me know. Will probably start ordering stuff in a week or two.

Main Chassis: approx 18TB usable space, 9TB to start x2 RaidZ2

Norco 4220 20 HostSwap $350
Supermicro X8ST3-F $330
Intel Xeon E5520 Nehalem 2.26GHz $380
KVR1333D3E9SK3/6GI Ram x2 $340
Western Digital WD1501FASS 1.5TB x9 (6 data, 2 parity, 1 spare/hotswap) $990($110each)
Intel X25-V 40GB SSD (L2ARC) x2 $200
Supermicro LSI MegaRAID LSISAS1068E x2 $240
1-Port SFF-8087 to SFF-8088 Adapter $30
Power Supply: $170
Total: approx $3030


If ever needed:
Second expansion: Just need to buy 9 more drives
Western Digital WD1501FASS 1.5TB x9 (6 data, 2 parity, 1 spare/hotswap) $990($110each)
First chassis: ($223 per usable TB)

If ever needed:
Third expansion: Put together JBOD and connect by SAS expander
Expansion Chassis's: approx 21TB usable space, 10.5TB to start
Chassis Norco 4220 $350
Chenbro CK13601 SAS Expander 24 internal, 1 External in, 1 External out $260
Power Supply: $170
Western Digital WD1501FASS 1.5TB x20 2RaidZ2(14 data, 4 parity, 2 spare/hotswap) $2200
Can do this in two stages as well. Buy 10 drives, then another 10 when needed.
Total: approx $2980 ($142 per usable TB)
 
Hey jaw4322

I have a similar setup or needs...

Currently running 20x 2TB hard drives giving me 28.4TB raidz2. Effectively two zpools of 10 drives.

With the chenbro 24 port expander I can add in a UEK-expander in another norco and connect it via the external sas connectors to add more space to the system.

My build thread is over at OCAU: http://forums.overclockers.com.au/showthread.php?t=882364
 
d00dz, awesome build...exactly what a noob would need. Great info, I already book marked it. I know i'll run into the same problems. Ha!
 
d00dz
From what I can tell you are using opensolaris build 134, from http://www.genunix.org/ ?. Why did you decide to go with opensolaris over FreeBsd? I'm very new to these OS's and trying to figure out which to go with. Also, how did you install the OS? Did you follow a guide somewhere? Any help with basic install instruction and basic setup would be greatly appreciated.
 
Im not sure, at the time ~3 months ago OCAU had a good tut on how to build a osol+zfs system so I read it and re-read it and followed it and yea got to like it.

IMO having osol dead doesnt really affect my situation as it's currently works and no need for updates, since the data or hardware wont be changing. Even expanding will only expand without another mobo (which is good).

If I was to redo it I might think of another OS but will have to decide the easy of use/setup.

Osol was very very easy to setup. Download iso (yes from that site) Burn iso to cd, boot from cd. Follow gui install. Setup NIC/IP's in GUI then it was shh to the box via another laptop. Im not too fond on gui so text was better for me. I didnt need to install any drivers at all! I liked this. :)
 
Opensolaris also does Kernel mode CIFS support, and not SAMBA like Linux and FreeBSD do. Kernel CIFS support is REALLY fast, and while some have had decent luck with SAMBA, it's very finicky and hard to reliably configure very high throughput rates that the kernel CIFS delivers easily.
 
You must log in or register to reply here.
Back
Top