YAZBT (Yet Another ZFS Build Thread) :)

W

wheelz

Weaksauce
Joined
Feb 4, 2011
Messages
100
Hello, this is my first post here but I have done lots of reading both here and elsewhere on this topic. However like many people, I would really like some input before I pull the trigger and spend all this money. I will be using this as a back end for VMware servers and also direct attached data drives (mainly file server but a couple databases as well). Right now I have already bought the following parts:

1 x Chenbro RM23212 2U 12 bay chassis
1 x Chenbro 84H323210-004 SATA2/SAS Backplane
1 x 820W Redundant Psu 3Y 1+1
4 x Seagate Barracuda 1 TB SATAII 300MB/s 7200rpm HD
1 x Intel Dual GB PCIe NIC

I was planning on buying these parts to get it up and running now:

1 x Supermicro X8ST3-F ----------> now 1 x Supermicro X8DT6-F
1 x PCI Riser ARC2-733-3X8
1 x Intel Quad Core i7-920 2.60 Ghz ----------> now 1 x Intel Quad Core Xeon E5606 2.13 Ghz
1 x Evercool Heatsink/Fan HPK-10025EA ----------> now 1 x Intel Thermal Solution STS100C Processor Cooler
3 x Kingston Value DDR3 1333Mhz 4GB DIMM ----------> now 3 x Samsung Dual Rank 8GB ECC RDIMMs
2 x SFF 8087 to sata breakout cable
2 x OCZ Vertex 2 60 GB SSD ----------> ?????
1 x Seagate Constellation ES 1 TB SATAII 300 MB/s 7200rpm HD
2 x Super Talent 16 GB MLC 7-pin SATA DOM (mirrored for OS)

So I have a few questions though. First I was trying to conserve bays, so I saw others had talked about using the same SSD drives for both cache and OS. My thought was to split those two drives into 3 volumes. One mirrored volume for the OS (Solaris Express or Open Indiana - perhaps on top of VMware), one small mirrored volume for the ZIL log, and then the rest of the space as a spanned volume for L2ARC. This way I can leave as many bays for actual data as possible, have redundancy where needed and still get the performance gains. Will this work? I saw some people suggesting this but no real confirmation on if this will work.

Of course now that the OCZ stuff has happened I am rethinking the model of the SSDs. Are there any other brands that have a similar performance/size/price ratio right now?

Unlike other motherboards, it seems to have a connector for each SAS. Does that mean I do not need the breakout cable, but just a separate SATA cable for each port? Would (or should) I be able to use the other SATA ports instead of having to buy an extra card after 8 drives (or can I use breakout cables for a total of 32 possible drives without a performance hit)? Additionally, this motherboard has only SATA II 3.0 Gb/s. Any recommendations for a similar motherboard with SATA III 6.0 Gb/s, if I want to upgrade my SSDs later to faster ones (for around the same price ~$340)?

I am really on the fence with the processor as well. The motherboard is nice that it lets me choose the cheaper i7 but at the same time I cannot get ECC unless I get a Xeon. Since I can't seem to find a good price on a Quad Xeon (I want to use dedupe and possibly encryption) so I am really weighing the extra cost vs the value of ECC. Any input on that?

Also looking at expansion later down the line, I can have a total of 10 data drives. Is it better to have two 5 drive RAIDZ vdevs or to have one 10 drive RAIDZ2 vdev, and why?

Is there anything else I am not seeing? I appreciate all input. Thanks!
 
Last edited:
For the SSDs, if you can live with one without power-loss protection, your best best in that price range would be a 64GB Crucial C300. If you need power-loss protection at a reasonable price, you probably need to wait a month or two for the 2011 SSD models.

For motherboard/CPU, have you considered 5520 chipset motherboards? That is a great chipset. The Supermicro X8DTE-F is an excellent motherboard, and not much more than the one you mentioned, but has a better chipset and more upgrade potential (dual CPUs, more RAM slots). You could start with a single E5606, E5607, or E5620 CPU, depending on how much you want to spend.

X8DTE-F
http://www.supermicro.com/products/motherboard/QPI/5500/X8DT6-F.cfm?SAS=N&IPMI=Y

http://ark.intel.com/Compare.aspx?ids=52583,52582,47925,41447,
 
wheelz said:
So I have a few questions though. First I was trying to conserve bays, so I saw others had talked about using the same SSD drives for both cache and OS. My thought was to split those two drives into 3 volumes. One mirrored volume for the OS (Solaris Express or Open Indiana - perhaps on top of VMware), one small mirrored volume for the ZIL log, and then the rest of the space as a spanned volume for L2ARC. This way I can leave as many bays for actual data as possible, have redundancy where needed and still get the performance gains. Will this work? I saw some people suggesting this but no real confirmation on if this will work.

Of course now that the OCZ stuff has happened I am rethinking the model of the SSDs. Are there any other brands that have a similar performance/size/price ratio right now?

Unlike other motherboards, it seems to have a connector for each SAS. Does that mean I do not need the breakout cable, but just a separate SATA cable for each port? Would (or should) I be able to use the other SATA ports instead of having to buy an extra card after 8 drives (or can I use breakout cables for a total of 32 possible drives without a performance hit)? Additionally, this motherboard has only SATA II 3.0 Gb/s. Any recommendations for a similar motherboard with SATA III 6.0 Gb/s, if I want to upgrade my SSDs later to faster ones (for around the same price ~$340)?

I am really on the fence with the processor as well. The motherboard is nice that it lets me choose the cheaper i7 but at the same time I cannot get ECC unless I get a Xeon. Since I can't seem to find a good price on a Quad Xeon (I want to use dedupe and possibly encryption) so I am really weighing the extra cost vs the value of ECC. Any input on that?

Also looking at expansion later down the line, I can have a total of 10 data drives. Is it better to have two 5 drive RAIDZ vdevs or to have one 10 drive RAIDZ2 vdev, and why?

Is there anything else I am not seeing? I appreciate all input. Thanks!
Click to expand...

about ssd:
do not split a ssd between os and cache- really a bad idea

i suppose, you have 2 x extra internal 3,5" bays. if you use 2,5" drives, you could add here 4 drives for os or cache. you may also use an OCZ X2 pci-e ssd as read cache.

i personally would use Xeons and ECC for a server when possible.
maybee with a X8SIL-F and an additional LSI 1068 Controller (Intel)

High-End 5520 boards are better but only really needed for all-in-one systems
(ESXI with integrated ZFS-SAN)
It might be needed for encryption and dedup, they are power hungry
(dedup is extremely ram hungry, add 2,5GB ram per TB Data)

If your backplane has mini-SAS ports, you need
reverse fanout cables to connect the onboard sata.

If your backplane has regular sata ports, you need a normal 8087 fanout cable to
connect the mini-sas controller port.


for sata3 you could use a LSI 2008 (SAS2) onboard or via extra adapter.
if its important, look at a SuperMicro X8...-F mainboard with lsi 2008 instead of 1068

Gea
 
Last edited:
For this build, you can probably look into Xeon W series of processors. W35xx processors are the xeon version of 9xx processors, and does not cost as much as the dual socket cpus, such as E5620.
 
john4200 said:
Note that the WXXXX CPUs cannot be used in a dual-CPU configuration.
Click to expand...

To be clear...the W3XX or X3XX CPU's cannot do dual CPU setups. The W5XX (and obviously E or X5XX series) DO support dual CPU setups.
 
Now I am rethinking quite a bit, but that's a good thing. :)

For the SSDs, if you can live with one without power-loss protection, your best best in that price range would be a 64GB Crucial C300. If you need power-loss protection at a reasonable price, you probably need to wait a month or two for the 2011 SSD models.

For motherboard/CPU, have you considered 5520 chipset motherboards? That is a great chipset.
Click to expand...

Supercapacitor would be nice to have but I have a good UPS for this system so in theory I shouldn't need the power loss protection. I did look at that motherboard, but the big downside was the lack of the onboard sata/sas ports (only 6 non-LSI). I have seen people saying only to go with these couple of LSI controllers to avoid "Fake RAID". Does that really make a difference?

about ssd:
do not split a ssd between os and cache- really a bad idea

i suppose, you have 2 x extra internal 3,5" bays. if you use 2,5" drives, you could add here 4 drives for os or cache. you may also use an OCZ X2 pci-e ssd as read cache.
Click to expand...

Why is that a bad idea? I just like to understand. Is it the OS and the L2ARC or the OS and the ZIL that is bad (or both)? I don't have any extra internal bays in my case. Depending on which combination is bad, perhaps I could use a pair of USB sticks for the OS and/or zil log?

So it sounds like I may have some options to get the ECC for an affordable price. Given that I want to use both dedupe and encryption, do you think a single W3XX would handle this or would I need to have the possibility of using two E5606's, for example?
 
wheelz said:
I did look at that motherboard, but the big downside was the lack of the onboard sata/sas ports (only 6 non-LSI). I have seen people saying only to go with these couple of LSI controllers to avoid "Fake RAID". Does that really make a difference?
Click to expand...

I prefer to avoid on-board HBAs (beyond the standard Intel SATA ports), but if that is something you want, there are other Supermicro 5520 chipset boards that do have an 8-channel SAS HBA. The X8DT6-F is similar to the X8DTE-F I mentioned previously, but it has an 8-channel SAS 2.0 HBA on-board.

I'm not sure I completely follow your question quoted above, but for HBAs, I think the best choices now are either the LSI 9211-8i or the IBM ServeRAID M1015 in pass-thru mode. The M1015 can often be found for less than $100 on ebay. Both of those are supported in Solaris and linux. But you may (or may not) have a problem with them in BSD.
 
john4200 said:
I prefer to avoid on-board HBAs (beyond the standard Intel SATA ports), but if that is something you want, there are other Supermicro 5520 chipset boards that do have an 8-channel SAS HBA. The X8DT6-F is similar to the X8DTE-F I mentioned previously, but it has an 8-channel SAS 2.0 HBA on-board.

I'm not sure I completely follow your question quoted above, but for HBAs, I think the best choices now are either the LSI 9211-8i or the IBM ServeRAID M1015 in pass-thru mode. The M1015 can often be found for less than $100 on ebay. Both of those are supported in Solaris and linux. But you may (or may not) have a problem with them in BSD.
Click to expand...

I was looking at the onboard from a cost perspective. It looked like I could get a server class motherboard w/ onboard controller cheaper than if I even bought a desktop mb with additional controller cards. I was hoping to use an AMD chip to save cost but all desktop mbs I found could only go to 16GB ram. Server side they have the opteron line but then I might as well go Intel at that cost. Additionally with a 2U case and riser, I only have a max of 3 usable card slots.

The question was in regards to whether it really makes a difference to have an LSI 1068e or SAS 2008 based controller vs using onboard SATA ports. I've seen people say to stay away from anything else because many of them have software RAID or "fake RAID" and no way to put it in a straight through mode or "IT Mode". I was wondering if this was true (this forums seems to be more on top things than others).
 
wheelz said:
The question was in regards to whether it really makes a difference to have an LSI 1068e or SAS 2008 based controller vs using onboard SATA ports. I've seen people say to stay away from anything else because many of them have software RAID or "fake RAID" and no way to put it in a straight through mode or "IT Mode". I was wondering if this was true (this forums seems to be more on top things than others).
Click to expand...

I think you must have misunderstood. You can always use on-board SATA ports for software RAID. Software RAID can usually use any disk that you can get to show up individually to the OS. FakeRAID is different -- it combines onboard ports into a RAID using the motherboard chipset and CPU. But you can always turn it off (or not turn it on).
 
john4200 said:
I think you must have misunderstood. You can always use on-board SATA ports for software RAID. Software RAID can usually use any disk that you can get to show up individually to the OS. FakeRAID is different -- it combines onboard ports into a RAID using the motherboard chipset and CPU. But you can always turn it off (or not turn it on).
Click to expand...

Sorry, I may have confused the issue with incorrect terminology. I meant just fake RAID where it uses the CPU, chipset, etc. I had seen some say that even when you disable this, it is not the same as having true pass-through or "IT mode" support (again, I may have the terminology wrong on this) and can still cause problems - performance or otherwise. They said the only ones that have this AND have OpenSolaris drivers is controllers based on the LSI 1068e and SAS2008. Is that true? If it isn't then it opens up my options a little more.
 
I'll take that as a no. I'm not confused; just verifying that it was clear enough to answer my true question. Thanks for you help.

I'm aware of the hardware list for opensolaris but it is horrible to search for specific controllers (especially onboard). Does anyone have any guidance on what to expect from an out of the box drivers support perspective? Is it more of a whitelist or a blacklist and what kinds to use/avoid?
 
wheelz said:
I'll take that as a no. I'm not confused; just verifying that it was clear enough to answer my true question. Thanks for you help.

I'm aware of the hardware list for opensolaris but it is horrible to search for specific controllers (especially onboard). Does anyone have any guidance on what to expect from an out of the box drivers support perspective? Is it more of a whitelist or a blacklist and what kinds to use/avoid?
Click to expand...

from my experience, it may be possible to create a black-list like do not use Intel 5100 server chipset or avoid Realtek nics.

but its very difficult to create a grey-list (may work, but only, until, up from..)
that does not help

on the other side, there is a definite whitelist, like use Intel server chipsets p5000, 3420 and 5520. For example, I have never heard of problems with Supermicro X8...-F series with these chipsets.

The same with SAS/Sata controllers and Solaris. LSI 1068 ones in IT-mode are simply the best (because Sun used them on theit own storage servers). The successor LSI 2008 (SAS2) is also working without problem, at least the ones from SuperMicro Boards. I use some of them

Gea
 
I have revised my parts list based on some recommendations here. I want to go with the Supermicro X8DT6-F and E5606 since it has much better RAM/CPU expansion capabilities and 6.0 Gb speeds for any future SSDs. I also bumped the memory to 8GB RDIMMs so it is easier to expand later.

I think my only question left is to figure out how to do the OS / ZIL / L2ARC config. Gea or anyone else know what the issues around sharing SSDs would be? I want to keep 10 of my 12 drive bays for actual data. Is there any kind of combination of 2 SSDs and USB sticks that would be optimal (keeping the OS and ZIL at least mirrored)?
 
You'll need a different CPU cooler with the new motherboard. The server motherboards have threaded holes embedded in the motherboard for the cooler.

You could use the Intel BXSTS100C in your 2U chassis. It will be loud, but since the power supplies in the 2U chassis are already loud, I gather that noise is not a priority for your build.
 
wheelz said:
I think my only question left is to figure out how to do the OS / ZIL / L2ARC config. Gea or anyone else know what the issues around sharing SSDs would be? I want to keep 10 of my 12 drive bays for actual data. Is there any kind of combination of 2 SSDs and USB sticks that would be optimal (keeping the OS and ZIL at least mirrored)?
Click to expand...

about booting from usb
unless usb3 is not supported and you use ssd-class SLC usb3 sticks,
do not use usb (unsecure and slow - boot time > 15min is possibe with slow sticks)

about sharing a ssd between os/ read and write cache
Its possible, but i would not do, because

its not suggested
its not supported
its not expected
its not fast
pools are no longer moveable
a pool with missing write cache is not importable
on a reinstall, disk is completely formatted (until you care about)

think about:
boot from one non-mirored disk (use a good one but there are no important data on it)
use a single MLC-SSD > 50GB as read cache (mirror is useless)

add a mirrored write cache (> 2 x Ram Size, best are SLC SSD's) only if your syncronous write performance is too bad.
(useful with NFS or database use only, not used with large sequential writes)

Gea
 
john4200 said:
You'll need a different CPU cooler with the new motherboard. The server motherboards have threaded holes embedded in the motherboard for the cooler.
Click to expand...

Hello - I purchased the X8SI6-F and it did not come with threaded holes. Not sure if this is because is may be considered a workstation/server MB instead of just a server board, but if you want threaded holes I think you need to purchase a heatsink bracket that goes under the board. I forget the SM part number for this bracket.

I mounted a Core i3 with retail heatsink directly to this board (pin based, not screws). Just wanted to point this out if the OP is thinking about this particular board that Gea suggested.

* Gea - thanks for napp-it. Great help with Nexenta Core.
 
drake3 said:
Hello - I purchased the X8SI6-F and it did not come with threaded holes.
Click to expand...

That is because it is a consumer-CPU motherboard.

The Xeon motherboards have threaded holes and need different CPU cooler attachments than the consumer-CPU motherboards.

The X8DT6 definitely needs a cooler that attaches by screwing into the threaded holes in the motherboard.
 
_Gea said:
about booting from usb
unless usb3 is not supported and you use ssd-class SLC usb3 sticks,
do not use usb (unsecure and slow - boot time > 15min is possibe with slow sticks)

about sharing a ssd between os/ read and write cache
Its possible, but i would not do, because

its not suggested
its not supported
its not expected
its not fast
pools are no longer moveable
a pool with missing write cache is not importable
on a reinstall, disk is completely formatted (until you care about)

think about:
boot from one non-mirored disk (use a good one but there are no important data on it)
use a single MLC-SSD > 50GB as read cache (mirror is useless)

add a mirrored write cache (> 2 x Ram Size, best are SLC SSD's) only if your syncronous write performance is too bad.
(useful with NFS or database use only, not used with large sequential writes)

Gea
Click to expand...

Hmm, with needing 10 bays for data, I'm thinking then that I could use a couple of these kinds of devices for the OS and the ZIL (maybe even get 4 to mirror each one):

http://www.kingspec.com/solid-state-disk-products/dom-sata2channel-mlcj.htm

The only problem is the performance sucks on these so these particular ones may only work for the OS. Does anyone know of better performing versions of these that aren't outrageously priced? It kind of sucks that all these high performing SSD companies don't release their new products in smaller sizes for cheaper. There are times like with a ZIL or just an OS where you want speed but don't need that much space.
 
I found another one with a little better performance but still not great:

http://www.innodisk.com/production.jsp?flashid=45

How about if I got one or two (if I mirror) of these just for the OS and then use the two CX00's for both zil and l2arc? I could make two partitions on each drive and mirror one pair for the Zil and Stripe 0 the other pair for the l2arc. The performance wouldn't be as good as if it was separate but I would think it would still be better than standard disks. I know that MLC is not ideal for the Zil but given the cost of getting a pair of SLC drives with decent speed, it seems that I could replace the MLC drives multiple times and still be cheaper.

Also how much is recommended for the OS drive if I am going to run either Solaris Express 11 or Open Indiana with napp-it?
 
wheelz said:
I found another one with a little better performance but still not great:

http://www.innodisk.com/production.jsp?flashid=45

How about if I got one or two (if I mirror) of these just for the OS and then use the two CX00's for both zil and l2arc? I could make two partitions on each drive and mirror one pair for the Zil and Stripe 0 the other pair for the l2arc. The performance wouldn't be as good as if it was separate but I would think it would still be better than standard disks. I know that MLC is not ideal for the Zil but given the cost of getting a pair of SLC drives with decent speed, it seems that I could replace the MLC drives multiple times and still be cheaper.

Also how much is recommended for the OS drive if I am going to run either Solaris Express 11 or Open Indiana with napp-it?
Click to expand...

these dom modules are ok for system

needed size:
at least with current NexentaCore: 4GB
SE11 and OI text installer: 8 GB
SE11 and OI Live with GUI and TimeSlider: 16 GB

Gea
 
I have an update - I have my parts in (power supply took forever to get shipped - don't get me started). I have it built and installed. I have been testing it for about a week now and overall am very happy with it. I'll post my numbers when I get a chance.

I still have not purchased any SSDs for it. I am going to wait until I get my actual load on it that will be running and then try to figure out if I am pushing it enough to need it. Anyone know how I could properly test to see if l2arc or zil log cache would help once I put my full load on it?

Also I am having a hard time trying to figure out what SSD I should get, should I need it. I was really interested in the Intel 320s because the have the power protection for cheap but then I realized that they are only SATA II compatible. Has anyone else followed intel's lead for a consumer priced SSD with power protection? I would be looking to leverage my SATA III investment and use a 3rd generation SSD but of course trying to keep the cost low.
 
You must log in or register to reply here.
Back
Top