Build log, Serah

[SquidAngel]

Hi folks, new forum member here.

I'm currently in the planning stages for an upgrade of my storage server. This time around, I've decided not to go at it completely on my own but instead post a build log and hopefully get some valuable feedback from other builders during the process.

My current system is as follows:

Sutera

Case: Antec Twelve Hundred
PSU: Corsair TX650W
Motherboard: Supermicro PDSMA+
CPU: Core 2 Q6600
RAM: 8GB
GPU:: Onboard
Controller Cards: 3ware 9550-12ML
Optical Drives: N/A
Hard Drives: 12 Samsung SpinPoint F4 2TB
Battery Backup Units: Yes, 3ware
Operating System: Debian

The server is running Debian with Openfiler in a VM under KVM. The storage is provided from the host as a single lvm block device, then managed in Openfiler as a PV with LVM managed through Openfiler as well. On top of this, I have encryption and a quad core CPU limited to dual core due to an old BIOS. Performance is understandably abyssmal, but it was an interesting frankensteinian build of a mix of new parts and old salvage from my previous server, Yuna. The intention was to build a so called "All-in-one" server on a fairly small budget.


New server, Serah:

For the new storage server, I intend to once again build an All-in-one server, although this time built around ESXi with PCI passthrough of a SAS controller. The current plan looks as follows:

Case: Antec Twelve Hundred (from Sutera, which will receive an older case)
PSU: Corsair TX650W (from Sutera)
Motherboard: undecided
CPU: undecided
RAM: 32GB
GPU:: Onboard
Controller Cards: LSI 9211-8i
Optical Drives: N/A
Hard Drives: 12-20
Battery Backup Units: No
Operating System: ESXi

I'm planning to use an LSI 9211-8i SAS controller with a HP SAS Expander card, which would allow me to a bit more flexibility than the old 3ware card. Assuming no compatibility issues arise, I intend to run ESXi with my storage OS in a VM, with PCI passthrough of the controller. The storage VM would then pool the drives with ZFS and serve them to my workstations via SMB and/or iSCSI.

I'm also toying with the idea of going 10GigE for my home network. Unfortunately, 10GigE switches are still hideously expensive, so I'd likely do direct links in that case. As I'm not intending to connect more than three workstations to 10GigE, this should be fairly affordable.


Right now I'm open to recommendations for the storage OS, motherboard, NICs and any other recommendations or advice based on my current plan. I don't intend to run hardware RAID at this point, instead going for ZFS. It is my, perhaps mistaken, belief that this would give me more flexibility when it comes to future upgrades and/or expansions.

For the motherboard, I'm aiming for 32GB of memory, at least 1 PCIe x8 and 2 x4 slots, plus any slots required by the NICs. I don't really have any requiremeny for dual sockets, although if it's within a reasonable price range, I might be tempted.

 

[WestSidaz]

I would recommend norco rpc 4224 case for you instead of antec.. it gives you 24 hot swap hdd slots what makes ur life ->alot<- easier if you plan to make fileserver with expander.. it saves time and money...
I have 1 x norco rpc 4224 case and runs with LSI 9211-8i and HP sas expander.. really happy..
second norco is commin and running with same cards coz those are good..

10 gig network? where do u need it?

 

[SquidAngel]

I've been looking at the Norco, but sadly it misses out on a few points.

* It's a rack case. While this isn't normally a problem, I'd feel embarrassed if I also didn't have a rack to put it in. As I'm living in an apartment, a rack is not an option and it wouldn't be approved by my husband either.
* The Norco does not have a local reseller as far as I can tell, and by local I mean the entire country. Judging by local forums, buying a Norco from the UK would end up costing me about $700. I imagine buying from the US would set me back an equivalent amount.
* The Norco is not already in my posession, whereas the Antec is. Sadly, this is a somewhat important factor in my limited budget.

That said, the Norco looks awesome and if I was building everything from scratch, it'd be a serious contender. The Antec is currently fitted with three 4-drive cages. A fourth can and probably will be fitted for a total of 16 drives. And yes, I know that cabling will be a mess compared to having a case with a proper backplane, but ohwell.

WestSidaz, what motherboard and OS are you running with your setup?

 

[WestSidaz]

In this server what is up and running at the moment i have P5Q Pro + e8400 @ 4 GHz and Windows 7 Ultimate 64 bit.. Works great..
Tho have been thinkin about other options aswel for OS if i am gonna make some larger raid system but that is somewhere in the future..

Other norco will also have samekind of parts..

Where do u live?

 

[Dangman]

Hi folks, new forum member here.

Welcome to the forums! Hope you enjoy your time here!

Tip though: Use the search function in this sub-forum as much as possible. Sometimes it takes days for this subforum denizens to respond to threads.

I'm planning to use an LSI 9211-8i SAS controller with a HP SAS Expander card, which would allow me to a bit more flexibility than the old 3ware card. Assuming no compatibility issues arise, I intend to run ESXi with my storage OS in a VM, with PCI passthrough of the controller. The storage VM would then pool the drives with ZFS and serve them to my workstations via SMB and/or iSCSI.

Do you have plans for expanding past 16 drives? If so, how so? New case like a Norco? Adding 5in4 drive modules to that Antec 1200? I ask because your current case, as you noted, will only fit about 16 hard drives with 4in3 modules. You might be able to save some money by going with two LSI 9211-8i SAS controllers instead of a LSI controller + HP SAS Expander card.

Right now I'm open to recommendations for the storage OS, motherboard, NICs and any other recommendations or advice based on my current plan. I don't intend to run hardware RAID at this point, instead going for ZFS. It is my, perhaps mistaken, belief that this would give me more flexibility when it comes to future upgrades and/or expansions.

OS wise, it seems like the more popular OS choice now for a large scale file server with ZFS is OpenIndiana plus Napp-It.

Also, just make sure that you know about ZFS as much as possible, I recommend reading this link (ignore the references to ZFSGuru and FreeNAS):
http://hardforum.com/showpost.php?p=1036644362&postcount=2

For the motherboard, I'm aiming for 32GB of memory, at least 1 PCIe x8 and 2 x4 slots, plus any slots required by the NICs. I don't really have any requiremeny for dual sockets, although if it's within a reasonable price range, I might be tempted.

How important is that 32GB RAM requirement?

I've been looking at the Norco, but sadly it misses out on a few points.
* It's a rack case. While this isn't normally a problem, I'd feel embarrassed if I also didn't have a rack to put it in. As I'm living in an apartment, a rack is not an option and it wouldn't be approved by my husband either.

Do note that you can remove the side handles from the Norco 4220 and 4224 cases which will then allow you to stand them up like a standard ATX case.

 

[rsq]

* The Norco does not have a local reseller as far as I can tell, and by local I mean the entire country. Judging by local forums, buying a Norco from the UK would end up costing me about $700. I imagine buying from the US would set me back an equivalent amount.

That is not true, x-case imports and resells the 24 bay norco in the UK.

It is called the RM424 on their shop website:
http://www.xcase.co.uk/24-bay-Hotswap-rackmount-chassis-norco-RPC-4224-p/case-xcase-rm424.htm

It is more expensive than in the USA, but it ships with rack rails, and the 120mm fan wall.

Regards, RsQ

 

[PigLover]

Instead of the 9211-8i, consider an IBM ServerRaid M1015. You can get them used for significantly less than the 9211. In the US it is common to find then for $70-90 on ebay, though I don't know what it would end up being in the UK with shipping and import duty. The ones on ebay are almost all pulls from IBM servers. They are based on the same chipset as the 9211 and can even be flashed to 9211 firmware if you like. Highly compatible with Solaris, OI and ZFS.

If you are limited to 16 drives anyway then just get two and ditch the expander. The two M1015s will cost less than a single 9211 and you'll save ~$300 by not buying the expander that you can use for other things.

 

[Trepidati0n]

Instead of the 9211-8i, consider an IBM ServerRaid M1015. You can get them used for significantly less than the 9211. In the US it is common to find then for $70-90 on ebay, though I don't know what it would end up being in the UK with shipping and import duty. The ones on ebay are almost all pulls from IBM servers. They are based on the same chipset as the 9211 and can even be flashed to 9211 firmware if you like. Highly compatible with Solaris, OI and ZFS.

If you are limited to 16 drives anyway then just get two and ditch the expander. The two M1015s will cost less than a single 9211 and you'll save ~$300 by not buying the expander that you can use for other things.

These kinds of posts are gems...why I love this place.

 

[PigLover]

Just noticed this in another thread. Don't know if this vendor is legit or if they ship to UK, but worth a look if you are going to consider the M1015:

Here is a m1015 for $44.45 shipped...
http://www.printsavings.com/Item.aspx?sku=0898559811OP&sgd=330d317d317d308d309

 

[midnkight]

Instead of the 9211-8i, consider an IBM ServerRaid M1015. You can get them used for significantly less than the 9211. In the US it is common to find then for $70-90 on ebay, though I don't know what it would end up being in the UK with shipping and import duty. The ones on ebay are almost all pulls from IBM servers. They are based on the same chipset as the 9211 and can even be flashed to 9211 firmware if you like. Highly compatible with Solaris, OI and ZFS.

If you are limited to 16 drives anyway then just get two and ditch the expander. The two M1015s will cost less than a single 9211 and you'll save ~$300 by not buying the expander that you can use for other things.

I would have to Agree with PigLover. I own the 9211-8i, and after more researching and more info on the M1015, that would have been the significantly cheaper option for me. But I already own my card for a few months.

As for the wire mess, Once you use the SFF8087 to Sata Break out Cable, the mess is very managable. I have help setup a computer with 4 - 5x3 Hot Swap Cages, and trust me, getting the wires in that was not messy at all. If you wanted, you could find 4x3 hot swap cages that can use the SFF8087, so you would need only a SFF8087 to SFF8087 cable.

 

[SquidAngel]

Do you have plans for expanding past 16 drives? If so, how so? New case like a Norco? Adding 5in4 drive modules to that Antec 1200? I ask because your current case, as you noted, will only fit about 16 hard drives with 4in3 modules. You might be able to save some money by going with two LSI 9211-8i SAS controllers instead of a LSI controller + HP SAS Expander card.

Yes, I do have plans to expand past 16 drives. Once I get to that point, a case like the Norco will likely be an option, but right now I can't justify that cost when I already have a useful case.

OS wise, it seems like the more popular OS choice now for a large scale file server with ZFS is OpenIndiana plus Napp-It.

Looks like something I'll have to look into then. I've worked with Solaris in the past, but not extensively.

Also, just make sure that you know about ZFS as much as possible, I recommend reading this link (ignore the references to ZFSGuru and FreeNAS):
http://hardforum.com/showpost.php?p=1036644362&postcount=2

That's definitely a given. I haven't worked much with ZFS, but I've been reading up on it, and I do have some experience with various file systems.

How important is that 32GB RAM requirement?

Not excessively so, but I definitely need over 24 gigs, as VMs tend to multiply in numbers very quickly.

Do note that you can remove the side handles from the Norco 4220 and 4224 cases which will then allow you to stand them up like a standard ATX case.

Oh, god! That's even worse! The server gods would smite me with prejudice for such a breach of hardware feng shui.

Seriously though, I'd love a Norco, but I can't justify the extra cost at this point.

 

[SquidAngel]

That is not true, x-case imports and resells the 24 bay norco in the UK.

It is called the RM424 on their shop website:
http://www.xcase.co.uk/24-bay-Hotswap-rackmount-chassis-norco-RPC-4224-p/case-xcase-rm424.htm

It is more expensive than in the USA, but it ships with rack rails, and the 120mm fan wall.

Regards, RsQ

I'm not in the UK. I'm in Sweden. Buying from X-case would cost me about $700 in the end. I can't justify that additional cost when I have a case that works just fine for now.

Instead of the 9211-8i, consider an IBM ServerRaid M1015. You can get them used for significantly less than the 9211. In the US it is common to find then for $70-90 on ebay, though I don't know what it would end up being in the UK with shipping and import duty. The ones on ebay are almost all pulls from IBM servers. They are based on the same chipset as the 9211 and can even be flashed to 9211 firmware if you like. Highly compatible with Solaris, OI and ZFS.

If you are limited to 16 drives anyway then just get two and ditch the expander. The two M1015s will cost less than a single 9211 and you'll save ~$300 by not buying the expander that you can use for other things.

This is awesome advice. I am intending to expand past 16 drives. Part of my planning right now is to ensure that I can do so, but this means I could get two M1015s and get an expander card later once I need to expand further.


Thanks everyone for your input so far. This is exactly what I was hoping for when signing up to the forum.

 

[Trepidati0n]

I'm not in the UK. I'm in Sweden. Buying from X-case would cost me about $700 in the end. I can't justify that additional cost when I have a case that works just fine for now.




This is awesome advice. I am intending to expand past 16 drives. Part of my planning right now is to ensure that I can do so, but this means I could get two M1015s and get an expander card later once I need to expand further.


Thanks everyone for your input so far. This is exactly what I was hoping for when signing up to the forum.

...or just have three M1015's and screw the expander? 24 port for < $150 seems wise...no?

 

[SquidAngel]

As far as I recall, ESXi can only pass through two PCI(e) devices to a guest VM, which limits me to two cards.

 

[SquidAngel]

Just found a local reseller of the M1015 for a very affordable price. Slightly more expensive than eBay, but local warranty is good to have.

 

[farscapesg1]

I may be wrong, but I believe with 4.1 they increased the number of VMDirectPath passthrough devices to 4 per VM and 8 total for the host...

 

[SquidAngel]

Actually, you're quite correct. I'm trying to find out if they've changed this for 5.0, but for now, 4 cards should be more than enough.