New file server time... advice much appreciated!

S

Steve01S4

Limp Gawd
Joined
Jan 30, 2006
Messages
212
So, my current file server has hard locked twice in the last 48 hours, prompting some unpleasant (but successful) RAID recovery. I'm going to consider this a Bad Thing (tm) and build a new one nearly from scratch, only transferring over the 1 month old PSU and the 2tb drives. The primary use of this thing is media storage, though backups of my desktop, htpc, and laptop would be nice as well, and I do have a relatively tiny amount of absolutely critical data (already backed up offsite, but more backups = more better).

My plan is slightly unorthodox, though I've at least heard of others doing it. The system will be virtualized, and run (at minimum) Linux and WHS Vail (upon release). It may run 2008r2 and/or something with ZFS as well. I'll use the UNIX VMs to handle the RAID and export those volumes over iSCSI to the WHS VM. I've tested this already and it has worked flawlessly with the WHS betas, including online RAID expansion, so I'm confident that the plan will work. Let's ignore the issue of software and focus on hardware, since I feel completely confident that I can manage any combination of server operating systems I decide to use, and that I can choose the ones that best fit my particular needs. I do need enough power to transcode video on the fly, so the CPU may be overkill for strict file serving needs.

Also... Budget. As low as possible but no lower. Doing it right is more important than saving $100. It's less important than saving $1000. Anywhere in between, use your best judgment.

I'm going to copy and paste some of DannyBUI's questions from another thread, just in case they help here.

1) Is this a fresh from the ground up build? As in you won't be reusing any parts?

Parts being reused: 8x 2tb EARS, Seasonic X750 Gold, Supermicro AOC-SAT2-MV8 (possibly, but only temporary until I get my hands on an HP SAS expander)

2) How much storage space do you need initially? i.e 2TB?

10tb today, and my needs are growing by ~2TB per month.

3) how much storage space do you want eventually? i.e 24TB?

Unknown. I expect this system will expand into a second chassis. Assume I want a very large amount of storage, and the capability to expand to multiple external enclosures.

4) Is noise a factor at all?

Absolutely. I plan on ordering the 120mm bracket for the 4224 and using high quality fans in it. I don't expect silence, but quiet is critical. (Yes, quiet is relative. Let's make it as quiet as possible. Fan recommendations are much appreciated)

5) Where do you live? As in, what state and/or country?

Denver, CO

6) When do you plan on building/buying the server(s)?

I'll most likely order it on Monday from the egg.

7) Do you have any familiarity at all with Linux or FreeBSD

My first Linux install used kernel 0.99.x. My first FreeBSD install was 2.x. I still have a couple of UltraSpac boxes in the closet with Solaris 2.6 on them. I'd consider myself fluent in Linux and very proficient in all other *nix operating systems I've used. I'm far more comfortable in my least favorite *nix than Windows.


Here are my current thoughts:

SUPERMICRO MBD-X8SIL-F-O
Intel Xeon X3440
Kingston 4GB 240-Pin DDR3 SDRAM ECC Registered DDR3 x2
NORCO RPC-4224
SUPERMICRO AOC-SASLP-MV8

The HP SAS expander will come later.I'll likely add 4 more WD EARS drives now, and hope that holds me over til 3tb drives are available. I understand the cabling requirements here, so I've left the cables out of this list. I'd love fan recommendations to silence the server as much as possible without cooking it, though. Any recommendations or advice are much appreciated. If I've left anything important out, please ask... Oktoberfest just started here, and that's my excuse for forgetfulness! :p
 
Last edited:
Only thing I can say is that your HD controller is PCI-X so you'll be stuck in the pci slot on that board, and that you need to be careful with 4k sector drives (and the HD controller if using solaris is a possibility)

edit: Also, make sure that RAM is on the compatibility list, the xeon's like x8 but not x4 afaik
 
novadude said:
Only thing I can say is that your HD controller is PCI-X so you'll be stuck in the pci slot on that board, and that you need to be careful with 4k sector drives (and the HD controller if using solaris is a possibility)

edit: Also, make sure that RAM is on the compatibility list, the xeon's like x8 but not x4 afaik
Click to expand...

I'm going to try to avoid using my old controller at all; it won't ever take more than a couple of drives before I get the SAS expander. It's just there in case I need to hook up a few extra drives sooner rather than later. It's been in a PCI-X slot until now. I may add a second SAS controller up front to make sure that I don't end up starved for bandwidth. The extra controller would eventually run the external enclosure after I get an expander. The 4k sector drives work fine in Linux for RAID, so if they're a problem in Solaris I'll stick with Linux RAID 6.

I'll check on the RAM, though. Thanks for the input.
 
You mention you want the ability to expand to 2nd and possible more chassis if so I would consider going with SM X8ST3-F motherboard. It has built in sas hba and 4x pcie slots so adding controllers for extra chassis wouldn't be a problem. The mobo + socket 1366 xeon would run you about extra $200 but you wouldn't need a sas controller to start + plenty of room to grow.
2nd I would ditch the SASLP-MV8 in favor of lsi based AOC-USAS-L8i. The LSI chipset has much better driver support then the Marvell chip.
 
That x4 RAM will not work with the Intel 3420 chipset. If you check the spec sheet for the X8SIL, it requires x8 RAM. If you want to stick with 1333 Kingston ECC registered 4GB DIMMs, it is going to cost you to get x8:

http://www.newegg.com/Product/Product.aspx?Item=N82E16820139141


Here's a quad-rank x8 1066 DIMM for less money:

http://www.newegg.com/Product/Product.aspx?Item=N82E16820139024

You can also look at the tested memory list for that MB. Some of them have Supermicro part numbers, which are a little hard to find. But if you google them, you can usually find them at provantage or similar retailers.
 
Thanks for the info on memory, guys. The 4GB sticks on the Supermicro compatibility list are not cheap. That may be enough to turn me away from that board, since I'm not sure I want to gamble with a chipset that's so picky about RAM.

axan said:
You mention you want the ability to expand to 2nd and possible more chassis if so I would consider going with SM X8ST3-F motherboard. It has built in sas hba and 4x pcie slots so adding controllers for extra chassis wouldn't be a problem. The mobo + socket 1366 xeon would run you about extra $200 but you wouldn't need a sas controller to start + plenty of room to grow.
2nd I would ditch the SASLP-MV8 in favor of lsi based AOC-USAS-L8i. The LSI chipset has much better driver support then the Marvell chip.
Click to expand...

That board looks great. Assuming it's more flexible about memory (which it appears to be) I'm going to go that way instead. Here's my current plan:


Supermicro X8ST3-F
Intel Xeon W3520
Kingston 12GB (3 x 4GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333
NORCO RPC-4224

I plan to add 4 more WD EARS 2tb drives to bring the RAID 6 up to 12 disks. For now, I'll run 8 of the drives on the SAS controller, and the other 4 using onboard SATA ports. My current SAT2-MV8 will go in a PCI slot and have a few individual drives connected for now, but nothing in an array. For the infrequently accessed data on those drives, I'm not worried about any potential bottlenecks, and I'll be using a HP SAS expander before too long either way.

And another question... will I see any benefits by using a faster drive like a Seagate Momentus XT for the OS drive? I'll be doing some virtualization so it seems like the SSD cache on that drive could help improve responsiveness significantly.
 
I may be biased, but I am not very impressed with that MB. It uses the X58 chipset instead of one of the server chipsets, like 3520 or 5520. I really like the 5520 chipset, it has excellent memory support -- ECC registered or unbuffered ECC or non-ECC, and few rank or bit restrictions.

http://www.supermicro.com/products/motherboard/QPI/5500/X8DT6-F.cfm?IPMI=Y&SAS=N

I've used the Supermicro X8DTE-F motherboard in a couple builds, and I am impressed. It is E-ATX size, but your chassis has room, and is a dual socket MB, but of course you can fill just one socket now and leave the second for expansion later. It has 4 x8 PCIe2, 1 x4 PCIe2, and 1 x4 PCIe1 slots, with no legacy PCI slots. If you really want embedded SAS, you can go with the X8DT6-F, but you lose one of the x8 PCIe2 slots (personally, I'd rather have the flexibility of the extra slot and choosing what SAS card I install). It also gives you a lot more room for future memory expansion.

http://www.newegg.com/Product/Produ...emory)-_-Kingston+Technology+Corp.-_-20134977

You can use Kingston 4GB 1333 ECC registered memory, KVR1333D3D4R9S/4G, which is actually on the tested memory list for that MB, and it is inexpensive. You can use 6 of those per CPU if you need to upgrade your RAM capacity.

http://www.newegg.com/Product/Product.aspx?Item=N82E16819117234

If you go with that board, you need a non-W CPU (the W- CPUs only have a single QPI, not suited for a dual socket MB). I like the E5620. Its clock is a little slower than the W3520, but it is Westmere (32nm) so it is lower power, and it has a larger L3 cache.

http://ark.intel.com/Compare.aspx?ids=47925,39718,

For your OS drive, have you considered an SSD? You could keep it lean and go with an 80GB Intel X25-M for $200. Or if you want more space, a 256GB Crucial C300 is about $550.
 
Last edited:
@John
Can you clarify why some boards/chipsets require x8 when others are fine with x4?

@Steve
I just recently had a thread which discussed x8 & x4 ram which may help you.
http://hardforum.com/showthread.php?t=1546311

Due to cost I went with a i3-530 so I don't have ECC support with my build. So far that build is working great.
 
dminja:

It has to do with the chipset and memory controller and how they were designed. Different variations of RAM need to be specifically handled. Some memory controllers were not designed to handle certain variations.
 
Last edited:
John, I would love to go with a dual socket board for this build but it's just not in the budget right now. That most definitely goes for the SSD OS drive as well. I'm ok with spending a little more for a faster OS drive to make the system feel more responsive, but for a home media server, that does a little transcoding, a SSD would be complete overkill. The x58 chipset may not be the ultimate server setup, but again this is primarily a home media server. Anything but the most basic server board is probably overkill here.

dminja, I initially planned on an i3 based server, but decided that due to the tremendous amounts of data that will have to be processed in case of an array rebuild, I'm not comfortable without ECC ram.
 
The X8ST3-F is much more flexible about the memory used that the X8SIL. You can use "regular" non registered DDR3 ram on it since it uses the x58 chipset.... And because it has a 1068E onboard, it's just about the perfect board for this application.
 
You must log in or register to reply here.
Back
Top