Sandy Server Build Critique

]|[ Mar']['in ]|[

Supreme [H]ardness
Joined
Sep 20, 2002
Messages
4,663
So I'm looking to build a linux file server with software raid. The end goal would be to have it be able to support maybe 10x2TB drives in raid 6 + another drive for the OS. This computer would not be expected to do anything other than sit and keep my files to stream to the other computers around the house.

This is what I've picked out so far.

Processor: Intel Pentium G620 Sandy Bridge 2.6GHz LGA 1155 65W Dual-Core ($75)
Motherboard: ASUS P8B-E/4L ($250) I could then get a Asus Pike Card ($120) for the rest of my sata ports.
Memory: Crucial 8GB (2 x 4GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 ($120) I could then get 8 more down the road if necessary.
OS Drive: Seagate Barracuda 250GB 7200 RPM ($37)
Raid Drives: 7xHITACHI Deskstar 5K3000 2tb ($400) + 3x2tb Samsung Drives I already have.
PSU: CORSAIR Builder Series CX600 V2 600W Is this enough power? Too Much?
Case: 3xExternal 5.25" 1xExternal 3.5" 9xInternal 3.5" bays I really don't care what kind of case I have (rackmount or tower) as long as it will fit everything I need inside.

I am in no way attached to any of these parts, I've gone through many iterations of this (some intel some amd) and figured it was finally time to post and get a critique.

Anyways thanks for the help in advance.
 
HW ooks okay I guess. I would recommend using OI+napp-it rather than linux. ZFS is much nicer than anything you can get with linux right now.
 
HW ooks okay I guess. I would recommend using OI+napp-it rather than linux. ZFS is much nicer than anything you can get with linux right now.
the main reason I'm leaning toward linux is because I have several friends that are quite linux savvy so it would just feel safer in case something went wrong.

On the other hand if I went with zfs not only do I know nothing about the os I would have to run it in, I would know nothing about the software. So if something went wrong, it would just be a huge hassle it seems like. If I'm wrong, I'm totally open to using something different I guess.


I don't think that CPU supports ECC memory, get a Xeon.
I was also unsure about that and googling didn't turn much up for me.
 
Well, your call. Generally, a NAS/SAN doesn't need much (if any tweaking), so I think that's not a big concern. I had zero solaris experience too - I'm far from an expert but know enough to do what little CLI stuff I need. ZFS is a huge win, particularly for raid configs...
 
I was more so comparing to ext4, but I guess you are sort of right on checksums, but any decent raid controller should have decent error correction built in, why add the cpu overhead
 
I have no idea what enterprise level raid cards do for error detection, but cheap ones? And software raid? When you say 'should', are you speaking from experience, or how you think it should work? As far as the cpu overhead, unless you are running on a pentium II, that is noise... Note also that the error detection in drives themselves is not that great (I was reading an article awhile ago that with large drive sizes and 7x24 access, the chances of an undetected error are not trivial.) ZFS does a 256 bit hash, which is going to be a lot more reliable.
 
I was more so comparing to ext4, but I guess you are sort of right on checksums, but any decent raid controller should have decent error correction built in, why add the cpu overhead

Well, if you are running raid-5 you can't recover - how do you know if your main data or your parity data is corrupted. You just know you have a mismatch, not what it is. (ZFS uses the checksum to determine which set of data has the "error").

With raid-6 hardware cards have effectively 3 sets of data, so you could determine which is most likely to be the correct match - and you could repair using that one as your template. Note that there are cases where certain patterns of corruption can not definitely say what is correct, but only give you good odds as to which the correct data set is
reference: http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf

So basically fixing single disk corruptions issue are assuming there isn't any extended corruption (bad cable, controller issues, etc) - so you are "guessing" that you are fixing the data correctly.

With ZFS and block level checksums you *know* if your data is correct or not.

The other issue with HW raid is that the controllers do it differently - what you get with an expensive Areca controller is going to be miles ahead of what you get with a random SI controller. But the actualy bits and details of the implementation are specific to the firmware on the controller, so you need to evaluate raid-6 on a specific controller, not just raid-6 in general.

And then there's the whole storage tiering issue (SSD caches for reads and writes, etc.).

It's a different picture when you, well, look at the entire picture instead of just a small portion.
 
Actually, it's more insidious than that. If you get bit by the raid5 write hole, you may never find out until it is too late. Scenario: in the middle of a stripe write, you crash or lose power. The data drives were written to, but not the parity info. Until and unless you read that block, you have no idea. Now, a data drive fails and you replace it, and start an array rebuild. You now find out that one block is unusable. Oops. What saves us here is not the end to end checksumming ZFS does, but that it is copy on write, so you are never rewriting data, always writing it somewhere new. This is also why ZFS doesn't need a filesystem check or journal in case of crashes.
 
I was more so comparing to ext4, but I guess you are sort of right on checksums, but any decent raid controller should have decent error correction built in, why add the cpu overhead

They do not have any more error correction then linux software raid. And the overhead for linux software raid is less than 12% CPU usage on a single core.
 
how does a UPS keep you from crashing?

It does not, but linux generally does not crash often. At home and at work I am seeing less than 1 crash per 5 years of uptime. This is nothing I would design my systems around to prevent..
 
That's certainly your call. I have had a raid controller die on me though. Rather not have that silently corrupt data before it goes. Anyway, not sure if this really classifies as 'design my systems around' - linux equivalent of ZFS (as close as you can get) right now is md+lvm+extN, which is fairly common, in my experience. I'd just rather start from something that doesn't expose me to silent data corruption, as unlikely as that might seem.
 
I have had a raid controller die on me though

HW raid is in my opinion less reliable than linux software raid. In the it industry I have seen more than a handful of times where HW raid controllers died wiping out entire arrays.

As for ZFS, I would use more (I do test zfs and btrfs for years now) it if the options for OS were anywhere near as good as it is on linux. I am watching both btrfs and zfs under linux however I would not consider either as stable as ext4 at this moment in the linux operating system.
 
For linux, I certainly agree. My main interest at this point is for use in a SAN/NAS, since my linux/freebsd servers are virtualized, so their filesystem choice isn't so important (IMO of course.) I wish some flavor of opensolaris wasn't my only choice, then again, once the SAN/NAS is set up, there isn't that much I need to do on the CLI.
 
Martin:

What's with the ASCII fleur-de-lis? Are you French? :D

I'd look at a Supermicro server board with IPMI (-F in the part number) since that is quite useful for a networked server. Some Asus server boards can also do IPMI, but I prefer Supermicro.

I have a linux file server that has been running well with three IBM ServerRAID M1015 HBAs in pass-thru mode. I got the M1015's from Ebay for $75/ea, and flashed them with LSI 9240-8i firmware. They are SAS2 (6 Gbps) with 8 ports each, and work fine with mdadm (linux software RAID) or individual disks, and smartmontools work also.

If you want something new, there is the LSI 9211-8i, which is similar to the M1015 in that it has 8 ports and SAS2, but the 9211 uses the IT mode firmware and the mpt2sas driver, while the M1015 uses the megaraid_sas driver.

If you got one 8-port HBA, then a motherboard with 4 or 6 SATA ports, you would have met your 11 ports requirement.

For PSUs, I recommend an 80+ GOLD unit. I'm partial to the Seasonic X series (Corsair also rebrands them as AX). You can get a Seasonic SS-560KM 560W for about $125. If you are looking for something cheaper, there is the FSP Aurum series, but I have never used one myself. You can get one for less than $100.

That case you listed is really cheap. Have you considered whether you want hot-swap? If so, you'll have to spend more, but have a look at the Norco cases.

If this is going to be a media server, by which I mean a lot of files that are written once and then just read (not modified in place), you may want to consider snapshot RAID. FlexRAID is available for linux, but there is a new snapshot RAID program called SnapRAID that looks good, and unlike FlexRAID, it is open source. I've just started testing it:

http://snapraid.sourceforge.net/
 
Back
Top