NAS4Free 36TB File Server Build

R

roach9

Limp Gawd
Joined
Jan 17, 2012
Messages
147
Last time around, I posted my prospective build. It fell through because when I went to buy my hard drives (from Futureshop), Seagate just pulled all their Barracudas with 5 year warranties, and they swapped them for the 1-year warranty 'Cudas. I wasn't buying that. Since then, I've had some luck and pieced together this unreal server, for a rather modest $2000 price tag.

This is for streaming media throughout my network. Primarily 1080p movies, 720p television, and FLAC music.
I am building this server with archiving in mind. I do not have a backup for this machine. It is too expensive and quite frankly, while 30TB of media seems like a lot, media is very replaceable. I am selling my 20TB file server in the coming days, but keeping the media on it. I do not get attached to my media, so a backup is not essential.

-------------------------------------------------------------------------

Modus operandi:
NAS4Free (http://www.nas4free.org/index.php/nas4free-home/)
This is a supported spinoff of FreeNAS 7. The guys who built the nighly-builds and kept the support going for FreeNAS 7 were asked to cease building under the FreeNAS name, as FreeNAS took off with version 8. The result is NAS4Free. While the community is small, I am choosing this operating system for a few reasons.
1. Stability. Been running the latest FreeNAS 7 nightly build for about 300 days now without any errors or downtime.
2. Updated ZFS. I love ZFS to begin with. NAS4Free offers the most recent ZFS versions.
3. Continued support.
4. Built on FreeBSD. I know that the new versions of FreeBSD have great driver support for all of my components.

Case:
Fractal Design Define XL (http://www./products/?sku=63338&promoid=1360)
63338_1.jpg

I have experience with this case. Love the space. Love the acoustic dampening material. The build quality of these are great. Not much to say about this choice. It makes the most sense for my build. I don't have experience with a server rack, nor do I have the space for one at the moment. Perhaps in the future I can transplant into one, but for now, I'd like to stick to the ubiquitous computer case.

Motherboard:
Supermicro MBD-X9SCL-F-O (http://www./products/?sku=61206)
61206_l.jpg

Lots of nice reviews on this board. Nice NICs, supports i3 & ECC RAM. Pretty easy choice. Nice feature set too; remote accessing of the BIOS is a nice feature. Truly headless motherboard.

Computer Processing Unit:
Intel Core i3 3220 3.3GHZ (http://www./products/?sku=75429)
75429_l.jpg

I know what you're thinking. You buy a Supermicro board with ECC... but your CPU doesn't even support ECC! But it does. I've read reports of users who have this processor and are running with ECC RAM. I think this will be suitable for a file server. It has low energy consumption and supports ECC. While a Xeon would be a nice, I consider it a luxury in this case and the i3 can do everything the Xeon can do, a little slower. But speed is not of major concern with this build. This is a newest i3. I think it will more than enough to support my server, but if anyone thinks I could do, easily, with a lesser model, please let me know as I could save a few dollars here.

RAM:
Kingston 8GB ECC Unbuffered RAM (http://www./products/?sku=62867)
62867_l.jpg

ZFS loves RAM, so perhaps 16GB would be nice here... but I think 8GB of ECC is enough for my needs. The speed difference at 8GB and 16GB of RAM is not something I've looked closely into though. I think 8GB should suffice, but if you really think I should just bite the bullet and spend the $60 on the extra 8GB, let me hear your opinion here.

SAS/SATA Controller:
IBM ServeRAID M1015 (http://ca.buy.com/PR/Product.aspx?sku=213027827)
213027827.jpg

I have already loaded up the IT firmware. These controllers have received a lot of praise from those who've used it. I've already purchased this piece, so this is a component I am sticking with. One of the main reasons I went with this over an LSI 1068 controller is that I aim to use 3TB+ hard drives. As far as I know, this controller and the motherboard will support 3TB+ drives... please correct me if I'm wrong here!

Hard Drives
Western Digital 3TB AV-GP (http://www.newegg.ca/Product/Product.aspx?Item=N82E16822136926)
Western-Digital-3TB-AV-GP.jpg

I know. WAY off the board here. Here is where I got lucky. I know a guy who does computer resales. His supplier gave him an overstock of these drives and I was able to pick up 12 of them for $118 a piece. All of them are in warranty until July 2015. Beats the 1 year warranty on most drives, and beats paying $200+ for the extra 2 years. They are designed to be more rugged than normal drives, so these should more than fit the bill in my server. Reviews aren't so hot around the net, but the one I trust most is here:
http://www.storagereview.com/western_digital_avgp_3tb_review
I plan on disabling Intellipark on these.

Power suppply:
Antec Truepower 650W (http://www.hookbag.ca/product/H3C06C1XE/)
original.jpg

I thought I had a 500W Antec that came with my Sonata, turns out I had this. Which is way nicer than the one I was speculating on and honestly, I was a bit uneasy powering such a nice system with a 500W generic Antec. This should definitely suffice.

----------------------------------------------------------------------------

I plan on running this off of an internal 8GB USB stick. Nothing special... everything is loaded onto the RAM post-boot anyway. I almost consider a RAM loaded OS safer than an SSD or HDD.

That's that! Thoughts? I'm trying to squeeze this in under $2200. Will be tough, but I'll see how close I can make it.



Price breakdown (in CDN $):
Hard drives: $1416

tl;dr - buying a bigass file server, the pictures are of the shit that's going inside it
 
Have you done any reading with the 4k sectors / 512 byte emulation WD green drives do, and how it affects ZFS?

Speaking from my own experience I was really unimpressed with my 5x1.5TB EARX green drive ZFS RAIDZ array with 8GB memory, with both freenas and solaris. The speeds just weren't there, though I admit you have a must stronger processor than I used.
 
I already have the core i3-3220 piece scheduled for Monday.

G2120 is also interesting, but for $30 more, HT is a nice-to-have.

Big issues: No VT-d (but you are not using that)

On the motherboard, I would go with the X9SCL+-F or X9SCM-iiF at this point. Both have dual 82574L ports which have fewer issues if you did ever want to change the platform.

One other thought: Think about snagging a E3-V1. At idle, not that much different but they are still fairly low power and can scale quite nicely. With the V2's, the V1's have dropped in price a lot.
 
I did go with the X9SCL -F... I won't be using those mobo features. This is just going to be plugged into one of my monitors upon first setup, unplugged, and tucked away.

And yeah, truthfully, I'm a bit worried about the WD drives... it's a risk that I'm willing to take though at this price point; the savings in HDs allowed me to really up the processor power, so hopefully that negates the issues with the AV-GPs.
 
As one who has built quite a number of XL based storage boxes for myself and friends, get yourself a couple of these, they will make your SATA power wiring neater and easier and the plug spacing is just about perfect for the XL.
 
Be careful with the MBD-X9SCL-F-O since it will not support IB CPUs OOTB depending on rev/bios version.
Even though you wont be pushing it that hard I would highly recommend you to get 16Gb RAM considering the storage size.
//Danne
 
I think I'll see how it goes with 8GB and go from there. Like I said, only going to be 4 users maximum on the server at all times. And that will be a rarity.
 
roach9 said:
Moreover, I see here:
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182251

That they support second generation i3... would that not be Ivy Bridge CPUs? I can't see a change log for the BIOS updates on Supermicro's site. Did you hear anywhere in particular that it won't work OOTB?
Click to expand...

No. The first generation of i3s were the Clarksdale based CPUs. The 2nd Generation are Sandy Bridge based CPUs. Ivy Bridge would be the 3rd Generation.
 
But the i3-3220 is an IB CPU so it also applies for you, I'd ask Supermicro since you probably need a newer bios than 2.0 since it was released much later than E3 v2 CPUs.
//Danne
 
Sorry but for a server, XEON is the only way. Buy the cheapest E3, you'll love it. I3 is a desktop CPU.

I run 16xWD Green 2TB on Opensolaris. Never had an issue with the drives, or the performance.
 
Feedback on the RAM - I have 40TB of storage in a similar ZFS build as you (around a Xeon E3 1230 SB). Have run it with 8GB and with 24GB - for simple home file storage write once/read many type loads, 8GB is enough. Do NOT run dedup.
 
I am running the i3 and it works perfectly. I get 500+ MB/sec internal on the IBM card, over 100 on GigE, and 6000-7000 MBs+ from RAM cache.

It also works with ECC ram, even with Rev 2.0a bios. 12 GB ram and I am sure 8GB is fine for a home server.

People who need 32 GB ram are running a huge server getting hits from workstations across a campus/building.
 
Just received everything.

My RAM isn't compatible with the board, so I'm going to have to return it and get something else.

FOR FUCK SAKES

It says right here that the RAM I chose should work. FFS

EDIT:
So I called Kingston who said that they support the motherboard.
Then I called NCIX and said that the board should support the RAM I chose, but it might not support the i3-3220.

So I call Supermicro, who confirms that the board does support Ivy Bridge i3, but not the i3-3220. So at this point I'm pissed off because I know that it's not officially supported but they have been shown to work.

And then I stumbled upon this gem on Supermicro's own fucking website:
http://www.supermicro.co.uk/support/faqs/faq.cfm?faq=14943

So it IS supported.
Now I'm sitting here with everything fucking unboxed and all support lines closed, with nothing working, and I fucking work 8-4 (support hour time) 'til Friday.

So I've been told by 4 parties that my shit isn't compatible, but every single piece of information on each of these vendor's websites suggest they ARE compatible. I'm diligent with my research, fuck me right?

So I'll update you guys later.

EDIT 2: Supermicro confirmed that my motherboard has the updated BIOS (checked via my serial number), so Ivy Bridge compatibility shouldn't be a problem. The Supermicro fellow said that it sounds like my board is shorted, because when I turn it on, the fans turn on for 3 seconds, then off, then on, then off... and he says it's probably because of the incompatible CPU. Fortunately I already have an RMA number from NCIX, so I'll be able to RMA this thing anyway.

I am livid right now.
 
Really brief update.

Built it like... 7 months ago?

155 days of uptime now... runs flawlessly. :)
 
How did you connect more than 2 breakouts of 4 SATA ports each to the 2 connectors on that IBM card? I noticed you have 12 drives. Did you order a 2nd card? SAS Expanders are the only way that I know of to go beyond 4 SATA connectors per mini-SAS. You did all software RAID anyways?

Very interested in your setup -- Thanks for posting.. I am a NAS4Free minion as well.. Loving the stability.. Planning a 12x3TB myself.
 
sparc500 said:
How did you connect more than 2 breakouts of 4 SATA ports each to the 2 connectors on that IBM card? I noticed you have 12 drives. Did you order a 2nd card? SAS Expanders are the only way that I know of to go beyond 4 SATA connectors per mini-SAS. You did all software RAID anyways?

Very interested in your setup -- Thanks for posting.. I am a NAS4Free minion as well.. Loving the stability.. Planning a 12x3TB myself.
Click to expand...

Most likely used the ports on the motherboard itself. 8 from the raid card and 4 from motherboard.
 
You must log in or register to reply here.
Back
Top