Need a sanity check on a NAS build

Hi,

I made a thread about a possible NAS build a few weeks ago, got some opinions and realized I probably needed to do some more research. My NAS server would be purely for storage to serve mainly media and ISOs, since I already have a dual Xeon server that does all my application and encoding related stuff.

So I finally figured out the setup I want to go and picked out all my parts. I wanted to go with a somewhat cost effective setup, as I also have to upgrade my gaming rig as well *sigh* what happened to my wallet?

I've decided to give FreeNAS with ZFS a shot, it looks pretty highly recommended. For the RAID array I will be going RAID6/RAIDZ2 for a total of 12 TB disk space. I have been looking at 8-Bay encsolures for this. Here is what I am considering buying:

http://pcpartpicker.com/user/booyaah/saved/68snTW (I have some custom parts at the bottom)

- This build comes out to $770 for a diskless setup, which is pretty competitive with the pre-built NAS appliances that are in the same class as this. Most of them come with a 2.13 GHz Intel Atom with 1-2 GB RAM.
- I went with a slightly more expensive 450w PSU vs a 300w mainly cause you can't really find many small form factor PSUs with 80 Gold rating. I feel like Gold >> Bronze when it comes to the energy bill, and will pay for itself easily, considering this server will be on 24/7.
- I've considered ditching the aftermarket cooler and using the stock one to cut down on the cost since there's no real need to OC (although would that not make me [H]ard enough?)
- Since the case is Mini-ITX, most of those motherboards don't have 8-12 SATA ports. So I decided to go with the IBM ServeRaid card to use as an HBA. I've heard a lot of praise for these cards, I assume they are still a good choice?
- I was kind of worried the HBA or cooler wouldn't fit, but a Silverstone rep claims an 11" PCIe card and any cooler under 53mm tall will be fine clearance wise...so we'll see.
- FreeNAS comes on a live image that consumes the whole boot HDD right? I assume it's best just to get a cheap thumb drive to use as the boot drive then? (vs getting a small SSD)
- I think I already know the answer to this one, but do you think I actually need to buy some backup drives right off the bat? I have 3 TB of data I will need to transfer right away when the build is completed, I was thinking of waiting until I crossed the 4 TB threshold before buying additional drives. What is the likelyhood of 3 drives failing during a rebuild with RAID6?

So yeah, I was basically just posting this to make sure I wasn't doing anything that seemed strange or missing any parts before I placed my order.

Honorable mention to these cases and prebuilt NAS devices as I had considered going that route, but ultimately I enjoy building my own hardware out:

Cases:
http://www.u-nas.com/xcart/product.php?productid=17617
http://www.cooldrives.com/8saiihoduenw.html
http://www.amazon.com/Rosewill-2-5-Inch-3-5-Inch-Enclosure-RSV-S8/dp/B00552PLIE
http://www.amazon.com/Sans-Digital-TowerRAID-TR8M-Performance/dp/B004WNLP8W

Pre-Built:
http://www.amazon.com/ASUSTOR-2-13GHz-2eSATA-Server-AS-608T/dp/B009WBO0CC
http://www.amazon.com/QNAP-TS-869-PRO-8-Bay-SATA-6Gbps/dp/B007K9WEFI/ (I really loved the look on this one, I wish they sold the enclosure separately)
http://www.amazon.com/Synology-DiskStation-Diskless-Attached-DS1813/dp/B00CRB53CU/
http://www.amazon.com/Sans-Digital-MobileSTOR-MS28XP-5-Storage/dp/B0053PSLUI

Useful guides:
http://blog.brianmoses.net/2014/01/diy-nas-2014-edition.html
http://www.servethehome.com/difference-hardware-raid-hbas-software-raid/
http://brycv.com/blog/2012/sas-and-sata-hba-options-for-nas-and-servers/
http://forums.freenas.org/index.php?threads/confused-about-that-lsi-card-join-the-crowd.11901/
http://www.servethehome.com/ibm-serveraid-m1015-part-4/

Anyways, I guess once I get everything I'll do a build log or something with pictures. I've only seen one other review of a NAS build out using the Silverstone DS380B.

Drive failure imho is all luck. You can be safe and bump it up to a rainz3 if you're paranoid enough.

I've got 3 3tb drives in my raidz1 and it's been 6 months since I've built this rig. I'm starting to get worried since I'm dealing with post-flood drives. I'm still not confident that they've fixed the quick dying drive issue. I have more important things to take care of financially right now, so, adding more drives has been low on the list.

I'm running an old 120gb drive for my OS, 750gb for a scratch drive(download crap to it, that I'm just going to delete in a short time), then a raidz1 for long term storage. Monthly document backups go to the external hdd.

I wouldn't run an OS off a USB drive. I'd spend the $30 and get a 30gb ssd as your OS drive and put everything else on your array.

If you want a file storage system, just get something that runs off an Atom cpu. You don't need a beefy cpu to serve files. You will need 1gb of ram per tb of zfs if you want to run all of the special ZFS goodies. You can start out with these features turned off and add more ram as cash comes, but you can't go the other way.

http://www.newegg.com/Product/Product.aspx?Item=N82E16813157475

Check that out. That should give you an idea of what I think a nas should be ran off of.

Oh, and ZFS needs ECC ram if you want ALL the special features.

ECC doesn't magically enable any ZFS features. It's an extra level of safety which should be used on any file server where the data is important, regardless of which OS/file system is deployed.

Also, the 1GB RAM/1TB data thing needs to die. This is not a requirement. It'll give you more ARC hits, but that's it. If you enable things like deduplication (don't) you will seriously hurt performance without enough RAM, this is true. But no features will be unavailable if you do not meet this artificial, but often cited, RAM-to-data ratio "requirement"..

At least under Linux ZFS runs fine with 4 GB RAM. And especially for a media server more RAM does not much, as you rarely read a file multiple times, and if you do, it should be cached in client RAM anyway.

@pheonix991

I've seen that ASRock board with the Intel Avoton before. I kind of shy'd away from it because I've heard it's one of those looks good on paper, but is really picky with RAM. I can't imagine that FreeNAS would actually utilize that many cores anyways? I understand it's only 20w, but I'm not really a fan of passive cooling either. It feels like it's only selling point are having 12 SATA ports OOTB which I'll concede is nice.

I don't think ECC RAM is really necessary for a media server. ECC IMHO is really for databases that hold important financial or medical data.

Honestly though, why should I buy a 32 GB SSD only to put a 2 GB image file on it? I realize it's not much more $$$, but it seems like a waste more on principal.

@spazoid

Yeah, I figured I would probably be just fine with 8 GB RAM, but spending just a little bit more in case that 'theory' does have any merit, I don't have a problem with it. The FreeNAS hardware page states 8 GB is the minimum though.

And hell no I'm not going to use dedup either

-----

I was most curious if anyone has used that m1015 HBA card before and what their experience was.

Lots of people on this forum are using the M1015, myself included. It's the go-to HBA for a few good reasons:
It's well supported
It's cheap
It's simple (no BBU, no cache, no XOR processor)

Regarding ECC; it can save your data. You mention that spending a bit more money to increase your RAM is fine, but spending that extra money on ECC will give you an actual benefit instead.
The 1GB RAM/1TB data is not a theory, it's FUD.

For a media server, getting 8 GB ECC will be better than 16 GB non-ECC, in my opinion.

spazoid said:

Lots of people on this forum are using the M1015, myself included. It's the go-to HBA for a few good reasons:
It's well supported
It's cheap
It's simple (no BBU, no cache, no XOR processor)

Regarding ECC; it can save your data. You mention that spending a bit more money to increase your RAM is fine, but spending that extra money on ECC will give you an actual benefit instead.
The 1GB RAM/1TB data is not a theory, it's FUD.

For a media server, getting 8 GB ECC will be better than 16 GB non-ECC, in my opinion.

Click to expand...

If I went ECC RAM though, I would need to use a server board instead right? How about if I was to switch out to some entry level server components instead?

CPU: http://www.newegg.com/Product/Product.aspx?Item=N82E16819116907
Mobo: http://www.newegg.com/Product/Product.aspx?Item=N82E16813151295
RAM: http://www.newegg.com/Product/Product.aspx?Item=N82E16820148677

You don't need a quad core Xeon to support ECC ram, most of the dual core LGA1150 chips support ECC. The cheapest one I'm aware of is the Celeron G1840 which is $44 on amazon.

In my opinion, the only rational reason to ever forego ECC on a file-server is if you're reusing parts from an old desktop. Any time you can afford to buy new, you can afford to buy ECC.

Yeah, I'll probably just end up sticking with that $90 G3450 Pentium I had picked out already.

Intel needs to make some dual-core sub $100 Xeons...I guess that's basically a Pentium anyways

----

edit: Oh yeah, one other thing, a lot of my parts are cheapest at NCIX US. I've never actually ordered anything from that vendor before. Do they have good customer service for handling returns? I think I'd have to pay an extra $30 to use Amazon instead so...just wondering.

Booyaah said:

@pheonix991

I've seen that ASRock board with the Intel Avoton before. I kind of shy'd away from it because I've heard it's one of those looks good on paper, but is really picky with RAM. I can't imagine that FreeNAS would actually utilize that many cores anyways? I understand it's only 20w, but I'm not really a fan of passive cooling either. It feels like it's only selling point are having 12 SATA ports OOTB which I'll concede is nice.

I don't think ECC RAM is really necessary for a media server. ECC IMHO is really for databases that hold important financial or medical data.

Honestly though, why should I buy a 32 GB SSD only to put a 2 GB image file on it? I realize it's not much more $$$, but it seems like a waste more on principal.

@spazoid

Yeah, I figured I would probably be just fine with 8 GB RAM, but spending just a little bit more in case that 'theory' does have any merit, I don't have a problem with it. The FreeNAS hardware page states 8 GB is the minimum though.

And hell no I'm not going to use dedup either

-----

I was most curious if anyone has used that m1015 HBA card before and what their experience was.

Click to expand...

I agree with the not NEEDING ecc. It is overkill if you're storing stuff like movies and what not.

I'm not familiar with how much reading/writing that freenas does, but I'd rather no worry about a usb drive burning out on me. An SSD will last much much longer and you can re-use it later as a cache drive for your zfs.

I run a usb stick for my mining rig, and so does my buddy. He runs about 8 or 10 rigs and he has had 3 of them die on him in the past 6 months. They weren't even the super cheap ones either.

Up to you, but I don't like giving trust to a $15 usb drive when when I could spend $30 and get a half way decent SSD to run it instead.

Booyaah said:

edit: Oh yeah, one other thing, a lot of my parts are cheapest at NCIX US. I've never actually ordered anything from that vendor before. Do they have good customer service for handling returns? I think I'd have to pay an extra $30 to use Amazon instead so...just wondering.

Click to expand...

I've never ordered from NCIX US but a lot of our Canadian posters do order from NCIX.ca. At worst, they're like Newegg as far as I can tell.

With that said, did you look at shipping costs too? More than likely Amazon will be cheaper once you factor in shipping costs.

Yeah, I'll probably just load up my shopping cart with identical parts for Amazon and NCIX and see what the total is, and if there's like more than $20 difference try NCIX.

I've had a bunch of 'dumb luck' with flash memory, but I went ahead and threw in a small SSD.

Final build I'm going to go ahead with: http://pcpartpicker.com/user/booyaah/saved/RrCFf7

You could also just build your own server from a small computer.

Booyaah said:

Yeah, I'll probably just load up my shopping cart with identical parts for Amazon and NCIX and see what the total is, and if there's like more than $20 difference try NCIX.

Click to expand...

Amazon does have fantastic customer support.....

Booyaah said:

I've had a bunch of 'dumb luck' with flash memory, but I went ahead and threw in a small SSD.

Final build I'm going to go ahead with: http://pcpartpicker.com/user/booyaah/saved/RrCFf7

Click to expand...

I would go with this SSD instead:
$76 - Crucial MX100 128GB SSD

Yes it is twice the cost but you're getting:
- Significantly higher quality
- Significantly higher performance
- FOUR times the space

That's well worth the extra cost IMO.

Recommend getting enough drives for offline backup before a fancy raidz. Raid is not the same as a good backup!

Okay, well stuff just really changed today. I actually had ordered everything last night from several sites (didn't end up going NCIX after seeing they make you pay for shipping insurance). Then this afternoon I see the announcement about the 6 TB Reds.

So now I am thinking of going 3x 6 TB Reds in RAIDZ/5 and pick up a 6 TB Green to run scheduled backups against the main RAID array (I don't plan on my storage needs going over above 6 TB for at least the next 9-12 months, so I figure I'd just wait until later to buy more backups as my actual storage grows. Yes I realize these new drives are 14% higher cost per TB, but less drives means less chance to get a bad drive and more open bays for later expansion.

I'm kind of wondering if there's any point in spending the extra $$$ on RAIDZ2/6 if I have the array backed up...also trying to decide if I need a spare drive as a hot swap so I can start rebuilding right away when a drive fails. Maybe I'll just wait to buy a spare until my first disk fails. Reason being if I go 2 years without a failure, I could probably buy the equivalent spare for half the cost of what it is right now.

Less drives also means a less chance of relative data safety as well. The main reason to go with RAID 6 these days is that due the large size of RAID arrays, the actual rebuilding process is pretty stressful on the drives in the RAID array. As such, there's a strong possibility that your RAID array could fail while the array is rebuilding after a single hard drive failure.

In addition, these are new drives we're talking about. So their reliability haven't been established yet.

1TB drives went more than 100MB/s.

6TB drives don't go at more than 600MB/s.

So rebuilding times get longer and consequently riskier, it's not more stressful than before (not less either).