New ZFS NAS Build, would like some input.

TheBlueChanell

TheBlueChanell

Supreme [H]ardness
Joined
Apr 15, 2005
Messages
4,658
Hey guys I'm gonna be building a NAS and I'd like some input. Originally I was going to use Linux/ext4 but I am thinking I would like to get my ZFS feet wet. I also looked into QNAP and Synology and decided I would rather build my own solution. I've got no experience with ZFS but have plenty of experience with centralized storage solutions.

It's my understanding that RAID-Z/Z2 provides all the benefits of RAID5/6 but without the write hole and gives you the added features of ZFS.

I haven't quite decided on the size of the array I need yet but anywhere from 2TB-4TB usable should be sufficient.

What I'd like to accomplish:

1) Migrate my media (music, movies, pictures, etc) from a 1TB drive into the pool(s)
2) Migrate my VM's from their current local storage onto the NAS
3) Ability to reliably serve my media to my HTPC as well as run 3-6 VM's simultaneously

Hardware I'm looking at:

OS: FreeNAS 8.3
Case: BitFenix Prodigy / Fractal Node 304
CPU/Mobo: Asus C60M-I http://www.newegg.com/Product/Product.aspx?Item=N82E16813131843&Tpk=Asus ITX amd
RAM: 8GB ddr3 1866 1.5v Kit (Already Have)
PSU: Seasonic X400FL Fanless
NIC: Intel PRO 1000 in 2Gb LACP Trunk
Storage: a) 4x 1TB in RAID-Z
b) 4x 1TB in RAID-Z2
c) 3x 2TB in RAID-Z

I'm torn on what type of array I want to build. I've got 4x 1TB RE4's in my server that I could re-use and cut down costs. If I were to migrate the disks I'd install xen to an SSD. If I were to purchase new drives it'd be 2TB Reds, though I don't think I can really stomach shelling out for four 2TB drives. Other than the ability to sustain one additional drive failure does Z2 offer any performance benefits over Z?

Does that CPU have enough horsepower to handle streaming my media as well as function as the backend for my Hypervisors? In the future I plan to stand up a second xen server so I can configure HA.

Based on everything laid out above what do you guys feel about this setup? What changes or suggestions do you have?

Thanks in advance!
 
looks to me like you're spending money on pretty parts that aren't needed
for example, this PSU is half the price
http://www.newegg.com/Product/Product.aspx?Item=N82E16817151117

you can get larger cases for the same price or cheaper that will hold more drives for future expansion

no ECC RAM?

if you want something small and low power, consider the N40L although I don't think it or your planned system would have enough CPU power for "3-6 VM's simultaneously"
 
3-6 VM's with a total of 8GB RAM? What kind of VM's? RAM is dirt cheap, go buy some! ZFS will love you for it, and so will your VM's.
 
EnderW said:
looks to me like you're spending money on pretty parts that aren't needed
for example, this PSU is half the price
http://www.newegg.com/Product/Product.aspx?Item=N82E16817151117

you can get larger cases for the same price or cheaper that will hold more drives for future expansion

no ECC RAM?

if you want something small and low power, consider the N40L although I don't think it or your planned system would have enough CPU power for "3-6 VM's simultaneously"
Click to expand...

spazoid said:
3-6 VM's with a total of 8GB RAM? What kind of VM's? RAM is dirt cheap, go buy some! ZFS will love you for it, and so will your VM's.
Click to expand...

Thanks for the suggestions guys. I'm not sure the C-60 supports ECC which is why I left it as is, plus I've got a kit just sitting here.

I'd like to keep the footprint small and both the Prodigy and the Node 304 have 5-6 3.5" bays available. Platter density is increasing year over year I can't see myself needing more than 4-6 drives, if I was ever in need of more space in the future I could always go with 2-4TB drives. I'd just like to get this off the ground and running as cheaply as possible.

I opted for the Seasonic X400 as I like the idea of having a Fanless PSU and with a mini-ITX case it's also nice to have a fully modular PSU. I plan on individually sleeving and shortening the cables and that's much easier to do with a modular PSU. That's one component I don't mind paying a little extra for, but I will mull over the idea of getting a cheaper PSU.

Keep in mind that this is only acting as the backend for my Hypervisor(s). The front end server is the 1090T rig in my sig, which currently houses 3 VM's. I plan on upgrading that to an FX-8350/E3-1230 v2 and adding another 8-16GB of RAM. ZFS, if I understand correctly, is a software based solution similar to MD RAID in Linux so my main concern is the CPU overhead needed to push the data to my endpoints. I don't want it to choke if I end up stressing a few of the VM's at once while trying to watch a movie on my HTPC.

Those of you that are running FreeNAS/ZFS what kind of CPU are you running and what are you using your builds for? That's still my main point of contention, I'd like to stay in the mini-ITX form factor but if I were to opt for something like a ULV i3 or a Opteron 3200 that would bring my TCO up by quite a bit.
 
TheBlueChanell said:
Those of you that are running FreeNAS/ZFS what kind of CPU are you running and what are you using your builds for? That's still my main point of contention, I'd like to stay in the mini-ITX form factor but if I were to opt for something like a ULV i3 or a Opteron 3200 that would bring my TCO up by quite a bit.
Click to expand...

You are not running dedup, or compression so any processor is capable enough for streaming capabilities. However you should have ECC otherwise why do ZFS? I would start with AMD (since even their desktop processors can be had with ECC and IOMMU) and as you need more power consider stuff from Intel. If you need to keep the cost low and get ECC this is still cheaper to do with AMD.

That being said if you think for even a second that you are going to want a HBA then you'll definitely want enterprise grade and that means you'll be in Supermicro land. Once there Intel makes all of the sense in the world.
 
I just built a NAS using FreeNAS (installed on a thumb drive) and a similar motherboard. The C60M is a lower TDP CPU than the 350M and also slightly less powerful, but you should have no problems streaming data with that CPU. I didn't think that the ECC would make any difference for me as I am using my NAS strictly as an always on repo for the data I have stored somewhere else. I am taking the approach that I will buy newer drives every few years and put them into my main rig and move the older drives to my NAS. Your situation may depend on how much you care about your VMs. If they are simply playgrounds and you have copies of the vrtual disks you can easily restore from then ECC is probably not a big deal, however if they are something you really need to have operationally then ECC may be something to look into.

One quick edit to add is that you may see what kind of performance you get with the onboard NIC. I got fairly good iPerf results when I performed testing earlier this year, as my current drive situation is what is holding me back.
 
I've decided to rethink my setup based on some of the recommendations here. It's going to cost a bit more but I figure if I'm going to do this why not do it right.

I'm gonna forgo the C-60 setup and pickup the following:

https://www.superbiiz.com/detail.php?name=MB-X9SMIIB
https://www.superbiiz.com/detail.php?name=I3-3220BOX
http://www.newegg.com/Product/Product.aspx?Item=N82E16820139262

The Ivy Bridge i3's support ECC and going with server based hardware will make management and deployment much easier with IPMI, that was actually one of my hangups with going the C-60 route. I looked into some of the Atom based server boards but for a few dollars more I can get a full mATX board and a much beefier CPU so it seems like a no brainer at this point.

What do you guys think of the above?
 
I dont like the e350 for freenas, let alone a c60. I think you will be happier with something more beefy like a core i3.

Remember, most services like samba and DLNA are single threaded.

shame llano doesnt support ecc or that we dont have a dual core piledriver.

core i3 3220 is really the best option. performancer+ecc+55w tdp
 
N40L is good. Low idle power, with just enough CPU oomph to drive a fully loaded case of HDDs in ZFS without the CPU being the bottleneck. RAM is limited to 8GB.

Any other option is likely to be either even more underpowered or be more upgradeable but eat a little more idle power.

I chose the latter route. I'm using a Fujitsu MX130 S2 that I picked up for $140 as a baseline because I was too cheap to buy the N40L and wanted a higher RAM ceiling. (It actually costs even less than that, effectively, because I'm going to sell or give away the included 250GB HDD.)

Fujitsu MX130 S2 mobo has 6 SATA-II plugs, room for 4 HDDs, max CPU is anything up to 95W TDP Thubans, max RAM is 16GB.

I swapped in an Athlon II X4 610e and that's frankly overkill for my needs. I could have just kept the Sempron 145 that came with the Fujitsu and gotten the same performance.

Currently I have a RAID10-style setup (stripe + mirror) with 4x 3TB HDDs for ~5.7TB of usable space. I boot off a USB stick. I also swapped in 16GB of ECC RAM [email protected] CAS9. I'm using NAS4Free as that seems to be the real successor to FreeNAS. Read more here: http://en.wikipedia.org/wiki/FreeNAS#Development_Fork

Starting with FreeNAS 8, the developers rewrote the product. The newer versions of FreeNAS have much more stringent hardware requirements such as 4Gb of RAM to run ZFS. The older version of FreeNAS forked to a different project called NAS4Free, which maintains the minimal hardware requirements useful for older computers. The ISO files needed for creating bootable installation CDs for both incarnations reside as different projects in Sourceforge.

If you keep reading you will find that FreeNAS 8+ is not really the same as the older FreeNAS, despite the same name.

If I had to do it all over again, I might go Intel even though it costs a lot more for an Intel board that supports ECC... or maybe not. I did crunch the numbers and determined that the annual power savings from going Intel would be less than $10. At that rate, I'd have to keep this NAS around for a decade to make up for the cost difference. That's unlikely; I will probably upgrade-and-sell in the future when Intel system idle power is even lower than it is now.

Too bad the MX130 S2 is discontinued else I'd recommend that... throw in 8GB ECC RAM and install NAS4Free and you'd have a nice 4-bay NAS box for only ~$190. (And if you're adventurous you could rip out the mobo of the Fujitsu and use it in an ATX case with 6 bays, since the Fujitsu mobo has 6 SATA connectors. It's also compatible with standard PSUs, though you have to line things up correctly. And you could easily upgrade the CPU if you need more grunt to handle your VMs. The 960T and 1045T have been on sale for cheap and might fit the bill; else a lower-idling CPU like the Athlon II X4 610e or the Phenom II x4 905e may be enough.

Lastly, yes you want ECC for an OS that resides almost purely in memory (if booting off a USB stick). You've been warned.

Anyway good luck with your build!
 
Last edited:
Shadohh said:
I dont like the e350 for freenas, let alone a c60. I think you will be happier with something more beefy like a core i3.

Remember, most services like samba and DLNA are single threaded.

shame llano doesnt support ecc or that we dont have a dual core piledriver.

core i3 3220 is really the best option. performancer+ecc+55w tdp
Click to expand...

k cool, thanks for the tidbits. It would be nice if Trinity supported ECC as well as A85 has a bunch of SATA 6G ports, with C202/204 you are limited to two. I could see myself expanding with some SSD's in the future.

I was really hoping I could find an Opteron 3200 in retail as they would fit directly into AM3+ sockets. They've got two low voltage 4 core models and an 8 core, only one I could find is the 8 core on NE and it's deactivated.

3250HE: 4c 2.5ghz $99
3260HE: 4c 2.7ghz $125

http://www.newegg.com/Product/Product.aspx?Item=N82E16819113005
 
Last edited:
TheBlueChanell said:
k cool, thanks for the tidbits. It would be nice if Trinity supported ECC as well as A85 has a bunch of SATA 6G ports, with C202/204 you are limited to two. I could see myself expanding with some SSD's in the future.

I was really hoping I could find an Opteron 3200 in retail as they would fit directly into AM3+ sockets. They've got two low voltage 4 core models and an 8 core, only one I could find is the 8 core on NE and it's deactivated.

3250HE: 4c 2.5ghz $99
3260HE: 4c 2.7ghz $125

http://www.newegg.com/Product/Product.aspx?Item=N82E16819113005
Click to expand...

The things I would do to her! http://www.newegg.com/Product/Product.aspx?Item=N82E16819113036

3260he looks nice
 
My pfSense box runs on an i3 3220 and its perfect. Don't bother with ULV / Low TDP CPU's because unless the CPU is PEGGED it wont matter. When they idle, all of them go down to the same low power states, the only difference is the low TDP ones don't get 'un-corked' as much ;)

If you are serious about going with an Opteron, you may be able to find some deals on the DC forum. My ZFS box runs an i7 980 @ 4.2Ghz. I went with such a beefy CPU as I also fold on the box and wanted good performance from that. I started it out with 6GB of ram but now it has the full 24GB. ZFS uses all that ram to cache stuff and with 24GB pretty much everything I access frequently is in cache.
 
TheBlueChanell said:
k cool, thanks for the tidbits. It would be nice if Trinity supported ECC as well as A85 has a bunch of SATA 6G ports, with C202/204 you are limited to two. I could see myself expanding with some SSD's in the future.

I was really hoping I could find an Opteron 3200 in retail as they would fit directly into AM3+ sockets. They've got two low voltage 4 core models and an 8 core, only one I could find is the 8 core on NE and it's deactivated.

3250HE: 4c 2.5ghz $99
3260HE: 4c 2.7ghz $125

http://www.newegg.com/Product/Product.aspx?Item=N82E16819113005
Click to expand...

Why go to Opterons? I did but that was because I A) already had one and B) was using an HBA which required an enterprise grade MB. I believe all AMD processors minus Semprons, Trinity and Zacates support ECC.
 
kac77 said:
Why go to Opterons? I did but that was because I A) already had one and B) was using an HBA which required an enterprise grade MB. I believe all AMD processors minus Semprons, Trinity and Zacates support ECC.
Click to expand...

Sempron 145 for sure supports ECC. I think also 140. The earlier Semprons I don't know.
 
Shadohh said:
core i3 3220 is really the best option. performancer+ecc+55w tdp
Click to expand...

The new Core i3's ECC support is still clear as mud, apparently you have to use specific boards which offer true ECC support (server chipset) and even then there's little evidence I've seen so far that ECC actually works. Intel stripped ECC support from desktop CPU's long ago reserving it for Xeon's, and while some previous Core CPU's "supported" ECC it was only that they would work with ECC memory, it didn't actually check or correct rotten bits. The third gen Core i3's are still pretty confusing as to what extent ECC works. For my ZFS build last year I went the Xeon route to be sure ECC functions correctly as not using it (or being unsure that it is working) really introduces a weak link into a ZFS setup.
 
kac77 said:
Why go to Opterons? I did but that was because I A) already had one and B) was using an HBA which required an enterprise grade MB. I believe all AMD processors minus Semprons, Trinity and Zacates support ECC.
Click to expand...

I would only go Opteron if I could find a 3250/3260HE. They are AM3+ compatible and not C32/G34. I could use any of the boards with SB950 and have 6 or more SATA6.0 ports available to me, allowing me to make full use of SSD's should I add them to my build in the future. C202/204 still only offer 4 SATA3.0 and 2 SATA6.0 ports. The price point is also similar to the i3 3220 so I'd being paying roughly the same or cheaper as my current planned combo.

I thought about maybe using a Piledriver chip but If I'm moving away from an ITX solution I would like to stick with server hardware. Having IPMI and native Intel NIC's are huge plus and make the increased motherboard cost worth the premium for me.

I think based on everything you guys have been saying here and the research I've done I'm going to stick with my revision from earlier:

Define Mini or something similar
Tyan or Supermicro C204 mATX Motherboard
i3 3220
8-16gb of ECC Unbuffered memory

I could add an SSD or two to the array pretty easily as I have a few spare, would I benefit by doing so or will I not see any tangible performance increases with my workload?

I've been doing most of my reading through Wikipedia and random forums I stumbled upon from Google searches. Are there any sites available dedicated to ZFS where I could read up on different configurations and performance figures? There's some good threads here as well but I'm looking for some more info.

This may cause me to upgrade my server earlier than I expected too, :p.
 
tdg said:
The new Core i3's ECC support is still clear as mud, apparently you have to use specific boards which offer true ECC support (server chipset) and even then there's little evidence I've seen so far that ECC actually works. Intel stripped ECC support from desktop CPU's long ago reserving it for Xeon's, and while some previous Core CPU's "supported" ECC it was only that they would work with ECC memory, it didn't actually check or correct rotten bits. The third gen Core i3's are still pretty confusing as to what extent ECC works. For my ZFS build last year I went the Xeon route to be sure ECC functions correctly as not using it (or being unsure that it is working) really introduces a weak link into a ZFS setup.
Click to expand...

I'll be a guinea pug then. :p It clearly states on the Intel site that they support ECC so it's a shame it's such a gray area. If worst comes to worse, I can always sell the 3220 or recycle it into another system and pickup a Xeon.
 
Last edited:
You must log in or register to reply here.
Back
Top