• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

NAS build - ZFS performance questions

B

Bun-Bun

Gawd
Joined
Sep 11, 2003
Messages
555
Building NAS case myself. Trying to keep this as small and low power as possible but still powerful enough for my needs.

Here is my starting point hardware wise.

ASUS P8H77-I H77 mITX mobo
Xeon ES 1230 CPU (possibly undervolt/downclock)
16gb (2x8gb) DDR3 ram
6x 2TB LP drives
Intel NetEffect Net Effect NEO20-CX4 10GB Ethernet Cluster Server Adapter PCI-E (with matching on my desktop)

my plan is to connect this to my desktop via an iSCSI lun. the purpose is to take internal storage and make it external as well as safer at the same time (currently running 2x2TB in Raid-0 for bulk storage). I want to keep the same performance level that I currently have with my two 2TB drives (100mb/s write seq. and 230mb/s read seq. and comparable iops).

1. First off is this level of performance possible with the above level of hardware? Or am I going to need to scrap that idea and use this as a streaming/backup device?

2. If the performance is possible is the above CPU required or could I go with a cheaper i3?

3. If the above performance is not possible would an i3 suffice for a streaming/backup box to maintain greater than 60mb/s throughput?

Basically I am unsure what the CPU needs are of ZFS. My google-fu is leaving me more confused and all I know is to have as much ram as possible. Which is why I went with a 1155 based mITX board so I could go 16gb of RAM. And if I am expecting way too much out of the hardware with the iSCSI lun then I will focus more on a normal NAS and build another device for my desktop or go direct attached with an expander.
 
Last edited:
You're spending more on the network adapter than almost anything else.... seems like the wrong way to go about your problem.

If you can live with gigabit ethernet speed (~100mb/s) you can do what you're trying to do for much less money.

You could try and get LAG working, too and multiplex gigabit ethernet ports.

The xeon should be as fast as you can get it, but there are often other factors that may limit ZFS performance including 4k sector drives, HBA bandwidth, NIC type, etc etc.

You're going to have to experiment and find out. i did something like this with an i3 processor and gigabit ethernet, and 6 hitachi 2TB drives, and ZFS. I don't saturate a single gigabit link, but its good enough to do backups (60-80mb/s)
 
Gigabit speeds will not suffice. If I can not obtain the same performance or better as my internal raid-0 then I will stick with my raid-0 or look to some other solution.

The 10GbE nic's I have in mind I can get for $80 shipped used plus a $20 cable.

What was the bottleneck in your setup? Was it the processor? the nic (I assume as a network share and not iSCSI lun)? or the raid config?
 
I never spent the time to figure it out, It just needed to do backup over FTP, and the speeds are sufficient.

Everything I read points it to FreeNAS not having a "high performance" rendition of ZFS.

People that use it that need to exceed gigabit use things like solaris vanila, nexentia enterprise or OI, and also use things like L2arc and ZIL caches. FreeNAS I believe has a port of ZFS, its not native to the operating system.

I think you should think about DAS (Internal SATA/SAS or E-SATA/E-SAS) with RAID with parity or mirror+stripe.

Backup your goodies to a ZFS server over gigabit ethernet, where performance doesn't matter.

You might also look into 10GB infiniband cards, they are 1/10th the cost of your intel NIC's.

I think you're going to have to buy everything, and then tune from there, its going to be hard to get a guaranteed speed when there is so much that isn't known about ZFS. What you have on paper looks great in theory but my gut tells me you're either going to have to tinker with it a lot or buy something else to get the performance to where you want it to be.

You might dig up a bit more here: http://www.zfsbuild.com, and read up some on infiniband here http://davidhunt.ie/wp/?p=232

Also, can you link us to these magical 10 gigabit ethernet adapters? Usually they are $800-1000 from what I know.
 
Last edited:
Thanks for the links! Got some reading to do but looks exactly like what I want to do just on a bit smaller scale.

And the nic's I found on ebay. RJ45 nics are expensive but the ones using sas CX4 cables are cheap on ebay.
 
That 1230 does not have video support, need the 1235 for a Xeon with video. How ever I think an i3 would work fine.
 
I actually have that same motherboard or very simliar, ASUS 1.6ghz amd dual core with 8gb mem, 6 SATA 6 ports.

Using FreeNAS 8 as well as OI and some other distros, I never had good luck, the ashift stuff never really helped the relatively poor performance I had with WD Green Drives.

With WD Green Drives I had pretty poor performance (30-50mb/s)? over ftp, even with an intel NIC, so I scrapped it, used the green drives with a disk pooling software in windows, and eventually invested in the 2tb hitachi disks, a better HBA, and a motherboard with ECC. Much happier now.
 
Rody said:
That 1230 does not have video support, need the 1235 for a Xeon with video. How ever I think an i3 would work fine.
Click to expand...

Video not needed for a NAS. I can use a PCIe vid card for setup and the rest SSH into the unit or use web interface of freenas.

hotcrandel said:
I actually have that same motherboard or very simliar, ASUS 1.6ghz amd dual core with 8gb mem, 6 SATA 6 ports.

Using FreeNAS 8 as well as OI and some other distros, I never had good luck, the ashift stuff never really helped the relatively poor performance I had with WD Green Drives.

With WD Green Drives I had pretty poor performance (30-50mb/s)? over ftp, even with an intel NIC, so I scrapped it, used the green drives with a disk pooling software in windows, and eventually invested in the 2tb hitachi disks, a better HBA, and a motherboard with ECC. Much happier now.
Click to expand...

Think your issue was with the drives then? Or was it the CPU/mobo holding back?
 
hotcrandel said:
I actually have that same motherboard or very simliar, ASUS 1.6ghz amd dual core with 8gb mem, 6 SATA 6 ports.

Using FreeNAS 8 as well as OI and some other distros, I never had good luck, the ashift stuff never really helped the relatively poor performance I had with WD Green Drives.

With WD Green Drives I had pretty poor performance (30-50mb/s)? over ftp, even with an intel NIC, so I scrapped it, used the green drives with a disk pooling software in windows, and eventually invested in the 2tb hitachi disks, a better HBA, and a motherboard with ECC. Much happier now.
Click to expand...

Sounds like a configuration issue. I have 8x2TB WD Green chucking out an avg of 630MB/s read and 500MB/s write in mdadm raid6...on a cheap FM2 socket dual core AMD with 8gb RAM. Serving the LAN via SMB users with realtek onboard nics were still getting 100MB/s +/- 5MB pending the type of file.
 
ziggo0 said:
Sounds like a configuration issue. I have 8x2TB WD Green chucking out an avg of 630MB/s read and 500MB/s write in mdadm raid6...on a cheap FM2 socket dual core AMD with 8gb RAM. Serving the LAN via SMB users with realtek onboard nics were still getting 100MB/s +/- 5MB pending the type of file.
Click to expand...

Wow thanks for that info. I was trying to decide between AM3+ and FM2 for building a Linux raid6 setup. Wasn't sure if it would be compute bound or what. I was thinking 6 x 2TB WD Red drives. You're using all 8 sata port for the raid drives -- where's the boot partition?
 
ziggo0 said:
Sounds like a configuration issue. I have 8x2TB WD Green chucking out an avg of 630MB/s read and 500MB/s write in mdadm raid6...on a cheap FM2 socket dual core AMD with 8gb RAM. Serving the LAN via SMB users with realtek onboard nics were still getting 100MB/s +/- 5MB pending the type of file.
Click to expand...

I was talking to someone else and they also were easily saturating a 1GbE with the fusion board I linked.

So for my little NAS I bought the C-60 fusion board and a intel pro 1000 ct NIC and will populate it with 6 2TB drives.

My SAN project will have to wait and require more research.
 
I have a ZFS server running OI on an Asus M4A79XTD EVO, Athlon II x4 2.9, 4gb ecc ram with 3x 2tb Samsung F4's in a Raidz1.
I see 90-100MB/s easily on large files with Realtek nics on both ends. I think there are many things you should be worried about before CPU performance. Granted, I don't use dedup or compression.

For those that aren't saturating a gigabit link, I would think it's either a protocol issue or disk bottleneck.
 
The NAS in my sig can read at ~900MB/s and write at ~650MB/s sustained using a 1TB file as a test.

I am currently limited only by the gigabit, I plan to upgrade to 10GbE or Inifiniband one of these days
 
Ok good so my setup should work for what I want.

I roughed out my case design

nasn.png
 
wd reds are 15.5watt each start-up. but even worst case scenario 22watt per drive is 132watts plus the 24 watts the board/cpu/nic take on start up (measured myself as I got the board/nic/ram yesterday). So worst case power is under the pico's rating with peak still left to spare.
 
Bun-Bun said:
wd reds are 15.5watt each start-up. but even worst case scenario 22watt per drive is 132watts plus the 24 watts the board/cpu/nic take on start up (measured myself as I got the board/nic/ram yesterday). So worst case power is under the pico's rating with peak still left to spare.
Click to expand...

Nice, will be a sweet little NAS then.
 
You must log in or register to reply here.
Back
Top