• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Building FreeNAS system, need help

S

securitux

n00b
Joined
Jan 31, 2012
Messages
44
Hi all, I am planning on building a FreeNAS system, out of some parts I currently have lying about or which I can get relatively "on the cheap".

CPU: Intel Q6600 (quad core)
RAM: 8GB
Motherboard: Asus P5K Premium
SSD Cache: Intel or other SSD, fast as possible
Storage Disks: 7 or 8 x 1 TB Seagates (7200.12), ZFS'd
FreeNAS OS Disks: 2 x 320GB Seagates (7200.11) mirrored
Controller: Areca ARC 1220, something else?
NIC's: Not sure, some dual or quad gig NIC's for link aggregation

Purposes:
- ESXi backup using VMware DR, Rsync, and / or presenting iSCSI for other backup (Windows)
- Primary filestore for media, music, etc.
- May eventually replicate to second FreeNAS server for snapshots

I am debating on upgrading the mobo, etc to something more recent but it's a NAS, it's not going to need a ton of hardware to perform (transfers are going across a network). So I will likely stick with this list unless someone suggests otherwise.

Power and case is not an issue. They are well spec'd, the power is 850W and I will happily add more if needed, case is a rack mount 3U from iStarUSA with 12 hot swap bays.

Here are my questions:

1) I plan on presenting JBOD on the Areca to FreeNAS so that ZFS can work the way it was meant to. Is this going to be an issue? I know ZFS likes dummy controllers, so I am not sure if JBOD on the Areca is ok or if I need to dumb it down further to an LSI. Note that I am not buying the Areca, I have 4 of them (an 1882i, 1680i, 1220 and 1210) and freeing up the 1220 isn't an issue, plus the ability to hot swap is vital. I don't really want to waste a 1680i on FreeNAS, it's more suited for another ESXi server. (I have 2 now). Reliability is key here, I don't want problems and I ask as I am not an expert on ZFS. Why I like ZFS is because it can handle large volumes and I can present this set of disks to any controller and it can still be readable. With Areca RAID if I want to switch controllers I am f'd unless it's another Areca.

2) Performance wise, I am concerned that FreeNAS might be slow for iSCSI etc. What sort of transfer rates can be expected off the NIC for iSCSI? CIFS? NFS? With 2 to 4 1G links in LAGG?

3) Any recommendation on an SSD for cache? Note I don't have a 6G controller in the mix there, is that going to be an issue? Should I toss one in for how cheap they are?

4) Any other thoughts on the proposed system? Does it look ok for what I need it for? I am not against upgrading the MB, CPU, RAM for something modern if that's what's needed, I just don't think that for something transmitting data over a network it's necessary. My biggest concern is of course transfer rate to multiple hosts so whatever buys me more of that, I will invest accordingly.

Thanks in advance.

-J
 
ZFS should be fine with a JBOD presentation as long as it sees N discrete disks. Curious why you are going with FreeNAS? Not common here (most here using some flavor of solaris/napp-it), so that might explain if you don't get lots of answers...
 
Areca JBOD is pretty darn good as a pass-through. I wouldn't worry about that actually. Believe it or not, you can swap the Areca JBOD drives onto a LSI controller like a 9211-8i and they will work fine.

One thing to think about is do you pay for power? That Q6600 is going to use a lot more power than some of today's lower-cost solutions like a Core i3. You could probably sell the old stuff, buy new for not much more but save a bunch on power.

On the speed side, if you want fast transfers between two or three servers, you might want to go 10GigE or Infiniband (I would steer away from IB unless you are fairly savvy and subnet manager does not scare you.)
 
Thanks for the replies so far...

danswartz said:
ZFS should be fine with a JBOD presentation as long as it sees N discrete disks. Curious why you are going with FreeNAS? Not common here (most here using some flavor of solaris/napp-it), so that might explain if you don't get lots of answers...
Click to expand...

If there's something better, I am all for it. I use FreeNAS as it seemed to be the best solution and easiest to set up for iSCSI and whatnot. If there's a better solution performance wise with support for a broad range of hardware, feel free to suggest :)

pjkenned said:
Areca JBOD is pretty darn good as a pass-through. I wouldn't worry about that actually. Believe it or not, you can swap the Areca JBOD drives onto a LSI controller like a 9211-8i and they will work fine.

One thing to think about is do you pay for power? That Q6600 is going to use a lot more power than some of today's lower-cost solutions like a Core i3. You could probably sell the old stuff, buy new for not much more but save a bunch on power.

On the speed side, if you want fast transfers between two or three servers, you might want to go 10GigE or Infiniband (I would steer away from IB unless you are fairly savvy and subnet manager does not scare you.)
Click to expand...

Yeah I sure do pay for power, a lot actually considering what i have in this house. A core i3 you say? Yeah, if that will save on power and do the job even better I'd go for it for sure. I'll browse around on what's available in that line, thanks!

As far as 10GigE, I could do that but I'd need to see what the cost would be of the adapter plus the appropriate switch. From what I recall it's pretty out there price wise. I'd want to put a 10G nic in my ESXi server, NAS, and connect both to my switch. So a switch with 2 or 4 10G uplinks would be needed.

-J
 
danswartz said:
OpenIndiana or Solaris 11 with napp-it web management ($0 cost).
Click to expand...

The only advantage I see of FreeNAS is the binary compatibility layer that gives you the infinite supply of Linux drivers and apps if you ever need something from that ecosystem for some reason.

Otherwise, yes, I'd go with Solaris to get the official ZFS implementation.

Solaris did used to have Linux binary compatibility but I think Sun abandoned that project many years ago.
 
nappit + solaris has bizarrely unintuitive samba/cifs support though.

Setting up permissions is mind-boggling. :(
 
True it is confusing. It works for me only because I go with the simple 'i have a username/password on solaris with same credentials as the windows box' so it 'just works'.
 
But in a environment with many connecting devices that have personalized passwords (that I cannot know) as well as changing IP addresses... FreeNAS might actually make more sense.
 
Just be sure to enable EIST on your Q6600 and it'll be more than fine for the job.
//Danne
 
Does it matter whether I use OI or Solaris Express 11 if I go with a Solaris OS rather than FreeNAS?

I assume OI and Solaris Express support Areca cards of course... :)

-J
 
You don't mention which version of freenas you are considering. I would definitely put some weight into the new 0.7.5/9S version based on freebsd 9 here: https://sourceforge.net/apps/phpbb/freenas/viewtopic.php?f=92&t=12138

I'm currently testing it with a simple raidz setup with pci-pasthrough in esxi 5, and next week i'll be evaluating it for an "all-in-one" build with a 12-drive raidz2 with similar requirements to yours, although i will be virtualizing it as well for an "all in one" setup. I've given OI + napp-it, and the brand new illumian a shot, as well as zfsguru and vanilla freebsd 9. I've exported and re-imported my pool without any trouble between all of these os's which is an encouraging start.

As of right now, barring any huge performance penalties that may arise i am leaning towards freenas 9S, it is very polished and easy to work with, and to me at least coming from a linux background freebsd is infinitely easier to work with than solaris. The biggest push for me to go the solaris route was the hope of a stable vmxnet3 driver for 10Gbit+ throughput between VM's, however it is proving to be unstable and throwing strange errors to console that i can't understand and google just returns more people (some on this forum) with the same issues and no resolutions. I'm not sure about you but i don't feel comfortable trusting a gui to handle all of my data, and with solaris that is what i feel i am doing with napp-it because it really feels like an alien system that requires an extensive amount of experience to thoroughly understand and debug when strange issues arise. If you are willing to put in the work to learn solaris from the ground up it seem's to be the ideal system to run ZFS on, however, if the freebsd 9 ZFS implementation proves to be stable that is the route i have personally decided to go.
 
I'm pretty sure OI and S11 have the same drivers - maybe not totally, but if the Areca HBA uses LSI drivers, most likely yes. Best is to pick one of the two OS and try it out...
 
Thanks for all the info so far everyone.

bleomycin said:
You don't mention which version of freenas you are considering. I would definitely put some weight into the new 0.7.5/9S version based on freebsd 9 here: https://sourceforge.net/apps/phpbb/freenas/viewtopic.php?f=92&t=12138

I'm currently testing it with a simple raidz setup with pci-pasthrough in esxi 5, and next week i'll be evaluating it for an "all-in-one" build with a 12-drive raidz2 with similar requirements to yours, although i will be virtualizing it as well for an "all in one" setup. I've given OI + napp-it, and the brand new illumian a shot, as well as zfsguru and vanilla freebsd 9. I've exported and re-imported my pool without any trouble between all of these os's which is an encouraging start.

As of right now, barring any huge performance penalties that may arise i am leaning towards freenas 9S, it is very polished and easy to work with, and to me at least coming from a linux background freebsd is infinitely easier to work with than solaris. The biggest push for me to go the solaris route was the hope of a stable vmxnet3 driver for 10Gbit+ throughput between VM's, however it is proving to be unstable and throwing strange errors to console that i can't understand and google just returns more people (some on this forum) with the same issues and no resolutions. I'm not sure about you but i don't feel comfortable trusting a gui to handle all of my data, and with solaris that is what i feel i am doing with napp-it because it really feels like an alien system that requires an extensive amount of experience to thoroughly understand and debug when strange issues arise. If you are willing to put in the work to learn solaris from the ground up it seem's to be the ideal system to run ZFS on, however, if the freebsd 9 ZFS implementation proves to be stable that is the route i have personally decided to go.
Click to expand...

That's quite interesting, I was going to go with FreeNAS 8, as it was the most recent version. I didn't know about FreeNAS 9... it's not on the FreeNAS site so I didn't hunt for it.

ESXi5 with passthrough sounds quite interesting, I just don't have the resources on the ESXi box to do it. It's a dual xeon 5520 with 32GB ram but it is also running 16 VM's some of which are things like Exchange and SQL Server. Load is an issue. I suppose I could build another ESXi server though, I'll have to think about that.

I will probably throw FreeNAS 9 and OI / napp-it on a VM and see which one I like or can deal with. I've worked with it quite a bit before with Solaris on Sun servers but if there are a lot of caveats and 'tricks' to be done on this version I don't know if I'll have time to deal with it.

I really didn't think this was going to be this complicated, lol.

-J
 
Well, I tried OI / napp-it. Wow, ugly. I followed the instructions to add a share and turn guest access on so that I could assign Windows permissions and although the GUI said guest was enabled, the command line said it wasn't. That's a pretty big signal to me that what happens in the GUI doesn't necessarily happen on the OS, and that's not acceptable. That, and setting up additional shares, etc is not an easy task. When I did have an error, a stupid orange box would pop up and say "oops" with no information as to why the error happened. Then if I did the same thing again just for fun, it wouldn't give an error. What happened? Why? Rather than dropping back to console to read the actual error message (which would have been more useful in the popup than "oops") I just stopped right there. Overall I just found napp-it to be a little weak and unpredictable. You just can't trust it. So command line is really the only way to work with this, and command line in Solaris is less than intuitive. It's downright overcomplicated actually, and not something I am willing to put the effort into at the moment. I am sure it works fine once you mess with it, but I just can't invest the days it would take to make it work.

Loaded up FreeNAS 7.5.0 or 9S or whatever it is, lol. I was a bit confused as to what exactly happened here. I guess it's just FreeNAS 7 with a 9 BSD backend. Good enough. My first question would be what is the improvement here? Is this primarily for the more advanced ZFS implementation? H/W support? Also, what happens when FreeNAS 9 (the real one) actually comes out? If I follow this branch, am I going to be able to migrate back over later?

Anyway I have FreeNAS 8 and 9S loaded up so I'll see which one works for me. I am leaning towards 9 as I am sure more H/W is supported in 9, and a more recent ZFS implementation is tempting. Plus the feedback I am seeing on 9S is VERY positive on the FreeNAS forums.

-J
 
Last edited:
Quick update, I did some more searching on NAS performance and OI / napp-it and it seems the performance is just that good (worth the painful interface). So I buckled down, fixed my issues, stopped my complaining :) and did some tests.

I set up 3 VM's with OI / napp-it (OI), FreeNAS 8, and FreeNAS 7.5.0 / 9S-9896 (hereon in known as 9S). I gave each one 4GB of RAM and a 100gig disk for testing. Each also got 2 CPU's (Intel Xeon 5520s, 2.27Ghz). All of this is on an ESXi 5 box so I am not expecting the same performance as if it were actual hardware, but let's see. Trying to compare apples to apples to see which one gets the hardware implementation.

One thing I noticed with SMB is that after a file transfer, if I tried to transfer another large file immediately it would not do any better than 34MB/s usually. I think this must be a ZFS thing which is where caching would help. Not sure though. Could be a number of things.

So far with SMB, this is what I've found.

FreeNAS 9S's SMB implementation crashes constantly. The transfer will suddenly stop and then a panic will show up in the logs. I can try to retransfer but I get the crash again. Small files aren't an issue but anything 200MB and above causes it to die. It affects SMB only, but still. I rechecked my settings, even rebooted the machine and it keeps doing the same thing. Really disappointing. Plus its performance on transfers from SMB are nothing special.

FreeNAS 8 was very stable, no issues. Transfer rates were a bit more random... sometimes I'd get 35-40MB/s, but most times it would be 50-65MB/s. Not too shabby! A few times I hit 70MB/s. I can deal with that for the moment.

OI blew my mind. 90-100MB/s! Sometimes it would hit 117MB/s... which is pretty much the theoretical max of a 1Gig link. Keeping in mind this is passing through a core and access switch with a 1G link between them, it's pretty damn impressive. It would never go under 70MB/s which is where FreeNAS 8 would peak.

I am not putting much into testing iSCSI, I sort of know who's going to win that one (OI), so my next test is to get OI on some hardware and try it out. I will likely run it for a few weeks and put it through the gambit of daily use just to see how it is to administer and deal with. I have 2 systems with identical hardware (Q6600, 4G RAM, onboard SATA 3Gig) so I can even run FreeNAS 8 and OI side by side and see what happens. The Areca controller can go to the winner :)

Eventually I plan on doing a proper NAS build with a Xeon and SSD's, but I might have to drop $4k on a server this month so that might be put on hold.

-J
 
I think Oracle provides a pre-built Solaris 11 VM... BUT it might be for VirtualBox only (I forget). I'd be curious to see how S11 stacks up to OI in terms of performance and configuration.

Those are some impressive SMB speeds and with it saturating the link it precludes any need to prefer NFS (unless there are CPU performance advantages).
 
@securitux On freenas 9S (9898) that I'm running, if I enabled SMB2 and AIO transfers together, it would crash and cause problems, but if I change it from SMB2 to NT1, AIO can remain enabled. In my testing though, SMB2 with AIO off performed the best.
 
securitux said:
FreeNAS 8 was very stable, no issues. Transfer rates were a bit more random... sometimes I'd get 35-40MB/s, but most times it would be 50-65MB/s. Not too shabby! A few times I hit 70MB/s. I can deal with that for the moment.

OI blew my mind. 90-100MB/s! Sometimes it would hit 117MB/s... which is pretty much the theoretical max of a 1Gig link. Keeping in mind this is passing through a core and access switch with a 1G link between them, it's pretty damn impressive. It would never go under 70MB/s which is where FreeNAS 8 would peak.

I am not putting much into testing iSCSI, I sort of know who's going to win that one (OI), so my next test is to get OI on some hardware and try it out. I will likely run it for a few weeks and put it through the gambit of daily use just to see how it is to administer and deal with. I have 2 systems with identical hardware (Q6600, 4G RAM, onboard SATA 3Gig) so I can even run FreeNAS 8 and OI side by side and see what happens. The Areca controller can go to the winner :)
Click to expand...

I'm at a similar crossroads. Currently my data resides on a windows server since at the time built it had the best SMB speeds. I only compared it to FreeNAS 7.3.

I have a second box I am trying to decide what to run on, currently running FreeNAS 8.0.3 with a ZFSR2 and I'm getting SMART errors, CHECKSUM mismatch, and the lackluster CIFS speed. Eventually I want to have 1 box with all 16x2tb drives under one OS (two ZFSR2 pools) each with a SSD cache

Keep us posted on your progress, it will certainly help me explore other implementations!
 
Just wanted to add that SMB2 crashing with AIO enabled is a known bug in freenas 9S, and also that i'm getting ~98-102MB/s sustained writes with it with 3 crappy WD green 1TB drives in a raidz without any tuning, just smb2 enabled AIO off everything else defaults (with esxi pass-through) also traversing 2 dell switches and a crummy d-link "green" switch that's definitely a pos.

Having said that, i personally believe OI is probably the best route either way, especially if you can get vmxnet3 working reliably, which is going to be my goal when the rest of my hardware arrives for a more serious build. I hope you continue to have success with OI + napp-it! Oh, one other side note, while you're in the testing phase you may want to take a look at nexentastor, it's free up to 18TB and is likely going to be the most polished of any option (although the web-ui is hideous lol). A lot of good people left oracle to work at nexenta.
 
danswartz said:
You dont need vmxnet3 - e1000 works fine.
Click to expand...

For gigabit lan you don't, but i'm not so sure when you're talking about an all-in-one setup where local VM's are writing to NFS shares via the virtual switch to drives capable of 600MB/s+ write? The best i've been able to get out of e1000 between VM's is 2-3Gbit/s with iperf :(
 
Sure, but that's not good enough? Honestly, I keep hearing the same complaints about vmxnet3 and it's been a couple of years and no indication it's going to be a non POS anytime soon.
 
In your FreeNAS 8, add this to your Axillary Parameters Field in your CIFS Services config:

Code: 
socket options = TCP_NODELAY IPTOS_LOWDELAY IPTOS_THROUGHPUT SO_RCVBUF=98304 SO_SNDBUF=98304

Also, make sure you have enabled Large RW support and AIO on the same page.

With a Broadcom NIC I get 110+MB/sec constant.

This is Xfering a 20gb File:
filetransfer2.png
 
@sstouffer, bleomycin: Interesting! I will try that. I still have the VM up and running so I'll reconfigure and test. I am pretty happy with OI and tbh a fork from FreeNAS with one person supporting it concerns me a little. I thought about Nexentastor, and I may give it a try. I'll have to see what time is like...

The vmxnet3 is something I definitely want to get working. I have 10GbE ports on my Dell 6224 that want to be used :) If they open up the gates for running ESXi VM's on a NAS (1Gig is WAY too slow) then this opens up the possibilities of vMotion and HA.

@leeleatherwood, I will do that as soon as I get home and try it out... funny how little settings like that can change performance so dramatically...

Thanks for the help so far, I will keep posting my progress.

-J
 
Ok, so I've built my temporary NAS.

OS: OpenIndiana 151a and napp-it 0.7g
Mobo: Asus P5K Premium
CPU: Intel Q6600 2.4Ghz
RAM: 8GB OCZ Platinum
NICs: 2 onboard and a dual Intel card (all detected)
HBA: Areca ARC-1220 8 port SATA in JBOD mode
Disks (zpool): 8 x Seagate 7200.12 1TB, RAID10 (4 mirrors)
Disks (rpool): 2 x Seagate 7200.12 500GB, mirrored
Case: iStarUSA D410-B10SA
P/S: Corsair HX750 (750W)

I am transferring my 2 TB of data to it now. As my files are on an ext4 2TB drive, I am copying them across from my Debian 6 box using a mounted NFS share. Speed? 960Mbit/s. CPU utilization? 10%. Basically it's asleep. RAM is about 6.5GB in use, so it's using what I have which is good. When I get everything over I am going to start teaming the NIC's to see what I can get.

So far so good! I am probably going to order a 10GbE adapter and SFP for the switch in the next month or so. I don't expect it to keep up with that, but I'll at least get more network throughput which is obviously my bottleneck at this point :)

If all goes well, I will duplicate the space in another machine upstairs in my office (maybe just a pair of 2TB disks) and do nightly syncs for DR purposes. Or more often, not sure. I have to do more research on the best way to replicate.

-J
 
i hate to resurrect an old thread but I'm eying a similar build

eventually here's what I want

24x3TB hdds - probably going to go with whatever has 7200rpm and a good price
areca-1220 with software raid using zfs (probably going to run openindiana and napp-it but I am worried about the unintuitive config for smb setup etc)
norco 24-bay case
ssd for the OS
750W psu - should be enough right?
32GB ram
xeon e3-1220
supermicro x9scm

questions:

get a dedicated gigabit ethernet card or use the integrated one?

what sort of writes/read speeds could I expect?
 
thanks for the heads up

I think the case I am opting for supports hotplugging harddrives - am I wrong?

also what are your guys' thoughts on iscsi (san) vs smb/cifs (nas)
 
You must log in or register to reply here.
Back
Top