Fast NAS 100+ MB/Sec Design Help

But if you don't really need super low power parts, Intel NICs, server quality, IPMI 2.0, KVM over IP, or ECC RAM support, and only plan on using one SATA controller, you might be fine with this:

The AMD chips support ECC ram if you can find a motherboard that does. The ASUS M4A79XTD EVO that I mentioned does, which is why I ended up with it.

It doesn't, however, support AMD-vi so no disk passthrough. Since he mentioned ESXi this might be something he wants. The 890fx chipset should support it, but I would verify that a particular board does before purchasing.
This board was mentioned in a build over in the virtualization forum: ASUS M4A89TD PRO/USB3
Looking at the ASUS website it should support ECC also.

For me it wasn't worth the added price of an Intel system to get IPMI and a better nic. I do wish there were more 890fx boards out when I built mine as ESXi would be fun to play with, but right now it's just not a priority.
 
I think I am going to go with the X9SCL-F mobo, E3-1240 cpu, and the kingston ECC ram. The norco cases are awesome but a bit too large for where I keep my systems. I would like to try a lang li tower case that can support 8+ 3.5 drives. If someone has some recommendation on what they used with good results, please pass the info.

Thanks

Danny Bui said:
If the Core i3 2100T does work with the X9SCL-F, then I recommend that route. RAM wise, Supermicro mobos tend to be a tad finicky when it comes to non-ECC RAM support. So I recommend this Kingston ECC RAM:
$80 - Kingston 2 x 4GB KVR1333D3E9SK2/8G ECC Unbuffered DDR3 1333 RAM

With the above RAM, the entire CPU + mobo + RAM setup is around $400 or so.
 
Last edited:
I've started the hardware orders and am going to do some lab test on my current hardware on the various setup options.

If I use this new server for an esxi 5.0 server, how does this impact the NAS design? Do I pass-through the M1015 to a FreeNas/Solaris VM, and then do the raid from there. On the VM, how much ram do you assign to still achieve high the end NAS performance?

Does anyone have ay decent setup guides for FreeNas, Solaris and ESXi when using it as a NAS device?
 
Make sure you get the X9SCL+-F (note the "+" in there) - the + means it has 2 82574L network adapters. The other boards have 1 82574L and 1 82579LM nic - the latter of which is newer and does not have drivers out for many OS's.

Since you are running ESXi on it now the E3-1240 is a good choice - go ahead and get 4 4Gb sticks of memory as well (the 8Gb non-registered sticks are to expensive to consider). I would leave the ZFS instance (whichever OS it is) from 8Gb of memory free and use the other 8 for whichever VM's you want.

Pass through of the M1015 is going to be the most performant option - and the motherboard & processor support it (Vt-d) so no problems there.

For a setup guide Gea has an excellent all-in-one pdf -
http://www.napp-it.org/napp-it/all-in-one/index_en.html
(note the pdf linked at the top of the page)

For cases if you don't want to go norco you might check out Fractal Design - their XL case can hold 10 internal 3.5 drives, gets solid reviews, and looks pretty decent (I have never used it before, but would be inclined to try it if I had a use case)
http://www.newegg.com/Product/Product.aspx?Item=N82E16811352005
 
Thanks ChrisBenn,

I did get the + version with the 2 82574L nics. So dedicate 8GB ram to the ZFS VM inside of ESXIi 5.0? I've been messing around with ZFS to get a feel for it.

What would most people recommend for Solaris? OpenIndiana or Solaris Express 11. For OpenIndiana, do I grab the server install or desktop ISO? I would assume server, then grab nap-itt to configure frmo there.

I will grab the PDF guide and go through that as well.
 
Last edited:
Thanks ChrisBenn,

I did get the + version with the 2 82574L nics. So dedicate 8GB ram to the ZFS VM inside of ESXIi 5.0? I've been messing around with ZFS to get a feel for it.

What would most people recommend for Solaris? OpenIndiana or Solaris Express 11. For OpenIndiana, do I grab the server install or desktop ISO? I would assume server, then grab nap-itt to configure frmo there.

I will grab the PDF guide and go through that as well.

for Solaris Express 11 get the Text install, if you ever need to in the future its easy to install the desktop manager and have it run only when required.

Dont know about OpenIndiana, didnt mess with it.
 
I was not even aware there was a text only install for Solaris 11 Express...if there is, I do not recommend you use that esp. if you are starting out. Having a gnome desktop (gui) uses bugger all extra resource and is very handy when setting up, administering and/or learning about the system. it also makes it easier to install and use apps like dropbox, crashplan etc which have a GUI interface.

I run a very similar setup and allocate 8GB for Solaris and the rest for vm's.

as for high end nas performance...stop stressing about this...and it's not about the ram anyway, it's about the disk setup/read/write demands... you are now limited by your Ethernet, not the nas., even with 4GB. if you want to run your vm's from here as well as multiple concurrent access, or large scale random writes/reads, that's another matter, but for your home use, you'll not have any issues maxing out Ethernet and having the fasted nas in your suburb!
 
I am mainly a windows guy, but have used linux a bit, but never solaris. I will take ur advice and go with the GUI version.

For networking, since Link Aggregation (LACP) it's not supported on ESX, how do you get around that for the Nas component, can you pass thorugh the dual nic and team it from the solaris vm?

lol, good point. I will try to stress less about the performance. Being new to ZFS, there is a lot to catch up on and figure out. Which for me is actually part of the fun. I am not doing anything major, on my vms. Mainly running AD,DNS/DHCP/RDS and about 4-8 XP/Win 7 VMs which I currently run most off of my Hyper-V setup. Even though its for home use, I have employee's use it as a portal for remote training.
 
Last edited:
I was not even aware there was a text only install for Solaris 11 Express...if there is, I do not recommend you use that esp. if you are starting out. Having a gnome desktop (gui) uses bugger all extra resource and is very handy when setting up, administering and/or learning about the system. it also makes it easier to install and use apps like dropbox, crashplan etc which have a GUI interface.

I run a very similar setup and allocate 8GB for Solaris and the rest for vm's.

as for high end nas performance...stop stressing about this...and it's not about the ram anyway, it's about the disk setup/read/write demands... you are now limited by your Ethernet, not the nas., even with 4GB. if you want to run your vm's from here as well as multiple concurrent access, or large scale random writes/reads, that's another matter, but for your home use, you'll not have any issues maxing out Ethernet and having the fasted nas in your suburb!

just a quick note but dropbox doesn't work on solaris (except with wine) but crashplan is a bit annoying to run without the gui (I do it be running the gui portion on my windows desktop, took some tweaking of the config files)
 
just a quick note but dropbox doesn't work on solaris (except with wine)

Ummm....no.

DropBox runs perfectly fine on my Solaris 11 Express box.

7skpb


Try the instructions here....http://forums.dropbox.com/topic.php?page=2&id=997
 
those instructions are using wine, no?

awe sh@t! :blush:

Apologies for that...my bad.

note to self : read before posting.


For networking, since Link Aggregation (LACP) it's not supported on ESX, how do you get around that for the Nas component, can you pass thorugh the dual nic and team it from the solaris vm?
i am yet to do this on my server,but yes,you can passthrough the nics and link agg in solaris. Be aware this does not improve the speed of single file transfers...and is only of use if you have multiple clients (ie. at a LAN party) or need nigh availability.
 
Last edited:
davros123 said:
i am yet to do this on my server,but yes,you can passthrough the nics and link agg in solaris. Be aware this does not improve the speed of single file transfers...and is only of use if you have multiple clients (ie. at a LAN party) or need nigh availability.

I see. I thought using the LACP on both the host and the switch you get a combined speed of 2 GBS?
 
awe sh@t! :blush:

Apologies for that...my bad.

note to self : read before posting.



i am yet to do this on my server,but yes,you can passthrough the nics and link agg in solaris. Be aware this does not improve the speed of single file transfers...and is only of use if you have multiple clients (ie. at a LAN party) or need nigh availability.

no worries, lets no derail the thread any further, haha
 
I see. I thought using the LACP on both the host and the switch you get a combined speed of 2 GBS?

Think of it like having multiple small straws. You can only pull 1GB/Sec of stuff through each straw, but if you have multiple "stuffs" to pull you can make use of multiple bandwidth.

Complicating this is the "stuffs" is defined by the LACP type - and this has to match on the systems and on the switch - typically you might have L2 - which gives each mac address a new "pipe", L3 which gives each ip address a new "pipe", or l4 which gives each port a new pipe.

On a file copy you are typically going to be using the same ip, mac, and port so no joy (though theoretically a client could be written which used multiple ip's, etc).

Practically you would have to have l4 LACP (port based), and copy some files via FTP, some files via SMB, some via NFS, etc. - so not so convienent.

Now if you have a server with lots of distinct clients connecting to it then it works well - each client essentially gets a 1Gb channel (within reason - you still only have so many channels to split amongst the clients - basically if you have 100 clients and 4 channels it's going to do a pretty good job of giving you 25 clients per channel)
 
In terms of best practice when using building a NAS using ESXi 5.0, how do you setup the networking environment? If I have 4 network ports on the server, do I:

2 Nics for ESX (1 for Service Console and 1 for VMs)
2 Nics pass-through to Solaris 11 VM then teamed for LACP connected to LAG on switch

Does this sound right or is there a better approach? How have you guys setup your ESX environement when using it for a NAS device and for hosting about 6-10 VMs.
 
I finally have the Solaris NAS up and running. I am using the following config:

- Supermicro X9SCL+-F
- IBM M1015 with IT Firmware (P11)
- Fractal R3 Case
- 6 x 2TB HITACHI Deskstar 5K300
- 16 GB ECC Ram
- Xeon e3-1270 CPU
- ESXi 5.0
- OpenIndiana 151 VM with 8 GB ram
- using ZFS Raidz1

Results:

I am getting pretty much 100 MB/Sec on both read and writes. I am very pleased with performance so far. How do you handle a failed drive or if you install a different version/flavor of Solaris. Can you re-import the Pool or how does that work. Also with back up solutions, using snap shots. How is that used exactly?
 
i would go with the LSI 9260-8i ( or 16i if you need more ports). Its one of the faster raid cards on the market right now. I've tested many raid cards and this one stands out. I've built 15-20 servers around it.
 
Back
Top