ZFS N00b: opinions needed

[Synthetickiller]

I'm planning a backup for my data that is cost effective. ZFS seems to be the safest file system since data integrity is the main focus. My network is wired 1Gbps, so throughput cannot exceed 125mb/sec even if read/writes are through the roof.

Purpose: Back up files and stream media

My current server will be running Windows Sever 2008 R2. I was thinking of running VMware ESXi and doing ZFS virtually since Hyper-V doesn't support VT-d and oracle isn't natively supported.

Here's my current hardware:
CPU: Xeon E3 1270
Motherboard: ASUS P8B-M
Ram: Let's just say at least 12gb (I can push 32gb+ so this should not be an issue)
HDD 1: 500gb Samsung F3 7200 for 2008 R2
HDD 2: 1TB F3 7200 for virtual machines (3 max, 1 windows VM to play with, one to host programs, 3rd would be ZFSguru, Freenas or something else for ZFS back up)
HDD 3 - 6: 5400 RPM Hitachi 3TB drives, Raid Z using onboard sata ports
Case: Lian Li V354r (holds 7 3.5" and 2 2.5" drives)


I really do not want to dedicate another build to this since cost skyrockets (more ram, another cpu, another board, psu, another hdd for OS, case, etc). My hardware obviously will have no issues handling the workload (it's basically an i7 2600 non-k).


What I don't know:
What disto to use? (FreeNas, ZFSguru, other?)
How to go about setting this (is my outline the way to go?)
Do you have a better idea? I'm all ears!

 

[fields_g]

Distro: Solaris based (Probably Solaris Express 11 or OpenIndiana)

http://www.napp-it.org/napp-it/all-in-one/index_en.html

Specifically: This PDF


I've played around with ZFS-on-Linux, without the ESXi Hypervisor, but I wouldn't recommend it. I haven't had problems, but the Solaris OS will be more stable for storage. Also remember that the new free version of VSphere 5 (ESXi 5) will be limited to 8gb of ram. This would be really bad for a storage setup. Embrace ESXi 4.x or find another host OS and use Virtualbox.

BTW... Haven't tested it yet, but Virtualbox 4.1 has experimental PCI passthrough on a Linux host.

 

[ukaudiophile]

Hello,

I'd check that the disk controller on that ASUS MoBo is known to work with Solaris / Illumos / Nexentastor / FreeNAS before going any further. ZFS can be choosy about which controllers it works well with. I'd also ensure the network chipset is fully supported, again that can kill your performance in no time. If in doubt, an Intel dual port card is always a worthwhile investment.

Other than that your setup looks good. I tried to do something similar with an old ASUS board myself, but eventually bit the bullet and went with a SuperMicro X8SIL-F with a pair of Supermicro AOC-USAS2-L8E controllers. Apart from some glitches initially with support for the chipset on the MoBo which caused the networking to vanish intermittently (fixed with a BIOS update) this setup has worked great so far and I'm very happy with it.

I am running Solaris 10 on bare metal, no ESXi layer, and I'd be tempted to go this way to avoid problems. Apart from the extra complication of configuring ESXi (though I know as long as the MoBo is on the VMWare HCL you're usually safe on this count) you also need to check that the boards chipset / BIOS has support for VT-D, as without this you can't passthrough the controller. Another issue might be that if you passthrough the on board controller (which from what I've read looks to be an all or nothing system, you can't pass a few ports through, it has to be all the ports on the controller) I wonder if that may cause a problem from ESXi's point of view regarding booting and running the hypervisor? I've never tried this myself but it could concievably be an issue.

With regards to which version of Solaris to run, I like the original (Solaris 10U9) but there is a bit of a learning curve if you are used to Linux, the learning curve from Windows is pretty steep. NAPP-IT has a big advantage in terms of it's GUI to get you up and running quickly, but it still needs some basic Solaris skills, whereas ZFS-GURU is still in it's early stages but looks like it is an excellent system and is supported heavily on this forum. Illumos is also worth looking at, I've not used it much but it's based on a stable OpenSolaris build and I've nothing negative about it's stability. I would avoid Solaris 11 Express right now, it uses ZFS V31 which nothing else can read or write to, whereas the other version use V28 so you have a variety of O/S's which can access a V28 pool, so if you start on one version and migrate, you can still import your pool into another version at a later stage.

I know it's not definitive, but hopefully this will help you on your way, if you have any further questions then please don't hesitate to ask.

Regards,

Dave

 

[Synthetickiller]

Thanks guys.

A few issues I'll cover.

My board is C204 chip based, so it supports VT-d. Also supports KVM over IP, but I need an E3 Xeon with built in GPU (1270 doesn't have it).

I could run Solaris bare metal and do virtual box to run Win 7/Server 2008 R2.

I'm a windows guy through and through. I justt want a solid set up that will be easy to set up. FreeBSD might be an option? Even virtually?

I wanted to keep 08' R2 as the main OS since I want to use this box for other things as well. Hardware costs for another rig are outrageous if I want any good excelleration for VMs and ECC ram is still expensive as heck.

I need to find out about the controller. I guess I'll get on that today.

Thanks so far!

 

[fields_g]

I would avoid Solaris 11 Express right now, it uses ZFS V31 which nothing else can read or write to, whereas the other version use V28 so you have a variety of O/S's which can access a V28 pool

Yes, be aware of the Solaris 11 uses a higher pool version by default. You can fix that though by "zpool create -o version=28 tank raidz <drive1> <drive2> <drive3> .... " or selecting the appropriate pool version in napp-it when creating.

Actually, when you are creating your pool, using the ZFS Guru Patched Live CD is pretty nice, then continue to use what ever OS you wish (as long at it supports 28 or later). The reason is that you can create a V28 pool with ashift 12. Ashift 12 will allow you to use 4k drives without performance drawbacks, or replace your 512 with 4k in the future without recreating the array. Only drawback is a loss on space efficiency, especially noticeable when storing lots of little files. The last block of a file might waste up to 4095 bytes as opposed to 511.