ESXi, ZFS, VT-d, questions.

I'm preparing to make the transition to ZFS very soon and I'm still mulling over how I would like to approach this. Ideally I would like to host my ZFS server within ESXi using PCI-passthrough of my H310/9211 HBA.

I just rebuilt my server with a xeon 1246 V3 and a supermicro X10SLL-F. Currently loaded wiht 16GB of crucial ECC.

On board SATA ports are currently unused
Have H310 card flashed to 9211-8i IT mode hooked up to 4x 3TB Red Drives to be my main ZFS volume. Remaining capacity for 4 more drives to add another ZFS volume later if I so choose. (I know I neeed to make my firmware version match the driver version in FreeNAS if I go that route)
Intel PRO/1000 VT Quad port NIC -- Not recognized by windows. But Freenas and ESXi pick it up alright. I have a Pro/1000 PT quad port as a backup.
9260-8I Raid card with BBU -- Was going to use this to setup a hardware RAID array for the VM's reside in ESXI

System is built and loaded with ESXI 6.0. Made 2 full passes of Memtest86+ without error.

As I said, I would like to make my ZFS server a VM. As far as other VM's, the system won't be loaded very hard. So I think I may have enough resources. The system will spent more hours idle than under load from user demand.

Here's my plan:


I know the general advice is to not run your ZFS server under VM. But, I think my system meets safe requirements for doing so. If not, I've got backups ready Nothing about this system is horribly mission critical. I have the important stuff backed up. The loss of service for a few days won't be a big deal if something breaks. I've already had it in a broken state for weeks now while I've been working on pulling this off.

I've looked into FreeNAS...I really really really hate their web interface...it's like someone took a good UI and ran it through a wood chipper. And their forums culture is pretty abrasive. But they have a crap load of support so.....

I really liked the looks of nas4free. The UI looks like my thing. But the lack of replication support just seems like a dead end. That, and I'm told the nas4free runs from the HDD instead of embedded which is a downer.

And then there's ZoL...for which I'm not sure there is a well finished a solution as FreeNAS on that side of things. I have a lot of tinkering to do to see if I think I can confidently and competently manage a Linux ZFS server.

Anyone have some advice for this ZFS noob?

Rock&Roll said:

I know the general advice is to not run your ZFS server under VM.

Click to expand...

This is not true in general.
I use and suggest All-In-One solutions with a VM based ZFS storage on Solaris or OmniOS for more than five years. Beside me many many users use this at home, labs or in production system.

I even offer a ready to use and free preconfigured ZFS storage VM with Vmware tools and Web-UI. You only need to upload the storage VM appliance to a local datastore.

I prefer Solaris or OmniOS for ZFS as this is where ZFS comes from.
Maybe you will add this as an option beside BSD and ZoL based systems.

http://www.napp-it.org/doc/downloads/napp-in-one.pdf
http://hardforum.com/showthread.php?t=1573272

I like both Gea's All-In-One and FreeNAS, but for different reasons.

AOI is great for ZFS, and the napp-it interface has always worked well for me. If ZFS is your primary goal, I'd go this option first, especially since you have the ESXi passthrough understanding down. My instance of napp-it has been rock solid from day one.

I also use FreeNAS, it has more of an "appliance" feel for just doing simple tasks such as running some simple Jails (Plex & SABnzbd for me). But, I feel you do lose some of the core tools that you get with a lower lever ZFS implementation.

I say try both. You can actually port the filesystem between the two, another benefit of ZFS. Spend a day on each, find which works best.

Through the years I started with Napp-It, went to FreeNAS to try something new, Had stability issues with it, back on Napp-It.

Napp-It is no FreeNAS gui but it does run faster and has been more stable in my situation. Once everything is setup you rarely go to the GUI anyways. As far as jails go, You have an esxi box to create as many vm's as you like.

My only recommendation might be more memory, figure esxi takes about 1GB, and 8GB min for ZFS, youll be wanting more if you expand.

Your setup makes sense.

Just one potential issue for you if you export a ZFS filesystem backed by only a few disks as ESXi storage via NFS:
ESXi will sync all writes (this is the only safe way to use NFS storage with disk images) and you'll find that random I/O performance suffers terribly (not so much an issue if you have a lot of backing physical disks). Adding an SSD SLOG (aka ZIL device) helps a bit but it's not a magic bullet. An SSD ZIL isn't a cache, merely a safe log device for random writes in case of crashes - and it still must write its log to the backing disks every few seconds.

Anyway, it's not an issue if you don't care a lot about I/O performance.
In my case I did care, and I learnt the hard way about these limitations for my 3 disks RAIDZ1 all-in-one. I ended up buying an intel S3700 for the ZIL which improved random write perf but it's still a few times slower than running benchmarks without forced sync. Obviously the performance gap wouldn't be as big for ZFS pools backed by a lot of HDs (10 or 20+) but it's not a typical home setup anymore

Not to nitpick, but it's writes not all I/O.

Thank you for the thoughts everyone. Looks like I have a good number more angles to consider...

Windows shares are soooo easy. Why must I be burdened with this drive to learn new things?

I don't have an SLOG, so I just disabled forced sync writes. Your write consistency isn't as guaranteed, but honestly if you have important data then you really should have an UPS with some type of automated shutdown. Basically, just protect your system against power failures.

Performance is just fine without sync writes.

raiderj said:

Performance is just fine without sync writes.

Click to expand...

Performance is far better without sync write.
Sync write on ZFS is a security mechanism, similar to a cache + BBU on hardware raid. On a powerloss it takes care of last writes on next reboot.

If you do not need this extra security, disable sync. If you need the protection, sync write is slow without a fast dedicated Slog.

_Gea said:

Performance is far better without sync write.
Sync write on ZFS is a security mechanism, similar to a cache + BBU on hardware raid. On a powerloss it takes care of last writes on next reboot.

If you do not need this extra security, disable sync. If you need the protection, sync write is slow without a fast dedicated Slog.

Click to expand...

Disabling sync write for NFS ESXi datastores seems risky... In case of a crash, ZFS would fix the inconsistencies at the top level but the filesystems inside the VM images could still be silently corrupted since the guests don't know that their sync requests are actually ignored. You may not find out until a few months later that some important file is broken...

In practice it probably will not happen often, if at all (I haven't had a single power outage or hypervisor crash in years, and even if a crash happens it would have to be during a guest sync write) but if it ever happens you won't really know if you can still trust your data...

I've been running FreeNAS in a VM for, oh, almost 5 years without ever having a problem.

The rationale against FreeNAS or ZFS in general in a virtualized really boils down to "don't do it, because you might be tempted to do 'something stupid'"

Something stupid is usually:
- Attempt to use incompatible hardware with the host
- Under provision RAM causing crashes and data loss
- Try to run the ZFS partitions on virtual disk images (very bad idea)

The truth - however - is that if you DON'T do anything stupid, it's actually a very stable approach that works very well.

My only comment on your configuration is I don't know if I'd run RAIDz2 with 4 disks. Raidz2 is essentially the ZFS equivalent of RAID6. it uses two parity drives. So, with 4 drives, you'll be using two of them for parity.


It will work, but if you have a hard stop on the 4 disks, maybe two striped mirrors would be better. (This would actually be closer to RAID10) You would gain similar redundancy, and they certainly would perform better, as you won't ahve to compute as much parity.

Personally I really don't think RAIDz2 is worth it until you hit 6 disks. Some people will likely argue with me here though (This is the internet after all)

I have FreeNAS in a VM with an HBA passed through to handle my hard drives. Has worked great for months now.

I agree that RAIDz2 with 4 disks isn't worthwhile. For that you might as well run a ZFS stripe/mirror (aka RAID 10). I run a RAIDz1 up to 7 disks, then I would switch over to RAIDz2. But only until 12 disks or so.... after that you'd want to run multiple RAIDzX sets.

I disagree with the RAIDz1 assessment.

IMHO anything with just one redundant drive is functionally obsolete today.

It takes several hours to rebuild a full vdev with large multi TB disks. If you are rebuilding it after a drive failure, you are doing so when you have no parity. Any error at this point (let alone an additional failed drive) and you are toast.

Allowing "any two drives" to fail without data loss should be the minimum, that's why I have two six drive RAIDz6 VDEVs in a RAID 60 equivalent in mine.

I think you mean two 6-drive RAIDz2 VDEVs? So out of a total 12 drives you have ~8 drives worth of usable space. Makes sense, and I agree that's a safer way to go. For myself I'm just storage a bunch of non-critical files and don't mind being a bit riskier

Rock&Roll said:

- 4x3TB drives in RAIDZ2 (nearest equivalent to RAID 10)

Click to expand...

lulwut? You can build RAID 10 in ZFS - it's just a pool containing multiple mirror vdevs.