NAS and VM server Software Architecture

[Gallager2014]

I am in the process of building a server that will act as a NAS/File Server and host some VMs. I have two plans, but I am unsure which one is better.

Plan 1:
1) Run EXSi with all servers running as VMs.
2) Solaris variant running ZFS, running as the file server/NAS accessible via FTP/SMB/NFS/etc
The hosted files for the NAS/file server will be on a separate HBA.
3) All other servers (media, dev, etc) running as VMs.

Plan 2:
1) Have Solaris variant running as base OS, acting as file server/NAS using ZFS as the filesystem
2) All other servers running as VMs

Questions:
1) With Plan 1, is ZFS being used as the base filesystem, or are the stored as VMDKs? I want the data protection/integrity benefits of ZFS.

2) Is there another option I am overlooking?

3) Which Solaris variant (Solaris, OpenIndiana, Illumios, etc) is recommend?

 

[_Gea]

1.
A local vmdk datastore is only used for the storage VM.
All other VMs and the storage pool is native ZFS with an NFS storage for ESXi.
You should look for a vt-d capable system with an extra HBA that you can pass-through directly to the storage VM

2.
Virtualizing on top of (virtualbox) or within (zones, KVM) are other options.
But ESXi as base and all other VMs on top offers best performance and flexibility.

3.
Main options are OmniOS (free, stable) or Solaris (commercial, currently state of the art regarding ZFS with Solaris 11.3)

 

[davewolfs]

_Gea, would you say most napp-it users go with OmniOS or a fair amount on Solaris as well?

 

[Gallager2014]

Is there an option to choose the filesystem type the vmdk datastore uses? Data integrity is a priority.
Does using ESXi (or other VM solution) as a base make future migration to newer server hardware (CPU and MB) easier?

 

[HammerSandwich]

Is there an option to choose the filesystem type the vmdk datastore uses? Data integrity is a priority.

For the storage VM? Not that I'm aware of. Best bet is hardware RAID1. As Geo will point out, reinstalling ESXi & restoring storage VM from backup is a ~30-minute job.

Does using ESXi (or other VM solution) as a base make future migration to newer server hardware (CPU and MB) easier?

Yes, yes, OMG, yes!

 

[_Gea]

Is there an option to choose the filesystem type the vmdk datastore uses? Data integrity is a priority.

If you care about data integrity, you can use a good hardware raid with cache and BBU with vmfs to reduce the risks due the raid write hole problem.

The best of all solution is using a ZFS NAS/SAN with shared NFS or iSCSI shares where you can put your VMs on. This avoids the write hole problem of hardware raid and offers checksums with auto repair, a crash resistent CopyOnWrite filesystem and unlimited snaps.

You can use either a dedicated NAS/SAN or as a VM with storage pass-through for performance.

 

[_Gea]

_Gea, would you say most napp-it users go with OmniOS or a fair amount on Solaris as well?

Most napp-it users are currently on OmniOS as its free and includes LZ4 compress. The only plus of Solaris was ZFS encryption that is rarely needed in production environments.

This may change with Solaris 11.3 as it offers LZ4, ZFS encryption, SMB 2,1 and faster sequential resilvering. Beside the first, these important features are missing on Illumos based systems.

 

[MAuVE]

Most napp-it users are currently on OmniOS ....

I am a happy Solaris follower.
Its a SOHO install, so I think free (unpaid) usage is allowed.
From the times of OpenSolaris 2008.11, which was manageable by a Unix novice like me - vs - straight Solaris that was over and above me, I kept following the Solaris path, not be tempted to check other distributions.