Is raid controller essential for virtualized ZFS

[dalarhan]

Hello,

I am in the process of setting up my all-in-one server/workstation. The purpose of the build is to run multiple vm concurrently including windows and linux. Below are my specs.

Supermicro X9SRL-F
Xeon E5-2620v2 CPU
4 x WD 4TB Caviar Black with AF (to be configured as ZFS Raid 10)
1 x WD 1TB Caviar Black (boot drive)
4 x 8GB RDIMM (to be added on with another 4 x 16GB RDIMM)
SSD for ZIL (Intel DC S3500/S3700/730)
GPUs (not yet purchased but intend to run them in passthrough mode)

I intend to share out the ZFS raid10 storage with iSCSI.

It is not mentioned in the supermicro documentation that the on-board raid controller is a LSI controller as some motherboards are. So I'm assuming that it is an on-board Intel RAID controller which I have read that it is junk hardware by raid quality standard. Furthermore, it appears from the motherboard documentation that it only supports 0,1,5 and 10.

I am considering between the following options
- VMWare ESXi 5.5 with hardware level 9 - OmniOS vm with Napp-it
- Centos 6.5 with KVM - FreeNAS vm

As VMWare cannot passthrough individual HDD, my options are to buy an raid controller or use KVM to passthrough the individual HDD.

So my question is it worth the money to buy RAID controllers only to be run in JBOD mode?

I would appreciate any advice.

Regards.

 

[bds1904]

If you are using zfs as an all in one you don't want a raid controller at all, you want an hba.

Zfs likes direct access to a drive. In order to achieve this in an all in one you passthru the entire controller.

Get a sas2008 based hba and flash it to it mode. Passthru the hba to your vm and you will be good to go.

You may be able to passthru the built in controller in jbod mode also, but I prefer the option of a seperate hba personally.

 

[Aesma]

I haven't done this myself but I own several M1015/LSI 9240-8i and can confirm it's a good and relatively cheap card (the latest I bought was LSI branded, new, from HK, 100$ on eBay so 70€ for me).

Intel is not garbage, for a system (SATA) SSD it's the fastest you can get, but you can't use it for what you want.

 

[_Gea]

You can do RDM on ESXi to pass single disks to a VM but if you like barebone like performance and reliability, only virtualize the storage OS - do not virtualize storage hardware like controller or disks. (pass-through controller and disks). I would add a good HBA like a LSI 9207 or LSI 9211 (or the eqivalent IBM 1015) flashed to raidless 9211-IT mode for ZFS.

The onboard Sata in Raid-mode is indeed junk.

 

[m4rm]

You can do RDM on ESXi to pass single disks to a VM but if you like barebone like performance and reliability, only virtualize the storage OS - do not virtualize storage hardware like controller or disks.

Before going the RDM route you should google and read all the nightmare stories about ESXi, ZFS and RDM.

 

[danswartz]

Most of the nightmare stories seem to be with an old version of ESXi (4.1?) or people who defined things wrong. It is a hassle, but if you do use RDM with ESXi 5.x, it works fine. If you don't have HW that supports passthrough, it's likely your best bet...

 

[patrickdk]

The problem with google and nightmares, is people don't often post the results of their success, so the stats via google are not going be accurate.

I have been using rdm volumes for years without an issue. I have never used rdm with zfs, but I can't imagine what specific issues that would cause. But I have run linux and windows systems on rdm's without issues. But as the above says, I have only done this using 5.x, and I never used rdm on 4.x

 

[dalarhan]

Thanks for all the replies. I will try RDM first with existing copies of my files.

_Gea, I'm not sure if I understand you correctly. Is it correct if I store the OmniOS vm vmdk on datastore1, which is the 1TB boot drive and RDM the other 4 hard disks to the vm as ZFS raid10?

I have a side question since RDM might be disastrous. Where do you guys back up your ZFS volumes to? Is it backed up to another ZFS volume in mirror/raidz2/raidz3? If I use a mirror, then I would need a total 4 hard disks to back up 2 hard disks worth of data. Sound like overkill to me.

Regards.

 

[_Gea]

Thanks for all the replies. I will try RDM first with existing copies of my files.

_Gea, I'm not sure if I understand you correctly. Is it correct if I store the OmniOS vm vmdk on datastore1, which is the 1TB boot drive and RDM the other 4 hard disks to the vm as ZFS raid10?

I have a side question since RDM might be disastrous. Where do you guys back up your ZFS volumes to? Is it backed up to another ZFS volume in mirror/raidz2/raidz3? If I use a mirror, then I would need a total 4 hard disks to back up 2 hard disks worth of data. Sound like overkill to me.

Regards.

Yes, you need a local datastore where you place OmniOS.
ESXi can now start this VM after reboot.

When OmniOS is up, it can create a ZFS storage pool on any disks, or blockdevices like iSCSI that OmniOS can see.
This can be a ESXi virtual disk (bad idea for a storage server), a RDM disk offered by ESXi (ok if you do not have a dedicated HBA)
or disks that are connected to a HBA (pass-through, OmniOS can access the HBA with its own driver, perfect).

You then NFS-share a filesystem from this pool.
ESXi is then able to place other VMs on such a shared NFS storage on ZFS.
This is the idea behind: Create a full featured highend SAN storage, virtualized on ESXI.
On a reboot, you only need to care that OmniOS starts first as you need its storage.

About backup.
The idea behind ZFS Pools is storage virtualisation. This term means that you can create filesystems on a pool
where the size of such a filesystem is not fixed or limited. It can grow up to poolsize and you set reservations or quotas to manage.
If pool space is low, you just need to add a new vdev to increase available space and performance for all filesystems. (A pool is build from one or more
vdevs, a vdev is a single disk, a mirror or a raid-z array). This is a main difference to the oldstyle partitions bound to a disk with a fixed size that you use without ZFS.

If you want to backup, you can use either replication (this keeps a filesysysten to another server/ filesystem in sync down to a minute delay) or filebased
sync tools like rsync or robocopy. You do not need to care about the disk layout of a pool as this is only relevevant to performance.

A pool where you store VMs where you need nigh IOPS is usually build from SSDs or in a Raid-10 manner with disks (one or more 2 or 3 way mirrors).
A backup or filer pool is usually build from one or more raid z2 or z3 vdevs.

 

[zrav]

- Centos 6.5 with KVM - FreeNAS vm

Why virtualize the storage OS if you can have native ZFS on your host OS? By letting CentOS provide the ZFS pool you reduce complexity and gain performance, not to mention avoiding controller pass-through hassle.

 

[staticlag]

Yeah I'm using RDM for my 40TB backup server/Win Server.

It's a hassle keeping track of the RDMs for 15 drives but it works.

 

[dalarhan]

Why virtualize the storage OS if you can have native ZFS on your host OS? By letting CentOS provide the ZFS pool you reduce complexity and gain performance, not to mention avoiding controller pass-through hassle.

Putting ZFS in a vm prevents it from using up all the ram. It is especially important since I'm using it as an all-in-one. Besides, there are some issues being floated around on forums about Linux not releasing memory quickly enough.

Most of the nightmare stories seem to be with an old version of ESXi (4.1?) or people who defined things wrong. It is a hassle, but if you do use RDM with ESXi 5.x, it works fine. If you don't have HW that supports passthrough, it's likely your best bet...

The RDM option was grayed out in the napp-it vm, so I used this command below that is documented on http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017530

# vmkfstools -z /vmfs/devices/disks/<diskname> /vmfs/volumes/<datastorename>/<vmfolder>/<vmname>.vmdk

Is this the method that leads to the nightmares?

Regards.

 

[zrav]

Putting ZFS in a vm prevents it from using up all the ram. It is especially important since I'm using it as an all-in-one. Besides, there are some issues being floated around on forums about Linux not releasing memory quickly enough.

I run multiple KVM hosts with ZoL and some pretty active guests at work and never had problems with that.
I can not recommend VM images on NFS shares, the performance is just terrible. With zvols via iSCSI its much better but still not on par with native/local datasets.

 

[_Gea]

I can not recommend VM images on NFS shares, the performance is just terrible. With zvols via iSCSI its much better but still not on par with native/local datasets.

ESXi requests secure but slow sync writes over NFS.
If you do not need this extra security, you can disable sync (on ZFS this is a filesystem property).

If you use iSCSI this is the same but sync (writeback cache) is enabled by default.
For best security you should enable (slow)
For fast but secury sync writes you can add a ZIL accelerator on ZFS.

If your network is fast enough (or All-in-one with ESXi with connectivity on software) NFS is as fast or faster as a local ESXI datastore as you can use ZFS with its advanced caching features. And with ZFS you have a fast shared storage for backup/clone/move together with ZFS snaps and checksum security.

 

[danswartz]

ZoL via NFS gets good read performance, but write performance sucks even with a good SLOG device (I was using an intel s3700). There is an open performance ticket against this issue (opened by me). The issue is not ZoL per se, but the fact that the in-kernel NFS server on linux breaks up all writes into 4KB blocks. Since ESXi sends the writes over with sync requested, every single write to the SLOG is done as a 4KB synchronous write. So across 1gb link I was getting 90MB/sec or so reads, but only 20MB/sec or so writes. Until ZFS has AIO this will not improve. I tried using unfsd (user-space nfs daemon), but it won't support ESXi, since it doesn't implement the READDIR+ NFS opcode. There is another user-space NFS server (ganesha), but it's incredibly complicated to try to configure, so I gave up

 

[zrav]

If your network is fast enough (or All-in-one with ESXi with connectivity on software) NFS is as fast or faster as a local ESXI datastore as you can use ZFS with its advanced caching features.

This is only true for sequential reads or writes (with sync=disabled). As soon as it's a random load the latency of the network in between just kills performance. And even with the ESXi AIO setup you still have two network stacks to go through for storage requests, which you can avoid by having native ZFS on your host system.
If you don't trust ZoL, go with OmniOS for the host system, it has KVM too. These days I see only a few reasons to go with the ESXi based AIO (sorry Gea)...

 

[_Gea]

No problem.
There are many use cases where KVM (Linux, OmniOS or SmartOS) or Hyper-V is a good technical alternative.

I currently stay at ESXi as it is the leading player regarding virtualisation, offers a free choice of storage technologies, offers drivers for all my OS's (mainly Windows and OSX) and advanced management options as I use several ESXi/ all-in-one machines with a centralized management. This is more than the question how you can just combine ZFS with virtualisation.