ESXi/ZFS All In One, designing the storage

I

idea

Gawd
Joined
Jan 24, 2005
Messages
615
My friends, I have decided to do try of these ESXi/ZFS all-in-one projects using a full size ATX enclosure with 8 hot swappable bays (plus two 5.25" empty spaces). How should I set up the storage?

It's the usual set up. The ESXi will passthrough the PCI storage to Solaris 11 Express, which will then present the disks back to ESXi via virtual 10GigE with iSCSI or NFS. I want stability, security, redundancy, and performance. Performance is what I am most scared of. Although I will be hosting the VMs off a ZFS pool being shared via virtualized iSCSI, I do not want the VMs to experience problems due to high I/O wait (or any other problems for that matter!)

This box will be a development/learning machine to host about 20 guests. The only user is me. The box will have very low load, almost none. To be honest anything more than RAID1 SATA is probably overkill. RAID10 SAS is DEFINITELY overkill. But I don't care, I'm just interested in building a nice machine. The RAIDZ pool will more than likely be cheap, slow, low power 2TB disks.

I have available to me:
  • ESXi 4.1 and Solaris 11 Express
  • Supermicro X8SIL-F
  • An enclosure with 8-hot swappable disks (plus two 5.25" empty spaces)
  • 4x SAS 15K 300GB disks 3.5"

I will be buying:
  • 2x 2.5" drives for ESXi and one Solaris 11 Express VM image
  • Xeon X3440
  • 16GB ECC UDIMM
  • 2TB 5400RPM disks
  • Dell SAS6/iR (LSI 1068e)

What I would consider buying if recommended:
  • 4x 2.5" enclosure, and SATA laptop drives for the VM datastore
  • SSD drives for L2ARC/ZIL





Right now I am considering a few different configurations. Remember, I am limited to 8 hot swappable disks plus two 5.25" spots I can use. For the configs with the 2.5" drives I would buy a 4x 2.5" hot swappable enclosure to fit in a 5.25" space.

This config may be overkill. Plus, the SAS disks take up half my slots and they run loud and hot.
  • 4x SAS 15K disks for the VM datastore (2x mirror vdevs, akin to 4 disk RAID10)
  • 4x 2TB 5400RPM drives to use for RAIDZ pool

This is a happy medium from the above config but I'm guessing not as good performance as a double mirror vdev
  • 4x SAS 15K disks for the VM datastore (mirror vdev, akin to RAID1)
  • 6x 2TB 5400RPM drives to use for RAIDZ pool

This config takes the SAS drives out of the equation and allows me to use all 8 disks for a RAIDZ pool. I'm wondering if laptop drives are fast enough though.
  • 4x SATA laptop drives to use for the VM datastore (2x mirror vdevs, akin to 4 disk RAID10)
  • 8x 2TB 5400RPM drives to use for RAIDZ pool

This is similar to the above config, but introduces SSD for cache. Something I've never played with before.
  • 2x SATA 2.5" laptop drives to use for the VM datastore (mirror vdev, akin to RAID1)
  • 2x SSD 2.5" drives to use for L2ARC/ZIL
  • 8x 2TB 5400RPM drives to use for RAIDZ pool

This is the most simple and probably not the smartest idea in terms of VM performance.
  • 8x 2TB 5400RPM drives to use for one single pool (RAIDZ).
  • No other disks. For the VM storage, cut a piece of the RAIDZ pool and share with iSCSI
 
Last edited:
i would

build a SAS raid 10
-> you will like the speed, opt add a ssd read cache
slow disks + SSD cache for VM's is better than nothing, but not really fast in most cases

build a second pool for data and backups
-> raid-Z (Z1-Z3)


One large Raid-Z pool is not good for ESXi.
You need IOP/s and good sync write speed,
good sequential rates does not count - not evident


Gea
 
Maybe I should trade these 3.5" SAS drives for some 2.5" SAS drives
 
Whats is wrong with large RAID-Z pools and ESXi? Will it be slower then small RAID-Zs?
 
You want to spread the I/O over multiple vdevs. So frex: instead of say, a 12 disk raidz, you would stripe 3 4-disk raidz.
 
I had stability issues with SE11 and ESXi when using >1 processor.

I decided to run SE11 bare metal
- using virtualbox w/phpvirtualbox for my VMs
- and Napp-It for system monitoring

Works very very nicely.
Currently running a couple diskless clients over iscsi, smb server, nfs server, tftp server (for the diskless clients).
All very stable and fast.
 
The X8SIL-F is great (I use one) but look into the new 1155 mainboards from SuperMicro and the new Xeons. They cost about the same - and price/performance is better. Make sure you get one with two ESXi compatible NICs.
 
X9SCL-F rocks. You do have a NIC that won't work out of the box - just check this board for chilly's instructions on using his oem.tgz and it will work fine.
 
I actually get better network throughput running SE11 under ESXi (using the vmxnet3 driver) than when running SE11 on the bare metal (using the Intel NICs on my X8SIL-F). Under ESXi, I can pretty much max out my gigabit connection indefinitely when copying large files (30-40GB) from a Win7 client to my raidz2 pool (around 100MB/s). With the server running on bare metal, I would get a short burst of 100MB/s transfer for the first few GB, and then it would level off around 60MB/s for the remainder of the copy. I guess ESXi's Intel and VMXNET3 drivers must be more heavily optimized than the Solaris Intel driver.

On another topic: Since you're using a 1068e controller, consider formatting/creating your storage pools using a FreeBSD/ZFSGuru bootCD (or create a VM and pass the controller through to it, which is what I did). Use the ZFSGuru UI to GEOM-format the disks and create the pools using the 4K sector override feature. Then reboot into SE11. Now you can import the pools and upgrade them to the latest ZFS version, and you will have nice, 4K-aligned vdevs. Even if you're currently using true 512b sector disks (e.g. Hitachi) this will allow you to easily add/replace 4K disks in the future without any performance issues.
 
You must log in or register to reply here.
Back
Top