ESXi / NAS

[fatguy]

I'm about 2 weeks into a project which to this point has been very educational. I'm trying to setup an environment which can host several isolated virtual machines (firewall/vpn, web & app servers, git servers, etc) and also serve my family's photo, music, and movie collection (CIFS/SMB). Data preservation is paramount, I back-up the really really really important stuff (photos, documents) to a USB drive and keep it at work. The appliances I run need resiliency as well, so I believe some level of RAID is necessary. Snapshotting (ZFS) would fit the bill nicely for 'oops I accidentally deleted something' moments.

For the VM server, I'm currently running ESXi 4.1 on some newly purchased hardware:

Motherboard: SuperMicro X8SIL-F
Processor: Intel Xeon X3440 Lynnfield 2.53GHz
Memory: 8GB (2x4GB) Kingston DDR3

The server is running completely disk-less. It boots the hypervisor off a USB flash drive, and contains a single datastore served via a QNAP T-219 NAS applicance using iSCSI. The X8SIL-F supports IPMI 2.0, making remote management a breeze. Everything works very nicely together, and if I could keep this setup, I probably would. The problem is that I need to ship the QNAP appliance off-site, since it was originally purchased to serve as a low-touch file server for my wife's business. Plus, having it off-site will give me a more regular backup strategy (via rsync).

So I'm at a point where I need to replace the SAN aspect of the setup. I'm researching (and debating) the possibility of having the ESXi server play double-duty - hosting virtual machines *and* serving as a SAN device. This would be ideal from a cost and power consumption perspective, but may not be feasible or practical. I'm all over the place at this point in terms of research/questions:

1. Can I utilize the RAID capabilities on my motherboard? The X8XIL-F has the Intel 3420 chip, and in the bios has options for 'Intel' and 'Adaptec' RAID codebases. I've played around a bit, but can't seem to get it working. I would want RAID1 for my primary VM datastore, but not for the drives hosting file-share content as they would hopefully run ZFS...
2. Has anybody created ZFS on top of VMFS datastores? Are there any performance / integrity concerns with this setup?
3. Would it be a bad idea to create a FreeNAS virtual machine and give it pass-thru/direct access to the disks used to serve my home file-sharing content? I've gone through the necessary steps and it works - but I'm a bit apprehensive about doing something that is off the beaten path...
4. If I need to build another (dedicated) SAN device, would iSCSI or NFS be the preferred protocol? Currently, I'm leaning towards NFS after reading about thin provisioning support and the benefits of direct VMDK file access for backup/snapshot purposes. The difference in performance seems to be negligable.

Does anyone have any experience or insight that could help me?

Thanks!

 

[NetJunkie]

Your build is almost exactly what mine is going to be here real soon. For NAS I just got a Synology DS1010+. You can't use the X8SIL's RAID..it's software RAID and ESXi doesn't support it.

 

[fatguy]

That's a nice little appliance - possibly a bit pricey for me given my budget, but definitely a consideration...

If I went the dedicate SAN route, I'd probably build off of the SuperMicro X7SPA-HF board, since it has six on-board SATA ports, Dual gigabit NICs, and IPMI 2.0 (which I love). I think I could juice up the memory and run ZFS on the Atom D510.

 

[NetJunkie]

Yeah..I don't blame you. For what I do I really needed something on the VMware HCL for vSphere. The free NAS solutions are a bit quirky and I don't need that when I do testing.

 

[fatguy]

Yeah..I don't blame you. For what I do I really needed something on the VMware HCL for vSphere. The free NAS solutions are a bit quirky and I don't need that when I do testing.

Are you going iSCSI or NFS (and why)?

 

[fatguy]

Something else I'm investigating - running the VM appliances off a single disk (non RAID) and having them periodically snapshot to another (external) disk. Anyone have any thoughts or experiences with scheduled ESXi snapshot capabilities?

 

[nessus]

Iomega Storcenter iX2 and iX4 series units are both VMWare certified under iSCSI or NFS. They cover a fairly good price range and have relatively low power usage.

 

[NetJunkie]

Are you going iSCSI or NFS (and why)?

I'm using NFS. I prefer NFS over iSCSI in most cases...and I expect it to be quicker on the Synology since it's a NAS device and not a block device. Also, I see that on the VMware HCL that it lists NFS as the tested method.

 

[fatguy]

ended up going with a separate NAS build for ESXi datastore storage and file sharing for my network.

I'd like to setup a domain controller + WINS + LDAP as virtual machines on my ESXi box, but I hate to introduce a circular dependency (NAS box uses LDAP, running on ESXi server with datastore on the NAS). Would that be ok?

 

[fatguy]

ended up going with a separate NAS build for ESXi datastore storage and file sharing for my network.

I'd like to setup a domain controller + WINS + LDAP as virtual machines on my ESXi box, but I hate to introduce a circular dependency (NAS box uses LDAP, running on ESXi server with datastore on the NAS). Would that be ok?

Turns out this is not ok - at least during the test/build-out of my lab.

I setup a DHCP/DNS virtual machine using a minimal Ubuntu 10.10 install - but since the VM is running on an ESXi server which uses my FreeNAS host as a datastore provider, every time I need to restart my FreeNAS server my entire network goes down the tubes. The VMs lock up, and I can't resolve domain-name look-ups, even to get into the ESXi server via IPMI. I need to specify actual IP addresses, which defeats the purpose of setting up DNS...

Taking a step back and revisiting my requirements, can anyone suggest a viable setup? Since the original post, I've purchased the hardware to run a dedicated NAS (using FreeNAS) which includes a SuperMicro X7SPA-HF, 4GB DDR2, and 4x1.5TB drives. The goal was to run ZFS, serve up my VM datasores (via NFS), and media/home directory file sharing using CIFS/SMB. I've hit a few snags:

• ZFS/NFS performance with FreeNas (0.7.2) is absolutely horrible. Using 'dd' within a guest OS, I never get more than 4MB/s transfer. This seems to be a known issue with FreeNas/ZFS/NFS. Using iSCSI, I see 40-50MB/s. I want to use NFS so I can take ZFS snapshots of my virtual machines, and manage the datastore w/o going through the vSphere client.
• FreeNAS has lost my configuration settings a few times already. I'm running embedded off a USB drive. Occasionally after some reboots, FreeNAS comes up without an IP address (statically configured) and my drives are no longer registered. Users and groups are also missing. It is possible the USB drive is bad - a new one is on the way.
• I've tried Nexenta, but it seems to really dislike USB drives. I was trying to keep my ZFS volume clean of any OS-related stuff.

At this point, I'm considering a few options:

• Trying Nexenta for my NAS, ditching the USB drive for the OS install
• Running Ubuntu w/VMWare Server (instead of ESXi) so I can create a 'single server' with DNS/DHCP/OpenLDAP, and also serve VMs.

Any advice?

 

[aaronearles]

I can just say that I spent the last week messing around trying to use FreeNAS as an iSCSI datastore for ESXi, it was completely unstable and would reboot on it's own whenever I put a load on it, and you have to reboot it everytime you make a single change, even if you have to make another change that requires a reboot right after, you have to reboot twice, not good for block level access to the host...

I ended up switching to openfiler and it's been 100% stable so far and required no reboots. It lacks a lot of cool features that FreeNAS has, but if simplicity means stability then I'm okay with that. Performance right now is about 80MB/s, hopefully that will increase a bit when I move off the onboard nvidia nic and onto a pair of bonded broadcoms with jumbo frames

 

[SpaceHonkey]

If you search for lopoetve's comments of FreeNAS and Openfiler - use Openfiler for iSCSI and FreeNAS for NFS.

 

[MassiveOverkill]

I can just say that I spent the last week messing around trying to use FreeNAS as an iSCSI datastore for ESXi, it was completely unstable and would reboot on it's own whenever I put a load on it, and you have to reboot it everytime you make a single change, even if you have to make another change that requires a reboot right after, you have to reboot twice, not good for block level access to the host...

I ended up switching to openfiler and it's been 100% stable so far and required no reboots. It lacks a lot of cool features that FreeNAS has, but if simplicity means stability then I'm okay with that. Performance right now is about 80MB/s, hopefully that will increase a bit when I move off the onboard nvidia nic and onto a pair of bonded broadcoms with jumbo frames

Let me know if you get MPIO working. I say MPIO because ESXi actually prefers Round Robin MPIO and does not support/recommend link aggregation. I was actually getting around 109 MB/sec on my onboard Nvidia without jumbo frames, but this was iSCSI served to Win7.

I can get MPIO working on the Vsphere side, but can't for the life of me get it working on the Openfiler side (using version 2.99). I fear a the multipathing module may have to be loaded manually. When I start my transfer, you can see both NICs on my Vsphere host blaxing away, but only one NIC on the OF box is active. This is all on Broadcom 5709 NICs.

Getting jumbo frames to work in Vsphere is also a PITA. It must be done CLI.

 

[aaronearles]

Yeah I have jumbo frames enabled on the esxi host, through the cli without a problem, I'll mess with it this weekend and let you know what I come up with. I basically stopped because I didn't have enough network cables, so I'll grab some from work before the weekend.