New ESXi build feedback/troubleshooting

[NobleX13]

I decided several weeks ago to create a VMware ESXi 4.1 host server to virtualize several machines that I usually leave running 24/7, and to enable me to play around with Server 2008, etc.

Ultimately, this is what I want to achieve:

-Windows Home Server, running with RAW access to either my SATA drives directly or a Dell Perc 5/i card running them in RAID 5

-Ubuntu 10.1 Counter-strike: Source dedicated server

-OpenVPN Server

-Server 2008 R2 machine as my domain controller/WDS imaging box

Without thinking, I purchased a hex-core 1055T for use in my current AMD AM2/AM3 motherboard. It turns out, most, if not all (including mine) AM3 boards do not support the AMD equivalent of VT-d for direct hardware access in VMware. As such, I have chosen to make my Core i7 rig become my ESXi machine.

Specs:

Intel Core i7 920
ASUS P6T-SE (confirmed VT-d support)
6 GB DDR3-1333
640GB WD Black drive for ESXi and datastores
3x2 TB Samsung F3 drives for WHS storage
5x1 TB 1TB drives in RAID 5 for additional storage.
Dell Perc 5/i RAID card
Intel PCI-express 4x Dual 10/100/1000 nic (http://www.newegg.com/Product/Product.aspx?Item=N82E16833106015)

Questions:

Will this amount of RAM and CPU be adequate for my current needs?

Is dedicating one NIC to WHS a good idea, or would I be better served by selling it so I can buy that Perc 5/i card? (I have a single gigabit intel nic I can use)

Will I have too much I/0 for that RAID 5 array? If so, I can get a 2.5" to 3.5" hard drive enclosure and run four 2.5" disks in Raid 1 + 0.

Thanks!

Any other suggestions are welcome!

 

[Child of Wonder]

RAM and IO are the first two bottlenecks you'll run into when virtualizing. 6GB will be pretty lean. I'd shoot for 9GB or 12GB if you can.

If you're going to be using Directpath IO, you'll be sharing the entire Perc5/i card with a single VM IIRC. That means you can't use it for datastores.

You can use RDM but you're still limited to 2TB per RDM just as with VMFS.

My advice would be to put together a simple file server with the Perc5/i to serve out your storage. A dual core, 2GB system would do. Load FreeNAS on it and you can even share out storage as iSCSI to your ESXi box to be used as a datastore.

At home I run two ESXi boxes. Each is just the CPU, RAM, and a 2.5" HDD. My file server runs Debian with a 4x250GB RAID 10 array which houses my VMs over NFS and a 3x1.5TB array for file storage. Works great. VMs get 50MB+ throughput and 8ms latency to the VMDKs.

If you don't want a separate file server then you can try your specs, but I think the single 640GB drive is going to get smashed trying to run 4+ VMs.

 

[NobleX13]

Hmm.... you're probably right... Now I need to rethink things a bit. Thanks for the input, though!

 

[lopoetve]

RAM and IO are the first two bottlenecks you'll run into when virtualizing. 6GB will be pretty lean. I'd shoot for 9GB or 12GB if you can.

If you're going to be using Directpath IO, you'll be sharing the entire Perc5/i card with a single VM IIRC. That means you can't use it for datastores.

You can use RDM but you're still limited to 2TB per RDM just as with VMFS.

My advice would be to put together a simple file server with the Perc5/i to serve out your storage. A dual core, 2GB system would do. Load FreeNAS on it and you can even share out storage as iSCSI to your ESXi box to be used as a datastore.

At home I run two ESXi boxes. Each is just the CPU, RAM, and a 2.5" HDD. My file server runs Debian with a 4x250GB RAID 10 array which houses my VMs over NFS and a 3x1.5TB array for file storage. Works great. VMs get 50MB+ throughput and 8ms latency to the VMDKs.

If you don't want a separate file server then you can try your specs, but I think the single 640GB drive is going to get smashed trying to run 4+ VMs.

you can't use local disk for RDMs, and I doubt that board will support VMDirectPath either for the PERC card

I'll second the rest though.

 

[Child of Wonder]

you can't use local disk for RDMs, and I doubt that board will support VMDirectPath either for the PERC card

I'll second the rest though.

True about RDM.

I read somewhere that Perc5/i does work for Directpath, but, still, why would you want to?

Best bet is to simply use the Perc5/i in a file server and create a 4 disk RAID 10 array for your VMs and serve it out with iSCSI or NFS. If you use NFS, format the array with XFS using the following options: -l size=64m,lazy-count=1 -f

Mount the XFS partition using: noatime,nodiratime,logbufs=8

Use the following options for the NFS export: rw,no_root_squash,async,no_subtree_check.

If you don't use a UPS with your file server, then remove "async."

 

[NobleX13]

Well then, here is how I may revise my current network topology:

PC1:
Gaming/admin desktop
Phenom II X6 1055T
GTX 280
3GB DDR3

ESXi server:
Core i7 920
80GB SATA boot drive
ASUS P6T-SE

FreeNAS server:
5x1TB ZFS array
Phenom 9150e 1.8GHz quad-core
2GB DDR2

Does this look ok? I want to run iSCSI on FreeNAS to host the VMs.

 

[Child of Wonder]

Well then, here is how I may revise my current network topology:

PC1:
Gaming/admin desktop
Phenom II X6 1055T
GTX 280
3GB DDR3

ESXi server:
Core i7 920
80GB SATA boot drive
ASUS P6T-SE

FreeNAS server:
5x1TB ZFS array
Phenom 9150e 1.8GHz quad-core
2GB DDR2

Does this look ok? I want to run iSCSI on FreeNAS to host the VMs.

Yep. FreeNAS will work nicely as an iSCSI datastore. I'd also get a RAID 10 array set up for your VMs to use over iSCSI. You can do this using software RAID in FreeNAS. Set the iSCSI target to fileio for better performance.

Give your ESXi box one Gb NIC dedicated solely to iSCSI and the same on your FreeNAS server. Heck, you could even connect the two ports with a Cat 5e or Cat 6 crossover cable for the absolute best performance but a Gb switch in the middle for future growth would also be nice.

Each of my ESXi boxes have 3 NICs -- one for management and VMotion, one for NFS, and one for the VMs.

EDIT: And if you use software RAID, use a chunk size of 256K or 512K for a little better performance as well.

Also, here's a great article about optimizing XFS performance based on the number of disks in your RAID array and RAID set.

http://www.mythtv.org/wiki/Optimizing_Performance#XFS-Specific_Tips

 

[NobleX13]

Yep. FreeNAS will work nicely as an iSCSI datastore. I'd also get a RAID 10 array set up for your VMs to use over iSCSI. You can do this using software RAID in FreeNAS. Set the iSCSI target to fileio for better performance.

Give your ESXi box one Gb NIC dedicated solely to iSCSI and the same on your FreeNAS server. Heck, you could even connect the two ports with a Cat 5e or Cat 6 crossover cable for the absolute best performance but a Gb switch in the middle for future growth would also be nice.

Each of my ESXi boxes have 3 NICs -- one for management and VMotion, one for NFS, and one for the VMs.

EDIT: And if you use software RAID, use a chunk size of 256K or 512K for a little better performance as well.

Also, here's a great article about optimizing XFS performance based on the number of disks in your RAID array and RAID set.

http://www.mythtv.org/wiki/Optimizing_Performance#XFS-Specific_Tips

Awesome! Thanks for the feedback! This should work great. If I use raid 10 I will need to have an even number of disks, so I think I will use my four Samsung 1TB hdds and reuse the WD Greenpower somewhere else.

At this point, I have a few more issues to address:

If I use freenas for my iSCSI datastores, I should probably use it for my local NFS/SMB fileserver as well. That would eliminate my need for running WHS in a VM, reducing my ESXi machine's complexity. The issue here is that I have 2 brand new 2TB Samsung F3 drives that I want to utilize to store my personal files.

Basically, my issue is this:

Of my 5 1TB drives, three are full of data that will need to be held elsewhere before I can use them.

Should I save up and purchase some more 2TB drives to run as ZFS, so that I can add some redundancy into the mix? In the past I had all 5 1TB drives running without redundancy under WHS.

Thanks!

 

[Child of Wonder]

Awesome! Thanks for the feedback! This should work great. If I use raid 10 I will need to have an even number of disks, so I think I will use my four Samsung 1TB hdds and reuse the WD Greenpower somewhere else.

At this point, I have a few more issues to address:

If I use freenas for my iSCSI datastores, I should probably use it for my local NFS/SMB fileserver as well. That would eliminate my need for running WHS in a VM, reducing my ESXi machine's complexity. The issue here is that I have 2 brand new 2TB Samsung F3 drives that I want to utilize to store my personal files.

Basically, my issue is this:

Of my 5 1TB drives, three are full of data that will need to be held elsewhere before I can use them.

Should I save up and purchase some more 2TB drives to run as ZFS, so that I can add some redundancy into the mix? In the past I had all 5 1TB drives running without redundancy under WHS.

Thanks!

If those are the F3 1TB Samsung drives they'll be nice and fast for your datastore.

However, you're going to have a 2TB datastore (the max you can have) so don't overload it with too many VMs. An array like that should be able to handle 6-8 VMs easily. lopoetve can throw in his $.02 there, too.

Yeah, I'd recommend some redundancy for your file share. I haven't used ZFS but IIRC it can do something similar to RAID 5.

If you're using 4 of those 1TB drives as your RAID 10 array, save the 5th to use as a backup drive. Any important data on your file share can be stored there. You can use VMware Data Recovery to backup your VMs there, too, if it's shared out as a Samba share.

Get ready to spend and build more than you originally bargained for.

 

[lopoetve]

True about RDM.

I read somewhere that Perc5/i does work for Directpath, but, still, why would you want to?

Best bet is to simply use the Perc5/i in a file server and create a 4 disk RAID 10 array for your VMs and serve it out with iSCSI or NFS. If you use NFS, format the array with XFS using the following options: -l size=64m,lazy-count=1 -f

Mount the XFS partition using: noatime,nodiratime,logbufs=8

Use the following options for the NFS export: rw,no_root_squash,async,no_subtree_check.

If you don't use a UPS with your file server, then remove "async."

The card does, but the motherboard has to support the IOMMU to make it work too, and a consumer level board won't

 

[lopoetve]

Yep. FreeNAS will work nicely as an iSCSI datastore. I'd also get a RAID 10 array set up for your VMs to use over iSCSI. You can do this using software RAID in FreeNAS. Set the iSCSI target to fileio for better performance.

Give your ESXi box one Gb NIC dedicated solely to iSCSI and the same on your FreeNAS server. Heck, you could even connect the two ports with a Cat 5e or Cat 6 crossover cable for the absolute best performance but a Gb switch in the middle for future growth would also be nice.

Each of my ESXi boxes have 3 NICs -- one for management and VMotion, one for NFS, and one for the VMs.

EDIT: And if you use software RAID, use a chunk size of 256K or 512K for a little better performance as well.

Also, here's a great article about optimizing XFS performance based on the number of disks in your RAID array and RAID set.

http://www.mythtv.org/wiki/Optimizing_Performance#XFS-Specific_Tips

YIPE! Run from freenas for iSCSI for ESX

I LOVE FreeNAS, but the BSD target has a nasty habit of corrupting VMFS volumes. OpenFiler is a safer bet there. NAS can do NFS great though.

 

[Child of Wonder]

YIPE! Run from freenas for iSCSI for ESX

I LOVE FreeNAS, but the BSD target has a nasty habit of corrupting VMFS volumes. OpenFiler is a safer bet there. NAS can do NFS great though.

I've had the exact opposite experience. Openfiler 2.3 still uses IET 1.4.16 which is riddled with command aborts when vSphere connects to it. ESX 3.5 is fine, however.

IET 1.4.20-1 is supposed to correct this but Openfiler won't have this in place until they release 3.0.

 

[NobleX13]

Hmm... I may have to reconsider my plans a little....

I just ordered an LSI 8-port SAS card and breakout cables for the FreeNAS machine.
My 5-in-3 3.5" drive bay adapter comes in today; this will house some of my fileserver drives.
I also got a new AMD socket AM3 ATX board for my gaming rig; as I am juggling parts around right now.

Because of the freeNAS box, I no longer need WHS in a VM. Also, I decided to save up for a Cisco 5505 ASA, so I will not need an OpenVPN machine running either. So, at this point, my VM load is as follows:

-Server 2008 R2 Domain Controller
-Ubuntu Counter-strike source server
-Windows 7 test client

I am also considering selling off my 5 1TB drives and purchasing three more 2TB ones, and creating one large ZFS array.

My second Western Digital Black 640GB drive gets in today, and I may use both of them in RAID 0 for my datastore. Then, I will back that up to my ZFS array and cut out iSCSI all together.

Is this a decent course of action, or should I pursue something else? I'm always open for ideas.

 

[Child of Wonder]

I would never rely on a RAID 0 setup even just for messing around at home.

Sure, you'll have the VM's backed up but I'd rather use RAID 1 and sacrifice performance for more stability. You may find, as I have, you want to spend less and less time troubleshooting and messing around with stuff at home and want it all to just work.

You could order a 3rd 640GB drive and set up RAID 5. However, you'll need to enable TLER on the drives or they'll have issues being in a RAID array with parity like RAID 5 or 6.

 

[NobleX13]

I would never rely on a RAID 0 setup even just for messing around at home.

Sure, you'll have the VM's backed up but I'd rather use RAID 1 and sacrifice performance for more stability. You may find, as I have, you want to spend less and less time troubleshooting and messing around with stuff at home and want it all to just work.

You could order a 3rd 640GB drive and set up RAID 5. However, you'll need to enable TLER on the drives or they'll have issues being in a RAID array with parity like RAID 5 or 6.

You're probably right, but unfortunately, I am just an Intern in our Systems Infrastructure team and must go back to college in August to finish up my CIS:Networking degree. Besides, I want to be the only person on campus (including my school's IT dept!) to be running a domain controller.

That might not be a bad plan; RAID 5 on a smaller array..... Hmm.

At any rate, this thread has evolved both my expectations and expenditures quite nicely; thanks to everyone that has provided me with feedback.

Now, the question is ultimately how I set up my NAS.... I'm open to more suggestions!

 

[lopoetve]

I've had the exact opposite experience. Openfiler 2.3 still uses IET 1.4.16 which is riddled with command aborts when vSphere connects to it. ESX 3.5 is fine, however.

IET 1.4.20-1 is supposed to correct this but Openfiler won't have this in place until they release 3.0.

Granted, but it doesn't have issues reporting the VPD pages, causing ESX to confuse volumes like the BSD target does (or did - haven't encountered it in a while). Always fun when ESX writes to two different volumes at once

 

[Child of Wonder]

Granted, but it doesn't have issues reporting the VPD pages, causing ESX to confuse volumes like the BSD target does (or did - haven't encountered it in a while). Always fun when ESX writes to two different volumes at once

That sucks. I had heard FreeNAS was working OK as an iSCSI target for vSphere.

OP, I'd just use NFS then. vSphere works just great with NFS and 4.1 has improved that functionality even more.

My NAS is Debian but you can still use FreeNAS. Just create an NFS share using XFS and the options I mentioned above and you'll be set.

 

[NobleX13]

It turns out the SAS card I bought is not compatible with FreeNAS, so I will run Server 2008 R2 and use a NFS share then. Thanks for the input!

 

[caveman-jim]

Noble, I was gonna say I've used Openfiler to serve iSCSI for VMWare and it's not great. Openfiler NFS is a better bet. Unless you've got dedicated iSCSI TOE hardware (plus switches with cache large enough for the frames etc), it might be better to go simple and NFS.

Sounds like a fun project.

 

[Child of Wonder]

It turns out the SAS card I bought is not compatible with FreeNAS, so I will run Server 2008 R2 and use a NFS share then. Thanks for the input!

I'm not familiar with Windows as an NFS share, but Linux will offer better performance with XFS than NTFS. I'd look into using Openfiler with NFS as caveman-jim suggests or simply installing Debian or Ubuntu. Ubuntu 10.04 may even use the latest IET if you want to give iSCSI a try. If not, you can install IET yourself.

 

[lopoetve]

That sucks. I had heard FreeNAS was working OK as an iSCSI target for vSphere.

OP, I'd just use NFS then. vSphere works just great with NFS and 4.1 has improved that functionality even more.

My NAS is Debian but you can still use FreeNAS. Just create an NFS share using XFS and the options I mentioned above and you'll be set.

FreeNAS NFS definitely works ~great~