Planning a home esxi build

D

Dephcon

[H]ard|Gawd
Joined
Jul 27, 2004
Messages
1,234
Hey guys,

I actually do a lot of work with VMware as part of my job, but my experience is limited to IBM/HP blades.

Currently I run a unraid server at home for my multimedia needs, which is working just fine but the mobo is a bit older doesn't have adequate PCIe lanes to support my new IBM M1050 cards. For now I can run one of them with 8 drives at full speed and an older LSI card with 4 drives at x4. This will work just fine in the immediate term. In the longer term, I've always wanted to build an ESXi whitebox so why the hell not now? Currently, the only hardware I have are the m1015s, Corsair 850pro PSU and a 640GB FIO ioDrive Duo. :D

Just after a bit of browsing, I think the GA-990FXA-UD7 paired with a FX CPU might be pretty slick. It'll give me (4) PCIe x8 and (2) PCIe x4 slots for HBAs, NICs, FusionIO and has IOMMU support in the BIOS. Direct-path is very important for presenting the HBAs directly to unraid.

I know AMD isn't exactly a player anymore in terms of raw performance, but I'm not really going to be pushing any intense workloads.

Any thoughts or suggestions?
 
So I decided to take the plunge, ended up with the 990FXA-UD5 instead as it was available locally and almost $60 cheaper.

I overclocked the FX-6300 to 4.0GHz and it's running ESXi 5.0U2 like a treat. IOMMU works great and have an LSI HBA directly connected to a test unRAID VM along with passing through the USB key. I gotta say it run pretty great so far keeping in mind I only have 8GB of RAM and using a laptop HDD for a datastore, lol. Ordered a 4x8GB kit on amazon and will grab some 1TB 7200.11s to do a RAID 1/5/10 via SB950 for a datastore until I decide on an SSD.

I've dropped FIO from my plans as the AMD mobo in general is pretty hot compared to what I'm used to with Intel and surprisingly so are my Broadcom dual/quad NICs. I guess that's what you get with server-grade hardware... I'm going to need more fans. I do want to try passing through a GTX680 to a win7 VM and see if I can play Crisis.

Anyway, just thought I'd let everyone know about my positive experience. I'm pretty impressed with AMD on this one, I may even get a second kit so my vcenter appliance doesn't look so pathetic with only one host.
 
Last edited:
When it comes to storage, I painted myself in a corner trying to do a RAID array with traditional disks, even the high-end consumer disks. In hindsight, being what it is, I should have gone SSD right off the bat. Mind you, my disk setup (see sig) performs well for my needs, but by the time I purchased three 1TB drives and the RAID card, I could have gotten a 256GB Samsung 840 Pro and with thin provisioning been smooth sailing.
 
all ssd for datastores here, big improvement from sata drives, unless you need a lot of space

(using 3x 240gb ssd and 1 640gb sata)
 
Yeah I know that doing RAID with SATA is bad news bears, but the drives are free and at this point I'm still testing. Once I move this hardware into my unraid case (Coolermaster 590) I wont have room for anymore 3.5 disks, I have all 12 bays filled with unraid disks. So it's either an SSD or Raid 5/10 with 2.5in 15K SAS drives lying on the bottom of the case, and I feel much more comfortable doing that with SSDs.

I have vCenter setup so I can do a live storage migration once I can afford an SSD. I know it's not a ton money for a 840 pro, but I bought a Swifttech H220 today for my upcoming 4770K/ROG Impact build which is probably going to be quite costly. Also the 32GB of RAM I have on order was almost $300 :(
 
Last edited:
SATA drives in RAID isn't that bad. Slower than flash, yes, but not horrible.
 
Well I also pulled them out of an old supermicro chassis we used for bacula like 3 years ago... I'm sure they have a lot of miles on them but they should work long enough until I can get something better. However, pretty much anything will be an improvement over the laptop HDD I'm currently using :)
 
After 10 days of testing I decided to exchange the UD5 for the UD7 and the FX-6300 for the FX-8320. The reasoning being that, 1- I tried the FIO card out and quite frankly it's amazing so it's back in the build requirements and 2 - with 6-8 VMs I was getting enough CPU Ready during load to bother me, so hopefully the extra 2 core will help out. (I'm used to dual E5-26xx systems at work so I might be getting up in arms over something I have no control over in a single socket desktop system :D)

Also the 32GB ram kit I bought wasn't able to hit 1866mhz, so I returned that and ordered a 32GB ECC kit from Kingston. The gigabyte 990FX boards don't officialy support ECC, but I've read reports of the rev 1 boards do work and open up hidden EEC options in the BIOS. The UD7 board I have on order has the newer UEFI so I have no idea if it'll work, but i'm willing to risk another return over it.


Does anyone have any suggestions on backing up my VMDKs? We used to use the NetApp SMVI plugin for vcetner but it sucked so we just stopped and use the volume snapshots if something breaks.
 
Last edited:
A snapshot is not a backup. Since you using free ESXi the easiest thing might be ghettovcb. Veeam will not work with the free ESXi version.
 
Yes, I know NetApp snapshots aren't backups, but with redundant heads, disks, nics, sas controllers, etc on the NetApp we're pretty safe. (lab environment)

I'm licenced for esxi and vcenter so I can use anything dependent on that... the fee-er the better :D
 
Another option is to throw a few large spinning disks into the machine and just schedual clone jobs to "copy" the VM to the spinners.

Simple and you don't need to spin up a third party app when things explode.
 
Also a cool idea. Can I schedule stuff like that in vSpehre or do i need to run a scheduled powercli task on a windows host?
 
Dephcon said:
So I decided to take the plunge, ended up with the 990FXA-UD5 instead as it was available locally and almost $60 cheaper.

I overclocked the FX-6300 to 4.0GHz and it's running ESXi 5.0U2 like a treat. IOMMU works great and have an LSI HBA directly connected to a test unRAID VM along with passing through the USB key. I gotta say it run pretty great so far keeping in mind I only have 8GB of RAM and using a laptop HDD for a datastore, lol. Ordered a 4x8GB kit on amazon and will grab some 1TB 7200.11s to do a RAID 1/5/10 via SB950 for a datastore until I decide on an SSD.

I've dropped FIO from my plans as the AMD mobo in general is pretty hot compared to what I'm used to with Intel and surprisingly so are my Broadcom dual/quad NICs. I guess that's what you get with server-grade hardware... I'm going to need more fans. I do want to try passing through a GTX680 to a win7 VM and see if I can play Crisis.

Anyway, just thought I'd let everyone know about my positive experience. I'm pretty impressed with AMD on this one, I may even get a second kit so my vcenter appliance doesn't look so pathetic with only one host.
Click to expand...


Just built a similar rig with the same mobo, fx8350 and 32gb ram for ESXi build. Running great so far
 
You must log in or register to reply here.
Back
Top