Considering Collapsing my Lab into a Single Host

[Vader]

So...i'm on the fence. I have three systems that handle my Private Cloud Lab and I want to start collapsing some things possibly going back to a single Host, nested ESXi. I also want to get rid of spinning disk in a Nexenta Virtual Appliance I have on my Infrastructure host.

My thought was to go with a single host, and load it with SSD's for fast storage.

Having the RACK and all the gear is getting old, it's loud, and uses too much power.

I'm going to list out my hardware, and maybe you guys/gals can give me some insight on going this route:

3 x SKT1366 Dual CPU L5520 systems with 48GB of DDR3 ECC each
Several Dual Port Intel Pro Gb PCI-E NICS
Couple of Dual Port Intel X520 10Gb
Single SKT 1156 Xeon 3400 w/32GB DDR3 ECC - Motherboard Contains LSI 2008 6GB Integrated SAS Controller
Silcom Six Port (3 Intel Controllers on one card) PCI-E NIC
6 500GB Western Digital Black Harddrives
120GB SSD Used for Nexenta Cache
60GB SSD Used for Nexenta Cache

I have an Iomega PX4-300d w/4 2TB 6GB SATA in a RAID5 as well as an HP 1810-24G switch and an Ubiquity Edgemax Router.

One of the above systems is actually my workstation so i'm leaning towards keeping it, expanding the memory and setting everything up via VMware Workstation 9.

Load it with some larger SSD's and just use it for vSphere Storage via a Nexenta Virtual Appliance and i could run my Infrastructure VM's off the PX4.

Ideally, I would like to use some Host caching solution but I don't have access to any as of yet and it wouldn't help me with Nested anyway.

Really wanted to get some thoughts...maybe I should sell it all and build a net new system and run everything via VMware workstation..?

 

[_d3_]

Really wanted to get some thoughts...maybe I should sell it all and build a net new system and run everything via VMware workstation..?

Or baremetal esxi and passthrough (VT-d) gpu (7xxx ati/amd cards, nvidia will work only with luck) and usb controler to one (autostarted) VM, so you can use your server as workstation.

On x79/C6xx platform VT-d is generaly more supported than on z77/87/C2xx. If 32gb ram si enough for you, than prefer Asrock or Supermicro boards (but found some records of VT-d working on Gigabyte boards) and avoid Asus.

Also avoid boards with PCIE bridge (PLX).

 

[_d3_]

Something like:

Supermicro X9SRH-7F board (LSI 2308 +2xGBe LAN i350 onboard) (or similar, just example)

Xeon E5-1620 (4c+HT) (or wait for IB-E lineup in september)

4-8x 16gb reg.ECC ram (or 8x 8gb reg.ECC ram if 64 is enough)

7750 Ati GPU - passthrough to Win VM (if you are not gaming) (or some older Ati/AMD card)

1x 60/120gb SSD you already own for ESXi installation + storage VM + Win VM (on onboard C602 chipset sata port)

2-6x 240gb+ SSD on onboard LSI 2308 passthrough to storage VM (striped or striped+mirrored)

UPS

You can passthrough entire usb controler (mice+keyboard plugged in) to Win VM and plug monitor to passedthrough GPU. For sound you will need monitor with speakers to get it from GPU or buy usb "soundcard", like Sound Blaster Play!.

Storage VM - mount datastores internaly through vNIC (iSCSI or NFS) for your VMs and use your Iomega NAS for templates/isos/backup/other. If using OI/solaris/omnios/nexenta set sync=disabled for lab usage.

Set storage VM and Win VM (aka your workstation VM) to autostart.

 

[Vader]

That's a bit too much effort with little gain. Truth is, I really don't have time to invest like that anymore. I need something easy, beefy (dual skt/128GB+memory) and very fast storage so i'm not bottlenecked.

I'm thinkin Workstation..but not too sure..maybe keep two hosts , on workstation and one straight ESXi..and beef up the ESXi host.

 

[QHalo]

I'd just put it all on Workstation. My laptop runs 3 nested hosts, a couple nested VMs, 2 vCenters, 2 Vyatta routers, and 2 ONTAP simulators on 32GB of RAM and one Crucial M4 SSD and it doesn't slow my machine down at all. It's plenty to test things at work. For my house, I'd do something similar and just get a couple more SSDs to put other VMs on to spread it out a bit. Much simpler and you don't need all that hardware. I still have my two hosts at home but I'm going to do the same as you at some point. Just scale up the memory and processors but still use it as my desktop.

 

[Vader]

Yeah..I was thinkin that..but I run most of my workloads for home in vCloud Director..so...I think what i'm gonna do is keep the best out of the three, load it up with memory, a 6GB SAS controller, and some SSD's...and sell the rest. I'll convert the single SKT 1156 Xeon system to my new workstation...and sell off the other two to fund this thing. The Single host will continue to run ESXi, along with two clusters of Nested ESXi hosts, some for infrastructure, some for vCloud.

The idea of a "data center" in the basement used to sound great...not so much anymore...maybe when the new house is built i'll put something dedicated in to handle everything including media etc.

 

[Lag]

I'm currently using a i7-950 / 48gb ddr3 / 256gb ssd on a Supermicro X8STE. I also have a few xeon x3440 / 32gb ram hosts. Everything connects to a single storage server (10x1tb in a raid10) w/ 2x 120gb ssds as cache / dual xeon e5405 / 16gb ram. All the virtual hosts connect with 4x 1gige ethernet ports (two to each) to a Powerconnect 5324 (acting as a SAN switch for NFS/iscsi) and a Powerconnect 6024 (for VM traffic / connectivity to other networks), the Storage server connects with 6x 1gige ethernet ports to the storage switch (3x LACP iscsi targets / NFS mount points).
The x3440 based hosts are clustered and run the "production" environment, the i7-950 is more my lab system, which hosts my win8 virtual desktop i connect to and use as my secondary workstation (primary being ubuntu 10.10 (still reluctant to give up my gnome 2.x)).

I'm always on the lookout for potential high memory density alternatives which can be had for reasonable prices. Based on my findings in the past year or so, the best option right now (as already mentioned) is a Xeon e5-1620 or e5-1650 starting with 4x16gb reg/ecc ddr3 (or 8x16gb now, or later). I would also go with 2x or 4x ~512gb ssds and some high capacity storage (a few 4tb ultrastars would do the trick depending on your needs).
You could also elect to go with the desktop variant of these cpus (i believe the i7-3820 and i7-3930k) if you do not have an issue with being limited to 64gb ddr3/1600.
This setup isn't just good for memory density, it's one of the fastest cpu architectures on the market, supports quad channel memory fully populated @ 1600mhz (when your entire infrastructure lives in memory, this comes in handy).
I believe HP sells a z-series with similar specs, but you could definitely build one with minimal effort. (forgive the incoherent ranting)