Your home ESX server lab hardware specs?

[agrikk]

I'm going to be buying some new hardware to build two 64-bit capable VMWare ESX 3.5 servers for a home lab and I'm interested in what you guys are using for your labs.

Before I go out and do some research on my own, I'm looking for your experiences with the hardware you've already purchased.


Right now I'm thinking about two machines with:

one or two quad-core 64-bit CPUs (Intel v AMD?)
Motherboard type?
8-16gb RAM
SCSI v SATA II?

What's in your boxes?

I'm also trying to figure out what to use for storage for the VMWare cluster (for DRS and VMotion stuff). I have a Fiber channel SAN that I'm currently using in my production environment, but I could pull it out and redesign things there to make my lab easier, but I'd rather go with something else.

Perhaps I could build a Linux box and use NFS as the shared storage, but I've never done it (which makes it more appealing a project ) and am not really sure how it works. Can I use a 32-bit linux machine (with 1.5TB of direct attached SCSI drives) as the storage host serving NFS to 64-bit VMWare servers? Which flavor of Linux works the best in this capacity?

 

[LittleMe]

I only have my ESX lab here at home and it's a Dell PowerEdge 1950. It's got 2 quad-core 2.66Ghz chips and 16GB of memory. 2 SAS drives mirrored for booting only and it's tied to a 12TB iSCSI SAN for guest storage.

You can put FreeNAS on your 1.5tb box and use it for guest storage (via iSCSI or NFS). It doesn't matter if that box is 32-bit or 64-bit linux if you don't go with FreeNAS.

 

[agrikk]

It doesn't matter if that box is 32-bit or 64-bit linux if you don't go with FreeNAS.

Wait, so you are saying that if I go with FreeNAS it does matter if the box is 32-bit or 64-bit but anything else it doesn't matter?

Or was that double negative a typo?

But looking at FreeNAS it looks really promising. iSCSI and NFS support? Booyah.

 

[sabregen]

I have used the following for ESX, and ESXi. Many configs, so I will attempt to container-ize each one:

Initial ESXi config

Supermicro H8DAE
2x Opteron 265s
8GB PC-2100 ECC Reg
HP NetRAID 1-M PCI SCSI U160 RAID controller
HP NetStorage 12 2U enclosure
73GB U160 10k boot drive + VMFS ISO storage
8x18GB SCSI U160 SCA drives in RAID-5 for VM storage
Ran ESXi

Second config

Supermicro H8DC8
2x Opteron 275's
4x4GB PC-2100 ECC Reg
80GB SATA boot + VMFS ISO storage
4x250GB SATA's in RAID-5 for VM Storage

Third config

Sager NP9262 laptop
Q6600 @ 3GHz
3x160GB drives in RAID-5
Windows Vista Ultimate x64
VMWare Workstation 6.5

Nested VMs
2x virtualized ESX servers (running as VMs)
1x OpenFiler VM (iSCSI target as a VM)
1x Virtual Center Server VM (Windows XP VM)

Double nested VMs
Orb Streamer VM (ran on VMFS3 formatted Openfiler VM's iSCSI storage) - Orb + PiMPStreamer
No-IP Dynamic DNS updater VM (Windows XP VM on Openfiler iSCSI storage)
2k3 x64 Enterprise AD controller (2k3 x64 Ent VM on OpenFiler iSCSI storage)

[H]orde Build - brief existence...was sold off

ASUS KFN32-D
2x Opteron HE8346's
4x2GB PC2-5300 ECC Reg
Dell SAS 5/iR SAS RAID controller
2x73GB Fujitsu 2.5" 10k's in RAID-0 (boot + VMFS ISO storage)
Emulex LPE-11000 PCI-X FC 4Gb HBA
LSI MetaStor 2101 2U FC enclosure
10x18gb FC 1Gb HDD's (VM Storage)
Ran ESX 3.5

Next - ESX and ESXi intermix licensed for HA + DRA + Vmotion as a test of ESX/ESXi intermix capabilities

ESX host
Supermicro X7DCA-L mATX dual socket771 boards
2x Xeon E5310's
2x2GB PC2-5300 ECC Reg
QLogic 4Gb FC HBA
2x WD 320AAKS in RAID-1 (boot)

ESXi host
Supermicro X7DCA-L mATX dual socket771 boards
2x Xeon E5310's
2x2GB PC2-5300 ECC Reg
QLogic 4Gb FC HBA
2x WD 320AAKS in RAID-1 (boot)

LSI MetaStor 2101 2U enclosure
10x18GB FC 1Gb HDD's (VM Storage)

 

[sabregen]

Wait, so you are saying that if I go with FreeNAS it does matter if the box is 32-bit or 64-bit but anything else it doesn't matter?

Or was that double negative a typo?

But looking at FreeNAS it looks really promising. iSCSI and NFS support? Booyah.

I don't believe it can do both concurrently, but has the capability to do an either-or type of setup. that being said, you could use VMWare server to run a FreeNAS VM as iSCSI and one as NFS...just a thought.

 

[ben chi(f4)]

HP ProLiant DL380 G5

6 NICs
1.06 TB SAS Raid0 (Raid0 because I don't care about fail)
2 Intel Xeon Quad Core Processors @ 3mhz

 

[Jay_2]

right now I have a really old Dell 1600SC with 2 x 2.4GHz Xeons 1 GB RAM and 3 x 36GB SCSI in RAID 5.

I curently have an exchange server, DNS server and a basic XP and 2003 test image for messing around with. I am soon moving to a monster of a system though as ESXi is really addictive!

 

[agrikk]

I don't believe it can do both concurrently, but has the capability to do an either-or type of setup. that being said, you could use VMWare server to run a FreeNAS VM as iSCSI and one as NFS...just a thought.

I'm not planning on using both concurrently. I'll be doing one or the other, but I like having the option to do either/or.

 

[FibreChanMan]

It's not at home (new infant at home, so no ESXi fun at home ), but it's my ESXi test system at work:

Dell PowerEdge 2850
Dual 3.0ghz Xeon processors
2gb RAM
2x 36gb drives in RAID 1, 3x 146gb drives in RAID 5, and 1x 146gb drive as hot spare (All U320 SCSI)
Dual gig nics
PERC 4 D/i RAID controller

Currently has 1 Windows Server 2003 DC, a Vista 32-bit host and one more Win2K3 box I haven't done anything with yet. Managed by a desktop system running Win2K3 that is also a DC replicating with the VM one.

 

[LittleMe]

Wait, so you are saying that if I go with FreeNAS it does matter if the box is 32-bit or 64-bit but anything else it doesn't matter?

Yes, typo. It would be easier to go with FreeNAS but if you don't, it doesn't matter if you go with a 32-bit or 64-bit install of Linux. ESX is only accessing it across the network, so it doesn't care what it's running as long as it conforms to NFS or iSCSI standards.

 

[agrikk]

Yes, typo. It would be easier to go with FreeNAS but if you don't, it doesn't matter if you go with a 32-bit or 64-bit install of Linux. ESX is only accessing it across the network, so it doesn't care what it's running as long as it conforms to NFS or iSCSI standards.

I think I might try starting with FreeNAS because it seems the easiest route to shared storage, but I've also seen a Linux iSCSI target by Ardis Technologies that I'm curious about. It would be a harder implementation, since I'm a windows guy who knows only the basics about navigating around a Linux box, but that would make it all the more interesting for me.

It also seems like people run VMWare on just about anything, so that's cool.

Right now I'm thinking about:
ASUS P5Q SE Plus motherboard
Intel Xeon X3220 Kentsfield 2.4GHz Quad-core CPU
8GB Crucial Ballistix DDR2 SDRAM DDR2 800 (PC2 6400)
Cheapo PCIe 16x Vid card
Whatever hard drive I have handy

 

[axan]

My esx server specs:

ESX 3.5 U3

2x Intel Xeon e5405 2.0Ghz Quad cpus
Asus DSBF-DE mobo
4 x Kingston 2x2 4GB memory kit
Adaptec 5805 SAS Raid Controller with battery backup
9 x Hitachi Ultrastar 36GB 15K RPM SAS drives, 8 in Raid 10 and 1 as a cold spare
1 x Samsung Spinpoint F1 750GB Sata drive to hold vm templates, iso images and serve as storage for torrent client.
Pioneer slim slot loading dvd drive + supermicro slim ide to sata adapter
2x Intel Dual Port Server network cards
Supermicro 825TQ 2U Rack mountable case with 8x hot swap bays

build log here

 

[xphil3]

a bit dated(almost 2 years), but still does the job quite nicely.
ESX 3.5(soon to be 4.0 FINAL)
2x Intel Xeon R5310 1.6Ghz
Supermicro X7DVL
6x 2GB super talent ECC
4x 250GB SATA1 Raid 5
1x LSI Logic 150-4

also a bit dated
ESXi - Dell 2650
2x Intel Xeon 3.0Ghz
2x 36GB U320
2x 1GB ECC

I wont even post of specs of my 2.5 rigs, lol. Eventually going to do some kind of NAS solution.... Work keeps kicking my ass though.

 

[lopoetve]

4x DL380 G5's.
4x Dell 2950s
1x Clariion CX3-40
1x MSA 1000
1x Netapp 3060 (IIRC).

... ^_^

 

[LittleMe]

4x DL380 G5's.
4x Dell 2950s
1x Clariion CX3-40
1x MSA 1000
1x Netapp 3060 (IIRC).

... ^_^

LOL, that's your "lab" at home or production at work?

 

[axan]

hehe i think that's his production, otherwise we can all go home ..

 

[Gott]

We're running 4xDell 2950's at work.

I don't have an ESX or ESXi machine at my home, though.

 

[REDYOUCH]

I guess I have somewhat of an "old" setup at home:

SuperMicro X5DA8
2x 3.2Ghz XEON
12GB DDR
LSI MegaRAID Sata with 2x80GB drives in RAID1

Gonna throw on some 1 tb drives soon.

Wish I had gone with 64-bit opterons though...

 

[lopoetve]

LOL, that's your "lab" at home or production at work?

... play lab at work

Although I'm looking at picking up a used Clariion and NetApp.

 

[Vito_Corleone]

I just grabbed a Dell GX620 off Ebay for ~$150. The specs are:

P4 2.8ghz
2.5gb of RAM (came with 512, I put the extra 2gb in)
250gb HDD (came with 40gb, I stuck the 250gb in)
GigE

I only got it to play with ESX. I'd like to get some exposure to it.

 

[`danny]

Self built server:

1) Q6600
2) 8GB DDR2 PC26400 RAM
3) 5xSamsung 500GB Spinpoint T series in RAID 5 using Perc 5i
4) Supermicro ATX Server Board (Forgot model #)

 

[shabazkilla]

Specs from my lab:

ESX hosts -
Q6600
Intel BOXDQ35JOE motherboard
8GB DDR2 RAM
1x250GB SATA hard drive
2 x Intel NICs

Shared Storage -
Openfiler 2.3 OS
Q6600 (overkill I know)
8 GB DDR2 RAM
2x250GB SATA drives in software RAID1 (OS)
4x500GB SATA drives in RAID 10 using a Dell Perc5i presented as iSCSI storage

The Intel motherboard works great for the ESX hosts. The only thing that isn't compatible with ESX is the onboard NIC.

 

[geiger]

Homebrew running ESXi:
Supermicro X7DVA-E-O
Dell Perc5i
8GB ECC DDR2(A-Data)
Dual Intel E5430(Quad 2.66GHz)
Supermicro CSE-743T-645B Case
5x 500GB 7200.10
2xOnboard Intel 1000
5x 1gb links between 2 dual port and 1 single port intel nics(1 dual PCIE, 1 dual PCI-X, Single is PCI).

 

[leward]

2x Athlon MP 2400+
Gigabyte GA-7DPXDW-P
3.25GB DDR266
2x Intel e100 NICs
Adaptec 2410sa SATA RAID controller
2x Maxtor 160GB in RAID 0

An oldie, but it's a great way for me to learn ESXi with stuff I already have.

 

[The Spyder]

Dell Precision 690
Dual Xeon Quad 2.33
20gigs FBDimms kingston
2x 640gig AAKS in raid 0 (Everything is backed up on the fileserver)
2 Intel Pro 1000 nics
2 Intel Pro 1000 Fiber nics

 

[jtg1993]

Hey, I have enough money from my bday and christmas to upgrade my server for running vm's I found 4GB ddr2, a E5200 (will overclock) and a Gigabyte GA-EP43-DS3L, does anyone know if the GA-EP43-DS3L is pretty much fully suppored by esxi 3.5? If it isn't I will haveto run server 03 with vmware server.

 

[_tar]

3800+ X2
Some MSI mobo (It's an nForce chipset with an onboard Quadro)
4GB DDR2 800
1x80 GB
1x200 GB
1x320 GB

It's no super server but I work with what I have. The onboard SATA controller is not on the HCL but it is one of those that still works.

I plan to change it around a bit by taking out the hard drives and booting ESXi off USB and throwing the hard drives into an OpenFiler system (Sempron LE-1100, 512mb).

 

[Sniper|3d-R|]

I have an ESX 3.5 server that's been running solid for the past year+. Nothing to special with the hardware... Just stuff I had laying around... I need a better SATA controller for starters.....

P4 3.0Ghz
Mobo: D875BZ
3.5GB Mem
GeForce3 Ti 200
HDD: 
Storage 3 SATA 120's,OS is on an IDE 120GB.
Intel Corporation 82547EI Gigabit Ethernet Controller
Intel Corporation 8254NXX Gigabit Ethernet Controller
Intel Corporation 8254NXX Gigabit Ethernet Controller 
Intel Corporation 8254NXX Gigabit Ethernet Controller

I have just acquired a free Dell Poweredge 2650 from a good friend of mine. I've already added the dual 3.2Ghz xeons, and 2GB of memory. I'm still waiting on storage. This may be my new ESX server, or just another ESX 3.5 server. I have 2 Intel D3368 dual PCI-X NIC's that I got from my buddy as well... These have TCP off loading so very cool indeed!

Thats all for now!

 

[Soulstice]

Mine is in the sig. And, I don't run ESX or ESXi anymore. I've switched fully to Hyper-V Server 2008. It hasn't caught up to ESX just yet (e.g. live-migration, etc.), but it will with the release of Win2K8 R2 (which includes MS's own version of live-migration, and a new dedicated VM disk systems... similar to VMFS).

 

[Soulstice]

ooops, my sig doesn't include drive and chassis info.

Here it is:

Chassis: http://usa.chenbro.com/corporatesite/products_detail.php?serno=41 (4U, 16-drive SATA backplane hot-plug)

Drive Setup:

Hyper-V Server 2008 OS RAID1 Array: 2 x WD Velociraptor 150 GB
VMs Array 1, RAID10: 4 x WD Velociraptor 150 GB
VMs Array 2, RAID10: 4 x Maxtor 300 GB
VMs Array 3, RAID1: 2 x Maxtor 300 GB

I use my environment to support my consulting business, perform test deployments, practice with new technologies (SQL 2008, Exchange 2007, etc.) I have roughly 12 machines running at all times.

 

[sam-sgc]

Q6600
P5Q Pro
12 GB Ram
4x Intel Gbits cards
1x 250gb disk - OS
1x 1tb disk - Backup
1x 320gb disk - SCVMM Library
2x Raid-0 WD 640gb disks - VM Storage
2x Raid-0 120Gb - ISCSI Storage

Pretty fast so far !

Planning on building a 2nd identical box for Host clustering soon.

 

[PurduEE]

I built this server up just before Christmas to run SBS 2003 R2 (which was on my old server) and WHS:

AMD 4850e
Asus M2N-E
8GB Geil DDR2 @ 800Mhz
LSI 8344ELP, LSI SAS3041-e sata controllers in the x16 and x4 slots, respectively
2x Intel Pro 1000 pci-e x1 Gb NICs
4x500GB in RAID-5 on the 8344ELP (VM's stored here)
4x1.5TB on the MCP55
2x1TB on the SAS3041
4GB USB key for ESXi installation.

Total space in WHS is like 7TB, with another 500 or so for SBS. Works outrageously great and total cost was probably less than $1500. I debated quad-core, but this thing sits in a closet, so I went the lower wattage route. If I added some more VMs, I'd probably need to revisit the issue.

 

[LittleMe]

I built this server up just before Christmas to run SBS 2003 R2 (which was on my old server) and WHS:

AMD 4850e
Asus M2N-E
8GB Geil DDR2 @ 800Mhz
LSI 8344ELP, LSI SAS3041-e sata controllers in the x16 and x4 slots, respectively
2x Intel Pro 1000 pci-e x1 Gb NICs
4x500GB in RAID-5 on the 8344ELP (VM's stored here)
4x1.5TB on the MCP55
2x1TB on the SAS3041
4GB USB key for ESXi installation.

Total space in WHS is like 7TB, with another 500 or so for SBS. Works outrageously great and total cost was probably less than $1500. I debated quad-core, but this thing sits in a closet, so I went the lower wattage route. If I added some more VMs, I'd probably need to revisit the issue.

You're almot ahead on drive space. Nice setup and at a great price.

 

[agrikk]

My question about processors is, is it worth it to go quad-core Xeon for a VM host, or will a simple Core 2 Quad suffice?

Can anyone point me to some benchmarks or something for VM performance?


Also,

Q6600
P5Q Pro
12 GB Ram

How do you get 12 GB RAM on a Motherboard with four slots? a pair of 4GB sticks and 2GB sticks? What memory did you use for your 4GB sticks? The ASUS web site says the Max Memory on this board is 8GB: how stable has it been with more than 8 in it?

 

[luie]

Home use:

CPU = Q6600 3.0Ghz
RAM = 8GB OCZ Reaper HPC PC2-6400
HDD = 750GB SATA2
Motherboard = Gigabyte P965-DS3 rev3.3

running 4 VMs. 2 Linuxes, 2 Windows.

 

[Dangman]

The ASUS web site says the Max Memory on this board is 8GB

Incorrect. The max RAM on the Asus P5Q Pro is 16GB:
http://www.asus.com/products.aspx?modelmenu=2&model=2269&l1=3&l2=11&l3=709&l4=0

4 x DIMM, max. 16GB, DDR2 1200 / 1066 / 800 / 667 MHz, non-ECC, un-buffered memory

 

[sam-sgc]

My question about processors is, is it worth it to go quad-core Xeon for a VM host, or will a simple Core 2 Quad suffice?

Can anyone point me to some benchmarks or something for VM performance?


Also,



How do you get 12 GB RAM on a Motherboard with four slots? a pair of 4GB sticks and 2GB sticks? What memory did you use for your 4GB sticks? The ASUS web site says the Max Memory on this board is 8GB: how stable has it been with more than 8 in it?

P5Q Pro support 16GB of ram

It's actually pretty stable.. even overclocked to 3.2Ghz

Sam

 

[lizardking009]

Do you get ESXi health status information for your RAID arrays on the PERC4?

It's not at home (new infant at home, so no ESXi fun at home ), but it's my ESXi test system at work:

Dell PowerEdge 2850
Dual 3.0ghz Xeon processors
2gb RAM
2x 36gb drives in RAID 1, 3x 146gb drives in RAID 5, and 1x 146gb drive as hot spare (All U320 SCSI)
Dual gig nics
PERC 4 D/i RAID controller

Currently has 1 Windows Server 2003 DC, a Vista 32-bit host and one more Win2K3 box I haven't done anything with yet. Managed by a desktop system running Win2K3 that is also a DC replicating with the VM one.

 

[silvercas]

this guy had a nice setup

http://www.tweak3d.net/forums/tech/building-vmware-esx-lab-37782

 

[Luzer]

Asus P5M2/SAS
Xeon X3220(Quad Core) @ 2.4Ghz
4GB Adata Ram
2 750GB Seagate 7200.11 in raid 1

~10 VMs 5 VMs for personal testing 3 VMs for production others are for "global" testing.

I think i might get another 4GB of ram and a 2 port gig PCI-X network card. Any suggestions on the network card?