Your home ESX server lab hardware specs?

PRAVIN1CA said:
Hello Everyone...

Thanks everyone for sharing wonderful labs detail & information.

Please guide me to make choice for homelab

CPU - i7-2600 (Not Sure i5 support direct i/o)

Motherboard...(Which one best for performance & direct i/o support)

BIOSTAR TZ68K+ or Intel DQ67SWB3

i am also thinking of AMD but no idea which CPU & MB will give me same performance & compatibility...

Learning Wishlist OS / APPS on vSphere 5

Linux + Oracle
Linux + Oracle RAC
W2K8 R2 + MS CLUSTER + MSSQL
Solaris x86 + Sybase ASE

Budget is also concern due to indian currency getting depriciated day by day

Thanks:)
Click to expand...

i7 2600 and i5 2500 (both non-k's) support directed i/o i.e. vt-d, but the real hang up for those are finding motherboards that play nicely and actually have a working vt-d implementation. Officially only the Q67 chipset will support vt-d with those, although some people have had luck on some of the 1155 ASRock motherboards. Same type of thing goes for the desktop AMD boards - formally, all the 900 series boards (like 990fx) should support IOMMU (amd's version of vt-d), but the reality is that many people have had issues because the IOMMU tables on specific motherboards aren't fully implemented.

In any case, if vt-d/iommu is a definite must for your build, make sure to do some extra checking to verify its working properly before you purchase.
 
ESXi 5 (Free Licence)

Shuttle SH67H3 i7 2600
16GB G-Skill
60GB OCZ Vertex 3
60GB A-Data S511
Zotac GT 520 1GB
1Gbe Broadcom NIC (replaces on-board Realtec tat)
(Replaced 92mm fan for quieter BB one)

Plex Media Server (Ubuntu Server)
PS3 Media Server (Sabayon/Gentoo)
Sandbox (Sabayon/Gentoo)
vCMA
 
vmware-1.jpg


Running a Intel Core i5-2400 Sandy Bridge 3.1GHz on a Intel BOXDH67BLB3 LGA 1155 motherboard (no pass through) and 32gb of ram.
 
So this is my build:

Cooler Master Elite 310 case
Seasonic SS-500ET 500W 80 PLUS BRONZE Power Supply
Intel DQ67SW board
Intel i7 2600
4xGSkill 8GB DDR 3 1333 memory modules
Two Intel Gigabit desktop adapter network cards
Adaptec 29320LP scsi LVD controller connected to external LTO 2 tape drive
OCZ 30GB SSD for ESXi
2TB WD Green drive for local data store
500GB Seagate drive for additional data store
 
ashman said:
Any reason you went with the 2600k instead of the 2600?
Click to expand...

I didn't really know I was building a vm server when I bought the hardware (and the processor is out of my desktop after I upgraded it to the 2700k), I was just planning to run a whs2011 box. but after reading a few things I decided running multiple vms would work out better for me.
 
PRAVIN1CA said:
Hello jjandrob,

Which NIC you are using??? Onboard is supported???

Thanks
Click to expand...

I had to use an iso maker that added the driver to the software install cd. Software package is called ESXi-Customizer. Simple google search will help u with instructions and driver download. Search for the model of the network card and esxi 5
 
ESXi Testbox (1U dual PSU whitebox)
1 x Core2Quad Q6600 @3.6Ghz w/MSI P35 Platinum
4 x 2GB DDR2 800 (I'm giving her all she's got Captain!)
1 x 8GB C10 SD (ESXi Boot)
1 x Gbit NIC Realtek (Management Network)
1 x 4-way Intel Gbit (SAN & Server Networks)

ESXi Stable Box (1U Dual PSU whitebox):
2 x 12 core Opteron 6238 G34 w/ASUS Dual Socket G34
192GB RAM
1 x 8GB C10 SD (ESXi Boot)
2 x Intel Gbit onboard
2 x 4-way Intel Gbit

Solaris SAN/NAS: (2x 4U Norco 24 bay boxes)
2 x 12 core Opteron 6238 G34 w/ASUS Dual Socket G34
256GB RAM
2 x IBM M1015 w/IT firmware
2 x HP SAS Expander
2 x 120GB SATA (Boot/Logging) (Mounted with: http://www.auspcmarket.com.au/assets/804x537m/products/e5406b07110c540bb5d4ae2478236ec4-0.jpg)
14 x 3TB 3.5" SATA
10 x 320GB 2.5" SATA
8 x 60GB 2.5" SSD SATA (6 Zil/2 L2ARC)
2 x Intel Gbit onboard
2 x 4-way Intel Gbit

If I can find another nice deal on 3TB disks, might buy more to fill out the 2 boxes and replace the 320GB's, but I bought the other ones before the great flood of HDD doooom ...

Still not sure if I'll switch the 6238 (12 core @2.6Ghz) with the 2 6276 (16 core @ 2.3Ghz) I have in, either ESXi or the fileserver.

On my workstation I kinda need per CPU performance more then 8 extra cores which the 300Mhz per core gives, and I also OC tested the CPU's and one pair of the 6238's runs with some substantial Vcore decrease, which should mean it can run at much higher clocks.

But for that to be of any use, I need to find a Dual Socket server/workstation motherboard that has overclocking features ...
 
Last edited:
nerdalertdk said:
why 256GB ram in a file server ?
Click to expand...

2 reasons, 1 because I had the ram and 2 because I'm using Dedup and Compression on several ZFS volumes, ZFS just noms away at any ram you give it.

And I'm moving tons of data back and forth, so huge caches are very handy.
 
My soon to be home built lab consists of the following hardware

Mini-Box M350
picoPSU-150-XT + 102W Adapter Power Kit
JetWay JNF9A-Q67 With 2 onboard Intel GB PCI-E Nics
I5 2400 from Microcenter
16GB DDR3 memory
16GB Samsung SSD(ebay deal) for ESXi install + local images and other misc storage
NFS network datastore from Nexenta NAS box

It this build goes well, I hope to build a second idetical one to mess with clustering, vmotion, and some of the new features that 5.0 has to offer
 
I'm getting ready to upgrade my ESXi server.

Currently it's running on an AMD E-350 system I built.

I'm not upgrading because I am unhappy with the performance. On the contrary it has been chugging along running my pfSense and linux server VM's just fine for some time now.

I want to upgrade because it doesn't support IOMMU. I want to try to assign my Intel dual port NIC to the pfSense VM to see if that improves latency and other performance through to the WAN.

I've been considering the following parts:

Any thoughts on the above?

Also, has anyone found any 1866Mhz unregistered ECC DDR3? AMD's specs state that this is the officially supported RAM for the platform, but I can't seem to find this ram anywhere. Right now all I can find is CL9 1333Mhz, and CL11 1600Mhz. If forced to chose between these, not sure if I prefer the tighter timings or higher clocks...
 
New and shiny:

Supermicro X9DR3-F
1x Intel Xeon E5-2609
4x Kingston 8GB reg ECC 1.35V (KVR1333D3LD4R9S/8GHC)
IBM M1015 (LSI 9211-8i P13 IT mode, is flashable from UEFI shell, works fine)
Seasonic X-Series X-760 (230V input)
1x Seagate 1.5TB (yes, THAT one, but it works. 5 more will be added which are currently running in the old box)

Box draws 50W with the above components when idle with Backtrack R2 booted natively.

Edit: 90W at 100% load.
 
Last edited:
ESXi 5.0 Update 1 Home Lab ready to go tonight! (hopefully)

SuperMicro X9SCM-F-O
Xeon E3-1230
16 GB KVR
Intel E1G42ETBLK x2
OCZ Vertex 3 60 GB
Seasonic X400 Fanless
Lian Li V351B

HP ProCurve 1810-24g

Synology DS1512+ with 5 x 2TB Seagate ST2000DM001

Planned VMs:
WS 2008 R2 (Domain and DNS)
WHS 2011
pfSense
 
RezQme said:
ESXi 5.0 Update 1 Home Lab ready to go tonight! (hopefully)

SuperMicro X9SCM-F-O
Xeon E3-1230
16 GB KVR
Intel E1G42ETBLK x2
OCZ Vertex 3 60 GB
Seasonic X400 Fanless
Lian Li V351B

HP ProCurve 1810-24g

Synology DS1512+ with 5 x 2TB Seagate ST2000DM001

Planned VMs:
WS 2008 R2 (Domain and DNS)
WHS 2011
pfSense
Click to expand...

Please let us know how you like that low powered Xeon E3. I was considering one of these, but then I decided that I'd rather support AMD :p

I'm also curious, why WS and WHS? They don't do anything that linux doesn't do better and with fewer resources, provided you aren't looking to run an Exchange server or something. :p
 
Usually I swap back and forth between Intel and AMD, but my last couple builds have been Intel :-\ I need to build an AMD box to even out the competition...

I will post tomorrow about how the E3 handles the few VMs I'm going to put on it tonight.

Quite honestly, I would love to be more familiar with Linux because I know it does use fewer resources. I was going with the Windows OSes because I am familiar with them and have a Technet Subscription. I've messed with Linux a little but need to really school myself on CLIs. :eek:

Not interested in exchange or anything like that, was going to run active domain, DNS, and do some media file serving.

Can you suggest a good distro (for the listed purposes above) and some resources to jump into?
 
RezQme said:
Can you suggest a good distro (for the listed purposes above) and some resources to jump into?
Click to expand...

Distro's are easier than resources, so I'll start with those.

I usually use Ubuntu server edition. I no longer care for their desktop version because of questionable choices in GUI, but the server edition is GUI free by default so it doesn't really matter. Ubuntu has done lots in the way of default hardware driver support, so you'll spend less time getting hardware to work, which is a plus.

Debian is probably equally good (very similar system, Ubuntu is based on Debian) but I don't have as much experience with it.

Other people also like RHEL (Red hat Enterprise Linux) and CentOS (which is based on it), but I don't have much experience with those.

As far as resources go, at least with Ubuntu there are a lot of guides you can easily google that give you step by step instructions. Some are in Ubuntu's official help files, others are written by end users. They are usually very helpful.
 
Thanks for the info! I sent you a PM so as not to hijack. I'm new here, not sure how touchy the mods are about that..
 
Zarathustra[H];1038628255 said:
Please let us know how you like that low powered Xeon E3. I was considering one of these, but then I decided that I'd rather support AMD :p

I'm also curious, why WS and WHS? They don't do anything that linux doesn't do better and with fewer resources, provided you aren't looking to run an Exchange server or something. :p
Click to expand...

never seen WHS machine backup? its pretty nice.
 
The E3-1230 is doing great so far, eating up everything I throw at it. This box puts out noticeably less heat than any of my other boxes. (The worst is my i7 920 machine, but a lot of that is the GPU)
 
RezQme said:
The E3-1230 is doing great so far, eating up everything I throw at it. This box puts out noticeably less heat than any of my other boxes. (The worst is my i7 920 machine, but a lot of that is the GPU)
Click to expand...

The 1230 is a great chip imo. When I was testing it out, it handled most everything without breaking much of a sweat, and was a pretty decent deal for what it provides. Just wish the platform was a little more robust.
 
hey guys I'm wondering what's the cheapest way to do this with nested virtual machines? Is the 1045t a better choice than a 2500?
 
I just went thru the last 15 pages of this thread trying to find a decent whitebox solution to build for myself in a home lab setting.

I have changed my mind so many times in the last three weeks on what type of system to purchase.

Could I get some recommendations for affordable ESXi 5.0 compatible motherboard / CPU / RAM / to purchase, more supported NICs the better.

I am looking to build a single host with 16 - 32 GB of RAM, with 4x 2 TB WD Green HDD for a file server - These don't have to be in any RAID level but I may need a SATA/SAS controller if the Motherboard SATA ports don't work in ESXi. Budget of hopefully under $1500.
 
josephdyland said:
I just went thru the last 15 pages of this thread trying to find a decent whitebox solution to build for myself in a home lab setting.

I have changed my mind so many times in the last three weeks on what type of system to purchase.

Could I get some recommendations for affordable ESXi 5.0 compatible motherboard / CPU / RAM / to purchase, more supported NICs the better.

I am looking to build a single host with 16 - 32 GB of RAM, with 4x 2 TB WD Green HDD for a file server - These don't have to be in any RAID level but I may need a SATA/SAS controller if the Motherboard SATA ports don't work in ESXi. Budget of hopefully under $1500.
Click to expand...

If you don't have any other major concerns or upgrade desires, that's definitely doable. I'd look at the E3-1230 xeon's with a supermicro board and either 4x4GB UDIMM's (or if you need the 32, pay the premium finding 4x8GB). This should allow passthrough of a SAS controller card for a file server guest vm without too much trouble. Including drives you would still come in well below your budget.
 
I just completed my new build.

  • ASUS MAXIMUS V GENE
  • Intel Core i7 3770 Ivy Bridge
  • Intel X25-M SSD 80GB
  • Corsair Force Series GT 240GB SSD
  • Corsair Builder Series CX500 V2
  • Corsair 32GB DDR3 1600Mhz Dominator
  • Windows Server 2012 Beta, Hyper-v 3
  • Lian Li PC-V354B Black

Pictures taken with Nokia Lumia 800, seems to be OK.

wp000044k.jpg

wp000057q.jpg

wp000042v.jpg

wp000050.jpg

wp000051c.jpg

wp000049.jpg

wp000055em.jpg

wp000039x.jpg

wp000029q.jpg

wp000023y.jpg

wp000021y.jpg
 
Mainly used for testing, but they also work as my backup servers if my VPS's go down, dosent need to be high priority/high connection speed. Just needs to work :)

Its the two bottom systems, the top system is a media pc

Master ESXi Spec:
Case: X-case 2U 550mm
Mobo: Asus P8B-WS
CPU: Intel Xeon 1245
RAM: 16GB Corsair DDR3
PSU: Tagan UG01 480w
NIC1: Intel dual port gigabit PCIe (passed through to VM)
NIC2: Intel single port PCI (passed through to VM)
HDD1: 320GB WD 2.5"
HDD2: 320GB WD 2.5"
OPTICAL: LG Bluray reader/writer (not passed through yet, as not had time to set this up!)

Slave ESXi:
Case: Xcase 24 SAS bay 550mm
MOBO: Asus P8B-WS
CPU: Intel Xeon 1245
RAM: Corsair 32GB DDR3
PSU: Corsair HX650
HDD1: OS drive 120GB Samsung 2.5" 7200rpm
GFX1: ATI HD4350 (passed through to windows 7 VM, work in progress)
RAID1: Adaptec 5805
RAID2: 3ware 9690SA-8I
HDDs:
RAID6: 8x 3TB Hitachi 5K3000
RAID5: 4x 1TB WD Green (not fully setup yet)

IMG_5262.jpg


IMG_5274.jpg
 
Here's my current ESXi 5.0.0 rig.
Motherboard: AsRock Z68 Professional G3
CPU: Intel Core i5 2500 / 3.3 GHz processor
Case: Fractal Design XL
RAM: 32GB (4x8GB) Corsair XMS3 DDR3 (CMX8GX3M1A1333C9)
Cooling: Corsair Hydro Series H70 CORE High Performance Liquid CPU Cooler
PSU: Corsair Enthusiast Series TX650 V2
NIC: Built in Realtek 8111-E (I think)
GPU: Builtin + Radeon HD 7870 (Trying VT-D with no luck atm)

Storagewise:
The following are on motherboard SATA inside a nice Chieftec 4 in 1 (ATM-1042S), only thing to complain about is that its made in plastic and can be a bit fiddly to get all the drives to lock in place, and handled by ESXi.
1x Kingston 64GB SSD (had it lying around)
1x Intel Solid-State Drive 320 series - 180 GB - SATA-600

On SATA but RMD to a Solaris for L2ARC & ZIL.
2x Intel Solid-State Drive 510 series - 120 GB - SATA-600

Then I have my controllers that are done with passthrough to Solaris.
Intel SASUC8I with 5x2TB drives in raidz-1
LSI SAS 9211-8i with 6x3TB Seagate 7200 rpm drives in raidz-2.

IMG_0137.jpg

IMG_0138.jpg


I've just ordered 32GB of RAM to replace the 16 I have since I need more.
Also a Radeon 7870 to play around with VT-D and see if I can get that up and running. If I can't I'll just continue using a dedicated gaming computer. If I get it to work I'm ordering a new i7 3770, 32GB of ram, chassi etc and making a dedicated ESXi machine and use the current one only for storage with Solaris.
Plus I probably will get two 10gbit cards.
 
Last edited:
vr00mvr00m said:
Here's my current ESXi 5.0.0 rig.
Motherboard: AsRock Z68 Professional G3
CPU: Intel Core i5 2500 / 3.3 GHz processor
Case: Fractal Design XL
RAM: A-Data XPG Ddr3 16Gb (8Gbx2) 1333Mhz Cl9-9-9-24, Low Power 1.35V, Game Series
Cooling: Corsair Hydro Series H70 CORE High Performance Liquid CPU Cooler
PSU: Corsair Enthusiast Series TX650 V2
NIC: Built in Realtek 8111-E (I think)

Storagewise:
The following are on motherboard SATA and handled by ESXi
1x Kingston 64GB SSD (had it lying around)
1x Intel Solid-State Drive 320 series - 180 GB - SATA-600

On SATA but RMD to a Solaris for L2ARC & ZIL.
2x Intel Solid-State Drive 510 series - 120 GB - SATA-600

Then I have my controllers that are done with passthrough to Solaris.
Intel SASUC8I with 5x2TB drives in raidz-1
LSI SAS 9211-8i with 6x3TB Seagate 7200 rpm drives in raidz-2.

Had no good pictures to use atm.

I've just ordered 32GB of RAM to replace the 16 I have since I need more.
Also a Radeon 7870 to play around with VT-D and see if I can get that up and running. If I can't I'll just continue using a dedicated gaming computer. If I get it to work I'm ordering a new i7 3770, 32GB of ram, chassi etc and making a dedicated ESXi machine and use the current one only for storage with Solaris.
Plus I probably will get two 10gbit cards.
Click to expand...

Post pictures.
 
dashpuppy said:
pics :)
Click to expand...

I'll be doing a full review soon. Getting it setup now. Put (4) Crucial 128GB M4 SSDs in it in a RAID0 set to test it... Was saturating a single Gb link (all I had connected) with 4KB I/Os getting 25K IOPs. Got 32K IOPS with 512byte I/Os. Copying all my data to my DS1511+ so I can pull the drives from my lab DS1010+. Going to fill the rest of the 8 bays with 7200RPM 1TB WD drives.

It's a good setup...not cheap, but powerful. Emulex sent me some 10Gb NICs a while back which are on the HCL for this so it'll be running at 10Gb for some testing. It's just as quiet as my smaller boxes..has 4 DIMM slots so can do 8GB. I might order some RAM for it.

It's a loaner unit from Synology..hoping I can work a good deal to keep it.
 
NetJunkie said:
I'll be doing a full review soon. Getting it setup now. Put (4) Crucial 128GB M4 SSDs in it in a RAID0 set to test it... Was saturating a single Gb link (all I had connected) with 4KB I/Os getting 25K IOPs. Got 32K IOPS with 512byte I/Os. Copying all my data to my DS1511+ so I can pull the drives from my lab DS1010+. Going to fill the rest of the 8 bays with 7200RPM 1TB WD drives.

It's a good setup...not cheap, but powerful. Emulex sent me some 10Gb NICs a while back which are on the HCL for this so it'll be running at 10Gb for some testing. It's just as quiet as my smaller boxes..has 4 DIMM slots so can do 8GB. I might order some RAM for it.

It's a loaner unit from Synology..hoping I can work a good deal to keep it.
Click to expand...

10gbe YUM! won't you need a switch for that or you doing a direct connection to the server ?
 
NetJunkie said:
I'll be doing a full review soon. Getting it setup now. Put (4) Crucial 128GB M4 SSDs in it in a RAID0 set to test it... Was saturating a single Gb link (all I had connected) with 4KB I/Os getting 25K IOPs. Got 32K IOPS with 512byte I/Os. Copying all my data to my DS1511+ so I can pull the drives from my lab DS1010+. Going to fill the rest of the 8 bays with 7200RPM 1TB WD drives.

It's a good setup...not cheap, but powerful. Emulex sent me some 10Gb NICs a while back which are on the HCL for this so it'll be running at 10Gb for some testing. It's just as quiet as my smaller boxes..has 4 DIMM slots so can do 8GB. I might order some RAM for it.

It's a loaner unit from Synology..hoping I can work a good deal to keep it.
Click to expand...

I was going through your twitter feed about this thing. Simply amazing. Nice benefits you got there. ;)

That's almost enough for me to get another QNAP and put those in. Not like I need the IOPS, but when they're that cheap why not. Kinda funny to be pushing more IOPS than most companies in a home lab. I've been contemplating purchasing a Synology to see what all the fuss is about. My QNAP just seems to do whatever I want it to do so I haven't really looked elsewhere.
 
You must log in or register to reply here.
Back
Top