Your home ESX server lab hardware specs?

New storage for the updated Lab is built, Nexenta 4.02 Installed.

 
What is the drive subsystem/layout?
Click to expand...

Supermicro CSE-216A-R900LPB 24 x 2.5" Bays, SAS/SATA 6GB Backplane-Redudant 900Watt PS.
Supermicro X8SI6-F (On board LSI2008)
Intel Xeon 3440 w/Thermaltake LP HSF
4 x 8GB Kingston ECC Registered 1333
1 x IBM 1015's Flashed IT Mode
2 x Samsung 830 Pro's 256GB - ZIL
2 x Corsair Force 120GB - L2ARC
2 x Corsair Force 60GB - Onboard Controller Mirror Nexenta/OS
12 x Seagate 2.5 1TB 64MB Cache Hybrid 8GBSSD Drives
Intel Pro Dual PT/1000 Low Profile Controller (4 x 1Gb NIC's, 1 x 10/100Mb iPMI)
 
I have a Dell 1950
2x2core Xeon
12gb memory
6x146gb RAID 5 Internal Drive
Dual Gigabit

Dell 2950
2x4core Xeon
24gb memory
4x73gb RAID5 Internal Drives
Dual Gigabit

ESXi 5.5 on both, vCenter 5.5 Server for managing

I end up rebuilding the vCenter server every 90 days with a new evaluation copy, but I only run about a dozen VMs, so I just need to cleanly shut all of them down and rebuild the Windows VM with vCenter and rejoin them to it. Pain, but I am not rich.

I have messed around with a separate Windows server (white box) that runs Starwind iSCSI software for serving drive space that can be shared for vMotion and migrations, seems to run fairly well for free software. Anyone tried CentOS iSCSI initiator or (cringe) the Windows one?
 
mourngrym1969 said:
I have a Dell 1950
2x2core Xeon
12gb memory
6x146gb RAID 5 Internal Drive
Dual Gigabit

Dell 2950
2x4core Xeon
24gb memory
4x73gb RAID5 Internal Drives
Dual Gigabit

ESXi 5.5 on both, vCenter 5.5 Server for managing

I end up rebuilding the vCenter server every 90 days with a new evaluation copy, but I only run about a dozen VMs, so I just need to cleanly shut all of them down and rebuild the Windows VM with vCenter and rejoin them to it. Pain, but I am not rich.

I have messed around with a separate Windows server (white box) that runs Starwind iSCSI software for serving drive space that can be shared for vMotion and migrations, seems to run fairly well for free software. Anyone tried CentOS iSCSI initiator or (cringe) the Windows one?
Click to expand...

Windows 2012 iSCSI actually works pretty good.
 
Alright,

My system has changed enough since my last post, that it is worthy of an update.

First off, unlike many of you, mine is not a learning lab, but rather a single home production server which I went with primarily for consolidation.

Here's a shot of my setup (which is probably sufficiently ghetto to make those of you in the IT profession cringe)

14468395409_a1eace2590_b.jpg


In this picture from top left:

Black box, very top left: Ceton InfiniTV 6 channel TV tuner (used with mythTV)
Switch to right of tuner: HP Procurve 1810G-24
Below Switch: Patch panel for wired Ethernet throughout house.
Below panel: Verizon FiOS Optical Network Terminal (ONT) supporting 150Mbit/65Mbit internet and Cable TV.
Further right: My ESXi server in a ghetto black NZXT case I got cheap on Newegg.
Further right: Small 800x600 $30 LCD screen I use on the rare occasion I need console access to server.

Server specs:

AMD FX-8350 8 core at stock speeds
32 Gigs of DDR3 RAM at 1600Mhz
Three dual port Intel Pro/1000 PT NIC's
One Broadcom NetXtreme single port NIC
IBM M1015 SAS controller, with (soon) 8x 4TB WD RED drives (right now a couple of greens still remain, but they are being swapped out.)
Datastore/Boot: Samsung 840 Pro SSD, plus an old WD Blue hard drive for backups.
My old PCI 2MB Matrox Millenium SVGA card for console output when needed. (i'm ecstatic I'm still getting some use out of this thing :p )

14468456239_a27de6e78d_b.jpg


Guests:

pfSense
Handles Routing / firewall for house. Has a dedicated direct I/O forwarded dual port Intel NIC (one port for WAN, one for LAN) WAN is connected to FiOS ONT, LAN to switch. Is assigned two cores and 256MB of RAM.

FreeNAS
Is - well - the NAS for the house. 8x drives (currently 4x 4TB WD Red's, and 4x aging 3TB WD Greens, which are being replaced one by one) in a RAIDz2 configuration. The SAS controller is direct I/O forwarded to it, as is a dual port Intel NIC, set up with link aggregation/lagg/trunking/"whatever term you prefer" to the switch. It also connects internally using a VMXnet3 10gig link to a separate subnet dedicated to storage traffic between the servers. It is assigned 25Gigs of RAM, and 4 cores.

MythBuntu Dedicated Backend
Handles DVR duties recording shows to the NAS via internal 10gig VMXnet3. Database and liveTV ringbuffer is recorded to SSD for performance. Has a direct I/O forwarded Intel NIC in link aggregated mode to switch. Assigned 2GB of RAM.

Ubuntu Server
Handles basic server tasks. Runs Ubiquiti Unifi server software, used for wgets and rtorrent to storage. Will be used for cloud backup in the future. Connects to FreeNAS using vmxnet3 10gig and has a direct I/O forwarded single port NetxTreme NIC.

I have two vswitches set up. One which just connects the management console to the on-board Realtek NIC, and the other on a sseparate subnet from the house network, which is for internal storage traffic between FreeNAS, Ubuntu Server and Mythbuntu Backend.


I am in desperate need of some future upgrades. As mentioned above, I am already swapping out my aging green WD drives for 4TB WD Reds one by one, but I am also short on RAM, which is a problem, as with 4 8GB sticks I am currently maxed out. FreeNAS really likes to have a base of 2GB plus 1GB of RAM per TB of storage. Once I hit 8x4TB, I would thus need to have 34GB of RAM dedicated to it... My other guests all have less RAM than I would like to grant them, in order to give FreeNAS as much as I can give it. (I'd like pfSense to have 512MB to a gig, my Ubuntu server to have 2 Gigs and my Mythbuntu backend to have 4gigs)

I am hoping to find some of those elusive 16GB ECC UDIMM's to bring me up to 64 which should last a while.

It would be nice to get a server motherboard that would give me more slots so I can use gobs of ECC UDIMMs or the larger RDIMMs. I could probably afford a entry level server motherboard, but the ones that support AM3+ CPU's all have WAY too few PCIe slots (I at least 5). If I get a non AM3+ board I need to buy a server CPU as well, then the budget goes up, and the clock speed goes down :(

If anyone is aware of an AMD 3000/3300 AM3+ server board that will take a FX CPU and has 5+ PCIe slots in an ATX compatible form factor, you could become my next best friend :p

I could also really use a UPS, which I haven't gotten around to setting up yet.

Step by step as budget allows I guess...
 
Last edited:
there's 4 3TB drives in the FS/FT threads going for 260 for the lot...

Not my thread but thought I'd give the seller a bump

Pretty good setup you got there though, are you not running SickBeard?
 
gigatexal said:
there's 4 3TB drives in the FS/FT threads going for 260 for the lot...

Not my thread but thought I'd give the seller a bump

Pretty good setup you got there though, are you not running SickBeard?
Click to expand...

Thank you!

I actually had a contract end on me a few weeks back, and I don't start my new job until next week, so my tech budget is pretty limited at the moment, or I'd definitely look at those drives in FS/FT.

I wasn't familiar with Sickbeard. I just googled it and it looks pretty awesome. Will have to look into it. Right now I'm getting everything I want to watch using MythTV, but that looks like an awesome setup to consider.
 
Hah, good luck. I'm buying a house soon, so perhaps a basement datacenter is in the works for me too.
 
Just picked up one of these. Been holding out, but couldn't resist for the price:

6pglj.jpg


192GB's RAM, 64 Logical Cores...
 
Some awesome otherworldly like setups in here!

I'm slowly building up my lab having sold off alot of it a couple of years ago.

2 x Dell R410's with 2 x Intel Xeon E5530's, 16GB RAM, Quad Gigabit NICs, No internal storage ATM - USB booted to ESXi 5.5. ---> ISCSI based.

1 x Custom Whitebox with AMD FX 4GHz 8C CPU, 32GB RAM, 2 x Intel 330 170GB SSDs, 5 x Gigabit NICs, USB booted to ESXi 5.1 (awaiting upgrade) ---> ISCSI based.

1 x Custom Whitebox with AMD 3.2GHz Quad Core CPU, 16GB RAM, Quad Gigabit NICs, 2 x 2TB HDD RAID 1, 2 x 1TB HDD RAID 1 running FreeNAS - needs upgrades.

1 x Dell 5524 switch providing ISCSI via LACP.and VLANS.

An assortment of Cisco routers and switches but nothing outrageous like a Nexus! Also have a Cisco fibre switch (4Gbps) for fibre testing / labs.

Had Infiniband but got bored of it and sold it off along with some nice Cisco switches that I wish I'd kept now.

Considering freeing up the FreeNAS for a Synology 1813+ full of 240GB SSDs, but havent decided yet. I like how the Synology has VMware VAAI support amongst other things.
 
In the process of getting a home lab setup.

I have 2x Dell R300, both with 24GB RAM and 2x 146GB drives.
Both have a 4-core Xeon. One at 3.16Ghz, and one at 2.66Ghz

The 3rd machine is an IBM x3650 7979 with 48GB RAM, 8x 146GB drives, and 2x 3.16Ghz 4-core processors.
 
Zarathustra[H];1040958589 said:
Thank you!

I actually had a contract end on me a few weeks back, and I don't start my new job until next week, so my tech budget is pretty limited at the moment, or I'd definitely look at those drives in FS/FT.

I wasn't familiar with Sickbeard. I just googled it and it looks pretty awesome. Will have to look into it. Right now I'm getting everything I want to watch using MythTV, but that looks like an awesome setup to consider.
Click to expand...

Might want to check out CouchPotato for movies and headphones for music too ;)
 
Vader said:
Just picked up one of these. Been holding out, but couldn't resist for the price:

6pglj.jpg


192GB's RAM, 64 Logical Cores...
Click to expand...

I keep telling myself that if I had somewhere to run it I would pick one of these up. Unfortunately, my Norco 4220 is already too noisy... couldn't imagine one of these in my home office. Maybe one day when I move to a climate that allows for basements and/or racking equipment in the garage. Houston area doesn't really work for either option...
 
It's going in a little Datacenter build at my brother's business, AC, UPS, etc..and you're right..it IS loud.
 
Zarathustra[H];1040958589 said:
Thank you!

I actually had a contract end on me a few weeks back, and I don't start my new job until next week, so my tech budget is pretty limited at the moment, or I'd definitely look at those drives in FS/FT.

I wasn't familiar with Sickbeard. I just googled it and it looks pretty awesome. Will have to look into it. Right now I'm getting everything I want to watch using MythTV, but that looks like an awesome setup to consider.
Click to expand...

I'd also check out NZBDrone, it has some of features sickbeard is missing, and they plan on adding Movie and torrent support.
 
My lab hardware specs
2 x HP ML110 (E3/1220, 32GB RAM, USB boot)
1 x Intel i3 @ 1.8 ghz NUC, 16GB RAM, USB boot
1 x Mac Mini i5 @ 2.3 ghz, 16GB RAM, USB boot
..... all running ESXi 5.5 release.

Storage:
Synology 211, 2 x 1.5TB 7200rpm SHR/RAID1 disk
Iomega/Lenovo IX2-DL, 2 x 3.0TB 7200rpm RAID1 disk
NFS shares for all

Used for home media, but could also be used for NFS/ESXi:
Synology 411, 4 x 2.0TB 7200rpm SHR/RAID5 disk

Obviously enough, disk IO is miserable - simply horrible. I have one 500GB OCZ4 that I've put into the 211 from time to time, but even with that, IO was pretty miserable (as JBOD, obviously - not tying the SSD to the HDD in any way; I quickly ran the CPU to 100% just doing basic IO)

What are some suggestions for what to do these days for good IO? I want something simple (Synology XS is too expensive!) and _fast_, and I'm OK to buy a few more SSDs if that will help. One scenario is to buy 3 more 500GB SSDs for a total of four, find somewhere else to put my media collection (drop it from 6TB to 3TB somehow?) and then make a 1.5TB RAID5 set on the 411 with the SSDs.

I'm _really_ unhappy with IO!
 
Starting to explore the world of ESXI so here is my little setup that I am experimenting with, I hope to grab an R610 down the road.

Dell Precision T3400
Core 2 Quad Q9300 2.53ghz
8 GB DDR2 1066mhz
1 x 160GB 7200rpm
Dual Gigabit NIC's

Running pfsense, Ubuntu Server with Squid soon, and Win 7 (for playing). Not bad for free :).
 
nice job

did you follow a guide getting pfsense working in esxi?
 
gigatexal said:
nice job

did you follow a guide getting pfsense working in esxi?
Click to expand...

I've found pfSense "just works" in ESXi. No need for weird tunables and settings like with FreeNAS.

One thing I have noticed is that using vswitches can add some latency, so I dropped a dual Intel Pro/1000 PT in my box and direct I/O forwarded it.
 
yeah the vswitch thing never made sense to me, I'm not a networking guy

good call on the dedicated nic tho (I would think latency would not be much of an issue if the esxi host had a decent cpu, guess not)
 
gigatexal said:
nice job

did you follow a guide getting pfsense working in esxi?
Click to expand...

I did follow a guide since it was my first time ever setting up pfsense inside esxi, but it was very straight forward and easy. The onboard NIC is the WAN and the dedicated NIC is the LAN. Just make sure to set your ESXI box IP to static in case you need to reboot your pfsense VM if that is where it gets its IP from :D.
 
MSI H67MA-E45 & Sandy Bridge G620 - $45 forum special
16 GB DDR3 RAM - $90 used
Corsair CX430 - $19.99 Newegg special
500 GB WD BLUE - Free, re-purposed
128 GB SSD for SQL - $50
Dual port Intel gigabit NIC - $35 used
Random old re-purposed white box case - free

All in all my ESXi host box cost me about $235 to build. Pulls 37 watts idle, so pretty easy on power, especially considering powersupply. It handles my 8 running VMs with ease and generally only maxes out the CPU during a reboot. The white box is a bit ugly, but it's in ta corner in the basement so it doesn't really matter. All in all, perfect for a home server.

piWjQW5.png
 
stats.jpg

4 nodes:
  • Rackable 1U Dual Xeon L5420
  • 16GB ECC
  • Q-Logic QLE-2460 Fibre Channel Adapter
  • ESXi 5.0U3
  • No local Disks, Boot from SAN
pools.jpg

Storage Server:
  • HP DL380 G2
  • 16GB ECC
  • HP P400 (4x 72GB SAS RAID10 for boot drive)
  • LSI SAS3801E (external SAS storage)
  • LSI SAS 9122-8i (controller for internal SSD's)
  • Q-Logic QLE-2464 4-Port Fibre Channel Adapter (in target mode, serves FC targets to hosts)
  • OI+Napp-It (bare metal)
strg.jpg

Regular VM Storage Pool:
  • HP MSA70 25 drive expander
  • 25x 72gb SAS drives
  • 2x 120GB SSD's for ZIL
media_files.jpg

Media Storage & TV Recording Pool:
  • Rackable SE3016 Expander
  • 5x 2TB drives
views.jpg

FC Views Exported to ESXi Hosts:
  • FC Topology is Point-to-Point
  • Each Host has a dedicated boot LUN, shared ESXi storage and the LUN for Windows Media Center
wsi.jpg

Offsite server from WholeSaleInternet. Used for hosting & DNS, as well as internal backup AD services, and off-site backup server.

ovh.jpg

Offsite server from OVH. Used for backup web services, backup internal AD services and 2nd offsite backup.

total.jpg

Total resources.

Yes, this is my HOME setup.
 
Last edited:
This is what I am currently putting together. I have a rack and have done a few servers and other Cisco equipment, but I am getting to the point where I want current technology with low power and low noise.

-RaidMax CobraZ case
-Antec 500W PSU
-SM X10SLH-F-O Motherboard
-Intel Xeon E3-1231v3
-32GB Kingston ECC
-4X1GBe Intel NIC
-2X1GBe Intel NIC
-3x 256GB Samsung SSD
-2x WD Red 2TB
-1x WD Black 1TB


This build will be used for ESXi 5.5. Probably be in conjunction with my laptop (i5, 16gb, SSD) to run different labs and setups. This setup is backed by a Dell Powerconnect 2824, EdgeRouter Lite, and Unifi AP.
 
My new "lab" esxi server:

HP Proliant DL360 G6
72GB 1333 RAM
Dual X5650 Xeon cpu's (2.66 MHz 6 core)
Synology DS214 NAS for storage
 
Just updated the old trusty all-in-wonder again:

15268538436_a8ac02e4ed_b.jpg


15104817929_f85d18d3a6_b.jpg


As before, in the pic above from top left:

Black box, very top left: Ceton InfiniTV 6 channel TV tuner (used with mythTV)
Switch to right of tuner: HP Procurve 1810G-24
Below Switch: Patch panel for wired Ethernet throughout house.
Below panel: Verizon FiOS Optical Network Terminal (ONT) supporting 150Mbit/150Mbit internet and Cable TV.
Further right: My ESXi server in a Norco RPC-4216 case
On top of server: Small 800x600 $30 LCD screen I use on the rare occasion I need console access to server.

Server Specs:

15291470712_95f030cd32_o.jpg


(Not sure why it says X8DT6, as it is definitely an X8DTE)

Server specs:

Norco RPC-4216 case
2 x Xeon L5640 (6 cores each, 2.27Ghz)
96 GB DDR3 Registered ECC RAM at 1333Mhz
Three dual port Intel Pro/1000 PT NIC's
One Broadcom NetXtreme single port NIC
Two IBM M1015 SAS controller's flashed to IT mode
Datastore/Boot: Samsung 840 Pro SSD, plus an old WD Blue hard drive for backups.

Guests:

pfSense
Handles Routing / firewall for house. Has a dedicated direct I/O forwarded dual port Intel NIC (one port for WAN, one for LAN) WAN is connected to FiOS ONT, LAN to switch. Is assigned two cores and 1GB of RAM.

FreeNAS
Is - well - the NAS for the house. The two SAS controllers are direct I/O forwarded to it, as is a dual port Intel NIC, set up with link aggregation/lagg/trunking/"whatever term you prefer" to the switch. It also connects internally using a VMXnet3 10gig link to a separate subnet dedicated to storage traffic between the servers. It is assigned 72Gigs of RAM, and 6 cores.

Storage is configured as follows:

Code: 
	NAME                       
	zfshome                    
	  raidz2-0                 
	    Western Digital RED 4TB
	    Western Digital RED 4TB
	    Western Digital RED 4TB
	    Western Digital RED 4TB
	    Western Digital RED 4TB
	    Western Digital RED 4TB
	  raidz2-1                 
	    Western Digital RED 4TB
	    Western Digital RED 4TB
	    Western Digital RED 4TB
	    Western Digital RED 4TB
	    Western Digital RED 4TB
	    Western Digital RED 4TB
	logs
          mirror-2                                      
            Intel S3700 100GB (underprovisioned to 15GB)
            Intel S3700 100GB (underprovisioned to 15GB)
	cache
	  Samsung 850 Pro 128GB
	  Samsung 850 Pro 128GB

MythBuntu Dedicated Backend
Handles DVR duties recording shows to the NAS via internal 10gig VMXnet3. Database and liveTV ringbuffer is recorded to SSD for performance. Has a direct I/O forwarded Intel NIC in link aggregated mode to switch. Assigned 4GB of RAM and 4 cores.

Ubuntu Server
Handles basic server tasks. Runs Ubiquiti Unifi server software, used for wgets and rtorrent to storage. Will be used for cloud backup in the future. Connects to FreeNAS using vmxnet3 10gig and has one of the two on board Intel NIC ports forwarded to it.

I have two vswitches set up. One which just connects the management console to one of the on-board Intel NICs, and the other on a sseparate subnet from the house network, which is for internal storage traffic between FreeNAS, Ubuntu Server and Mythbuntu Backend. It gets 2 gigs of ram and two cores.

SFTP
Wanted an isolated and jailed SFTP server, and was having trouble combining it on the Ubuntu server above, so I gave it its own guest. Network is shared with the management interface on one of thw teo on board Intel ports. It gets two cores and 512MB of RAM

In the future I may add a dedicated guest for remote backup, if I can't comfortably combine it with the Ubuntu server above.

Only thing I am missing now is a good UPS, and I'll be "done" (for now :p )
 
Last edited:
farscapesg1 said:
I keep telling myself that if I had somewhere to run it I would pick one of these up. Unfortunately, my Norco 4220 is already too noisy... couldn't imagine one of these in my home office. Maybe one day when I move to a climate that allows for basements and/or racking equipment in the garage. Houston area doesn't really work for either option...
Click to expand...

Where is your Norco? My 4216 is actually rather quiet. I got the optional 120mm fan divider, and use a temp probe fan controller to automatically adjust fan speeds.

That being said, everything is relative, as I came from a HP DL180G6 2U rack server which was a freaking jet engine, and wasn't even quiet enough for basement use (I could hear it from bed on the second floor!) which is why it only lasted about a month, before I replaced it with the Norco.
 
Zarathustra[H];1041106456 said:
Where is your Norco? My 4216 is actually rather quiet. I got the optional 120mm fan divider, and use a temp probe fan controller to automatically adjust fan speeds.

That being said, everything is relative, as I came from a HP DL180G6 2U rack server which was a freaking jet engine, and wasn't even quiet enough for basement use (I could hear it from bed on the second floor!) which is why it only lasted about a month, before I replaced it with the Norco.
Click to expand...

Same here in the Houston Area. My 4220 now has the 120mm fan divider as it is definitely necessary if you intend on having it in your area. It is just too noisy to be in my office by default and still a bit noisy for my tastes with Hardwoods and such in the space.

I don't know how you are going to deal with the additional sound of more rack mounts in your space.
 
Nth-ing the 120mm fan divider. My 4220 is tolerable with several undervolted 120mm fans in the divider instead of the shrieking howl of the 80mms.
 
esxi-dl380.png

Two of these, the second one has less storage. Moving over to iSCSI as soon as my new fileserver is finished :D
 
This is my setup at home.

1x D-Link DFL-800
1x D-link DGS-1248T
1x Brocade 300 SAN fibre switch
1x QNAP TS-212 2x 750GB in RAID1, Dedicated backup for VMs via Veeam.

1x Fujitsu Primergy RX300 S4 ESXi 5.5 U1
1x Dell PowerEdge R710 ESXi 5.5 U1
2x Dell PowerEdge R610 ESXi 5.5 U1

1x DotHill 2730T dual controllers
1x Fujitsu Eternus DX60 dual controllers

2x APC Smart-UPS 1500VA
1x APC Smart-UPS 1000VA

IMG_20140904_210152.jpg


IMG_20140904_210040.jpg


IMG_20140904_210053.jpg


IMG_20140905_184354.jpg
 
You must log in or register to reply here.
Back
Top