Which Server Build to Choose for ESXi

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
I am trying to decide what make more sense for building a new ESXi cluster. It will most likely have between 7-10 servers depending on which build I choose.

Build One

Barebone - Supermicro SuperServer 6027R-3RF4+
Processor - Intel Xeon E5-2680 v2 (x2)
Heatsink - Supermicro SNK-P0048P (x2)
Memory - 16X16GB ECC REG DR X4 (256 GB per Server - Expandable to 384 GB)
Hard Drives - WD RE WD2503ABYZ 250 GB 3.5" (x2)
NIC - Intel Ethernet Converged Network Adapter X520-DA2 (x3)
RAID - LSI Logic MegaRAID 9271-8i 8-port SAS Controller
BBU - LSI Logic Battery Backup Unit - LSIiBBU09

Each of these servers has 20 cores for total CPU resources of 56 GHz without Turbo. ESXi 5.5 would be installed on a RAID 1 array.

Build Two

Barebone - Supermicro SuperServer 6028R-TR
Processor - Intel Xeon E5-2683 v3 (x2)
Heatsink - Supermicro SNK-P0048PS (x2)
Memory - 256 GB Total Per Server (Either 16 GB or 32 GB Chips)
SATA DOM - SSD-DM016-PHI
NIC - Intel Ethernet Converged Network Adapter X520-DA2 (x3)

Each of these servers has 28 cores for a total CPU resources of 56 GHz without Turbo. (Yes the total capacity is the same, but I see it as more cores, with more cache, a faster system bus and DDR4 memory.) I would do away with the RAID and Hard Drives in favor of either the SATA DOM listed, USB stick internal on the motherboard, or switching to PXE boot to load ESXi 5.5. I am hoping the lack of RAID offsets the rest of the cost and would prefer to start using customized images.

If I had 10 of build one, that would give me 200 cores and 2.5 TB of memory.

If I had 7 of build two, that would give me 196 cores and 1.75 TB of memory.

I already know what build one costs, so I guess I am wondering if it is even worth getting a quote for build two. Does anyone see any advantage or performance gains from build two or have a better recommendation? It needs to be Supermicro.
 
Eulogy said:
What is your design goal?
Click to expand...

This would be a single DRS cluster running at least two environments.

One of those environments already has 84 CPU cores, but is running a combination of Intel X5660 and Intel E5640. Memory is around 900 GB. Memory is used a lot more than CPU at the moment. Two MSSQL servers (which aren't configured correctly) both using 196 GB of memory each and 16 vCPUs. Total combined databases sizes are over 1 TB. Databases are all connected through 1Gbps iSCSI and will be upgraded to 10Gbps in the new environment. Everything here is located in Site A.

The second environment is a full IaaS implementation with an RDS Farm for the users to connect to. This currently resides on five of the servers described in Build One, but without the 10Gbps networking. This is all located in Site B.

The new cluster I am looking to build would be Site C and would combine both A and B. I am referring to physical datacenters in different locations, not creating a new datacenter in vCenter.

Sites A and B each have a NetApp FAS3240 which is 1Gb. Site C has a FAS8040 which is 10Gb and has more IOPS than both the other NetApp's combined.

There would probably be another environment added at some point which has many large Oracle databases.

Cost verse performance is a factor which is why I am wondering if build two would give us a better bang for our buck.
 
No RAID = Poof you re now offline. If you have a large enough cluster this is not a problem, but I think it's a bad idea.
 
Mackintire said:
No RAID = Poof you re now offline. If you have a large enough cluster this is not a problem, but I think it's a bad idea.
Click to expand...

Huh? You do realize you can boot ESXi off a USB stick and once it is running in memory, you can pull the USB stick out without any issues. Syslog will be done remotely and not on the server. We have a Dell Blade Center and all the blades use a 4 GB internal USB stick. It is about 3 years old and never had an issue.
 
I got some pricing back and they are basically the same price for build one and build two.

The biggest benefits I see with build two is each server has 8 more CPU cores and faster memory. The downside is it is not VSAN ready, but so far VSAN is not on our road map. There is another Supermicro motherboard that cost a little more with the LSI 3108 built in.

Where is NetJunkie, loop, and others? I need to pitch one or the other to management and want to make sure I am making the right decision.
 
KapsZ28 said:
Huh? You do realize you can boot ESXi off a USB stick and once it is running in memory, you can pull the USB stick out without any issues. Syslog will be done remotely and not on the server. We have a Dell Blade Center and all the blades use a 4 GB internal USB stick. It is about 3 years old and never had an issue.
Click to expand...

Er... As long as you have notification that the USB drive failed, sure I guess that's ok.

Again if you have enough redundancy it's only an issue if your uptime is critical.
 
Mackintire said:
Er... As long as you have notification that the USB drive failed, sure I guess that's ok.

Again if you have enough redundancy it's only an issue if your uptime is critical.
Click to expand...

There would be notifications about any type of storage failing which would include the USB. But again, the ESXi host will continue to operate as normal. Goal is to always have enough resources in the cluster. :D That is a big focus of mine. To me if a single host failure puts everything in the red, then that is not acceptable. Also with a scripted install the goal would be able to rebuild and put the ESXi server back into production in less than an hour.
 
I always try to build enough redundancy that I don't have to come in, in the middle of the night because the risk is minimum or non-existent for business critical server or lesser services.


Mission critical servers....my butt is logged in, on the phone or out the door in-route within the first 20 minutes.
 
SuperMicro yuck.

I'm just finishing a large cluster of 512 Dell R630s with E2699s ( cache was the top goal ). I would say go build 2 the ram is already cheaper for larger sizes and will only get better. The lower power will also be nice over the life of the servers.

You don't need the raid anything onboard is fine for a LSI maga raid is over kill you don’t need cache for a host raid. I tend to prefer to have some local storage in at least a few hosts shared storage has good up time but when it’s down its down big. The last set of VMware servers I built (mostly Hyper-V these days) I used onboard SD cards for the host OS cheap and you can raid 1 them.

I have done usb boot in Zen, VMware, and Hyper-V all the ended in tears ports going bad all fine till a reboot then you had to take them out and put back in to get anything. the drives showing fine but being worn out everything showed up fine but they had a R\W of .000000002k 6 hour host boot times suck. The last time I gave it a try with VMware approved drives direct from Dell they all died within 2 weeks of each other there was a MTF in the drives no one told me about.

Would not go PXE for host boot as well the extra network infrastructure needed is not worth it. Depending on your network it’s an extra vlan or more nics and switching. I have done it once with a Netapp twice with EMC and once with an EqualLogic. The netapp took about 3x to long to get working as it should tons of issues with getting the traffic on the correct vlan the traffic would not bind to the correct Vnic. The emc worked ok but there was consent time out issues reboot a host 2 3 times and then it would work. The EqualLogic was a mess as well the hosts using up all the iscsi connections and guests not getting access. All in all a great way to spend less on hardware and tons on labor.
 
Yeah, I am thinking build 2 with the 32 GB memory chips. It is nearly $30k less, gives us almost identical number of cores and GHz, but also adds additional CPU cache, and faster DDR4 memory. Memory is slightly lower to start with. 1.8 TB verse 2.5 TB, but can be easily upgraded in the future as needed. Off the bat we really only need about 1.2 TB of memory which should put it around 65% utilization. Spread over 7 servers means with a single server failure the memory utilization will average out to about 75%. Still well below the red.
 
Any other opinions? These are my primary concerns.


  1. Is it worth ditching RAID and running ESXi on SATA DOM or USB?
  2. Comparing the Intel Xeon E5-2680 v2 to the E5-2683 v3. Which do you think is a better choice?
 
You must log in or register to reply here.
Back
Top