Cloud Virtualization - a Hardware Question.

[RaphaelVinceti]

Now this is a question that I have been wondering about for the last couple of months, and have been trying to figure out which hardware set would be better for a Cloud-type hosting service.

The question is what hardware should you run? More specifically, what type of hardware setup should you run, a handful of massively powerful, and massively expensive, servers (4 socket, 2ghz+ systems with massive amounts of ram), or many less powerful, much less expensive systems.

Example: small datacenter (<100 VMs

2 or so servers
each 4x X7460 @ 2.67GHz
64GBs ram

or

8 or so servers
each 2x L5240 @ 3.0GHz
16gb ram

With ESXi you can double allocate memory, since similar VMs have many of the same memory pages, so having 64GB ram in each of the faster ones would allow for roughly 128 VMs (at 2GB RAM in each). with the smaller servers, you would have the same amount of ram, spread over more servers.

With the two powerful servers, you have 48 processors, but you have 64 in the smaller machines (when combined). Same amount of ram, slightly faster with less cache processors in the smaller machines.

From an Operational reliability standpoint, I have been assuming that the more, less powerful machines would be better, but from an energy efficient standpoint, I was thinking that maybe the more powerful systems would be a better choice. The only thing I can't really decide is a performance standpoint, which of the two would end up with better performance in a completely Virtualized environment. Anyone have any thoughts of their own, or numbers I might have missed to point the finger at which would be better?

 

[AceXsmurF]

Well let us just think about it practically in terms of maintenance.

Let's go with the 2 server scenario first.
Say something happens to one of the machines, like something stupid rare like both power supplies die or something equally stupid that shouldn't happen to one of the machines. That machine is now dead in the water and all the VMs on it are either no longer running or have migrated to other machines putting a much heavier load on those other machines. You also have the matter of maintenance, eventually you will have to do some maintenance on the machines, having to take down a heavily loaded machine will cause a lot more stress than having multiple smaller machines.

In general, multiple smaller machines, depending on economics, seem like a better option to me.

 

[Thuleman]

Anyone have any thoughts of their own, or numbers I might have missed to point the finger at which would be better?

Is this an actual implementation question, or are we just discussing it for the sake of discussing it?

In either case, 2-8 machines are not a "cloud", they are a cluster at best.

 

[RaphaelVinceti]

Is this an actual implementation question, or are we just discussing it for the sake of discussing it?

In either case, 2-8 machines are not a "cloud", they are a cluster at best.

discussing it for the sake of discussing it. And I was really referring to the number of logical servers, not physical ones Anyway, Cloud is just another way to refer to Mainframe style computing. Everything stored in one location, (or multiple locations, but the clients are all remote, with nothing but very simple software on the clients, if anything more than a browser.)

 

[RaphaelVinceti]

Well let us just think about it practically in terms of maintenance.

Let's go with the 2 server scenario first.
Say something happens to one of the machines, like something stupid rare like both power supplies die or something equally stupid that shouldn't happen to one of the machines. That machine is now dead in the water and all the VMs on it are either no longer running or have migrated to other machines putting a much heavier load on those other machines. You also have the matter of maintenance, eventually you will have to do some maintenance on the machines, having to take down a heavily loaded machine will cause a lot more stress than having multiple smaller machines.

In general, multiple smaller machines, depending on economics, seem like a better option to me.

While that is all true, would not the power requirements for having more servers end up equaling out the TCO? That would pretty much eliminate the economics issue from the considerations. I mean, 2 of the more powerful machines would cost nearly 30-40 thousand each (if they use SAS 15k drives), while each of the less powerful ones would cost 4-5k each. But, the power requirement differences should end up evening it out, depending on deployment length.

The issue I suppose becomes reliability, parts go down in the more powerful systems, and you cut your resource pool in half. parts would have to go down in 4 machines at the same time to cut your resource pool in half with the smaller ones. I'm personally on the side of small/many.

 

[Thuleman]

While that is all true, would not the power requirements for having more servers end up equaling out the TCO?

This is one of the reasons why "discussion for discussion's sake" can't really yield meaningful results.

The difference between the number of physical servers isn't limited to the requirement to power them. Other factors include cooling, rackspace availability, networking equipment availability (at 7 ports needed per server things add up quickly), and any other number of things we don't think of such as budgetary requirements and assorted politics.

SAS drives don't amount to much per server, in fact, you could easily get away with SATA drives on each server since you will need high performance shared storage either which way. The drives in the server are just 2x mirrored drives to host your hypervisor.

Reliability is a non-issue. If you run mission critical tasks you will have spare parts on the shelf and can instantly replace anything that goes down. If you do not do mission critical tasks, you are still going to have 4-hour on-site replacement. During that period you can get by on half the capacity.

 

[da sponge]

Also you need to think about licensing requirements. If you go with ESX, more sockets == lots more money. The cost for enterprise is NON TRIVIAL, especially with 8 additional sockets in the 8 server farm.

 

[AreEss]

If you don't have a hardware criticality requirement (must have X servers or N-way on failover direction) then go with less servers.

ESX will set you back around $12,000 per socket per server based on our last quote.

That said, your assumptions are quite wrong. Smaller servers can be more efficient than larger servers, depending on any number of factors. You should also be factoring in proper external storage such as FC attached arrays or multipathed SCSI or SAS, rather than using internal disks, if you're going with Enterprise.

At first blush, my recommendation? None of what you've suggested. VMWare is memory hungry above CPU hungry. I would recommend 4 dual socket quad cores with 32GB of memory each. Again, this presumes moderate to low CPU loading. But it also buys you a great deal more in terms of scalability, because it allows you to load up large numbers of low load VMs alongside higher load VMs. It also balances your costs a little more evenly.

My $0.02.

 

[bigshooter]

Are you including storage or other hardware in your estimate of per socket pricing? I don't remember what we paid for our last set of licenses, but I think $12-13,000 gets us 2 VI 3.5 enterprise licenses w/1 year of support. With the cost of the server (I use Dell R900s with 128GB memory) and an additional enclosure for whatever disk array you're using I could see $12-15k per socket being accurate though.