Calling Iopeteve - upgrade + hardware question.

[AreEss]

Semi-theoretical configuration here. I'm sure you can puzzle this mess out better than I can and I could use a hand. There's too many wild variables, and I'm not the VMWare admin so I don't have access to realtime hard numbers.

Existing is 4x HS21's in two dual 5150's, and two dual 5340's, 3x LS20's all in dual dualcore 2.0's. HS21's are all 16GB and supposedly averaging 60% memory utilization, LS20's are all 12GB and averaging 70% utilization. All on 3.0 dunno what patch level.

Does switching all that out for 5x HS22's in single quad 2.83GHz (almost all VMs are single core allocation) with 32GB or 48GB each sound reasonable as part of a 3.5 upgrade? Or would the cut in core count be a significant concern at least with regards to the internal scheduler. Is there a specific ratio to target for VM cores to physical cores?

Also, can you point me to a best practices for 3.5 versus 3.0 disk allocation perchance? This environment's entirely SVC backed, but I'm working some magic with mdisks.

Thanks!

 

[xphil3]

Semi-theoretical configuration here. I'm sure you can puzzle this mess out better than I can and I could use a hand. There's too many wild variables, and I'm not the VMWare admin so I don't have access to realtime hard numbers.

Existing is 4x HS21's in two dual 5150's, and two dual 5340's, 3x LS20's all in dual dualcore 2.0's. HS21's are all 16GB and supposedly averaging 60% memory utilization, LS20's are all 12GB and averaging 70% utilization. All on 3.0 dunno what patch level.

Does switching all that out for 5x HS22's in single quad 2.83GHz (almost all VMs are single core allocation) with 32GB or 48GB each sound reasonable as part of a 3.5 upgrade? Or would the cut in core count be a significant concern at least with regards to the internal scheduler. Is there a specific ratio to target for VM cores to physical cores?

Thanks!

Whoa memory bottleneck....

I would be curious as to weather this "rule of thumb" has changed, but I know that even today most people are sticking to this. 4 VM's per physical core.
http://pubs.vmware.com/server1/vm/w...wwhelp.htm?context=vm&file=intro_vm.2.16.html

 

[sabregen]

we've got the info on some of the physical hardware, but not any information on how many VMs you're running. You give memory usage, but not CPU. What features do you have licensed? If you need HA, DRS, etc...do you have the Intel and AMD blades in separate resource pools so that you can float in between similar architectures? Obviously, the HS21's with the 5150's would stay the same on core count...and would get a bump in speed. The HS21's with the 5340's would lose 1/2 their cores. If I am reading this right, you currently have this:

4x HS21's with 2x Xeon 5150's +16GB RAM = 4x 2.66 = 10.64GHz +16GB RAM (each blade)
2x HS21's with 2x Xeon 5340's +16GB RAM = 8x 2.33 = 18.64GHz +16GB RAM (each blade)
3x LS20's with 2x Opteron 270's +12GB RAM = 4x 2.0GHz = 8.0GHz +12GB of RAM (each blade)

TOTAL INTEL COMPUTE POWER = 42.56 + 32.78 = 79.84GHz + 96GB of RAM
TOTAL AMD COMPUTE POWER = 24.00GHz + 36GB of RAM

If you replace it all with HS22s, you will get the following:

5x HS21's with 1x Xeon X5570 + 48GB RAM = 4x 2.93GHz = 11.72 +48GB of RAM (each blade)

TOTAL INTEL COMPUTE POWER = 58.6 + 240GB of RAM

You'd lose half of your raw processing power, but would almost triple the amount of available RAM for your Intel based blades. You would gain the faster bus speed of the newer CPU architectures (you'd go from a 1066FSB on the Intels and 800FSB 'effective' on the AMD CPUs to a 1600FSB on the new Intels), so things like internal only NICs, for NAT'ing between VMs would operate at the now faster bus speed, etc.

Additionally, please note that the HS22's are dual socket blades, and as with all NUMA based architectures, you must populate both CPUs in the system in order to have the ability to use all of the 12 RAM slots. In order to achieve 48GB of usable RAM with only one CPU, you will only have 6 slots usable (without a second CPU), and you will therefore have to buy 3x 16GB DDR3 kits in order to meet the density requirement to get the 48GB you're looking for. The reality is that it will very likely be cheaper to add the second CPU, so that you can use the lower density DDR3 modules to reach the target of 48GB, since then you'd have all 12 slots available, and would only need 4GB DDR3 DIMMs.

Does that help?

Update: Misread the initial post...you're looking to get rid of the LS blades, and the HS21s, and replacing them only with the HS22's...updating my calculations.

 

[AreEss]

*facedesk*
sabre, your math is so far beyond flawed it's not even remotely funny.
Gigahertz and Megahertz are not a comparable power rating at all and never have been and never ever will be. I don't care what VMWare reports utilization as; MEGAHERTZ IS NOT A VALID MEASURE OF PERFORMANCE.
This is why I was asking Iopeteve; I know far, far more about the hardware than others here - to put it mildly, since you got the entire HS22 architecture wrong. And clearly do not understand how computing power actually works. The only thing you got right is the memory.

Yes. I'm testy this morning. I have a 6 node SVC with 576.8ms response times on the high performance mdisk sets.

 

[lopoetve]

Yipe, I'll get to this at lunch. Way to many acronyms for an already insane morning.

 

[sabregen]

*facedesk*
sabre, your math is so far beyond flawed it's not even remotely funny.
Gigahertz and Megahertz are not a comparable power rating at all and never have been and never ever will be. I don't care what VMWare reports utilization as; MEGAHERTZ IS NOT A VALID MEASURE OF PERFORMANCE.
This is why I was asking Iopeteve; I know far, far more about the hardware than others here - to put it mildly, since you got the entire HS22 architecture wrong. And clearly do not understand how computing power actually works. The only thing you got right is the memory.

Yes. I'm testy this morning. I have a 6 node SVC with 576.8ms response times on the high performance mdisk sets.

Wow...just WOW. I merely responded with the calculations that ESX will present to you. I do not make the claim that CPU speed is a good performance metric, and I certainly understand (and do not condone) the way that ESX reports the aggregation of CPU cores x frequency = available processing power. I am well versed in the differences of CPU architectures. That being said, the specs to the HS22 are right here:

http://www-03.ibm.com/systems/bladecenter/hardware/servers/hs22/specs.html

I think perhaps YOU should review the specs of the blade, and the chipset that is employed, to understand the architecture a bit better. I'm usually not one to toot my own horn, but I do hold a System x cert, and work with the Bladecenter chassis, and blades, option modules, and all other parts on a frequent basis.

You've never given me trouble before, so I'll hold my tongue...you're on your own, dude. "Testy" isn't the phrase I would personally use. Responses like this are the reason I think we need a mod here in Virtual Computing. There should be a forced redaction on your post, as you're bordering on inciting a flame war. I could go on and on...

 

[lopoetve]

Whoa memory bottleneck....

I would be curious as to weather this "rule of thumb" has changed, but I know that even today most people are sticking to this. 4 VM's per physical core.
http://pubs.vmware.com/server1/vm/w...wwhelp.htm?context=vm&file=intro_vm.2.16.html

that's for vmware server, and is way conservative especially for ESX.

 

[AreEss]

HEADER: Hi. I'm here. I'm in process of you don't want to know what. I am going to apply much brain bleach tonight.

Yipe, I'll get to this at lunch. Way to many acronyms for an already insane morning.

I will gladly, GLADLY trade you mine. And my afternoon too, for free! Thanks, Iopoetve. (Also, I will yet learn to spell your name. Promise.)

xphil3, you're absolutely right that there's an ugly memory bottleneck. But as Iopoetve pointed out, that's for VMWare server, rather than ESX. I also was an idiot and forgot to mention that the target environment number is ~80-90 give or take a few here and there. Stupidity on my part, but it has SERIOUSLY been one of those weeks.

Wow...just WOW. I merely responded with the calculations that ESX will present to you. I do not make the claim that CPU speed is a good performance metric, and I certainly understand (and do not condone) the way that ESX reports the aggregation of CPU cores x frequency = available processing power. I am well versed in the differences of CPU architectures. That being said, the specs to the HS22 are right here:

You presented the well debunked math that I had "79.84GHz" and "24.0GHz" of computing power, and made the statement that at "58.6GHz" of 'compute power' I would, AND I QUOTE

lose half of your raw processing power

People who do their homework and don't spout known garbage never have a problem with me. Attempting to claim that 1MHz of AMD is equal to 1MHz of 3 generation newer Intel is the absolute exact same as claiming that a Pentium III 1.6GHz should be every bit as fast as a Core Solo at 1.6GHz. It is like saying your Q6600 is not a 2.6GHz, it is a magical 10.4GHz! Or to be more current, it is the exact same as claiming that an Opteron 2000-series at 2.8GHz is 100% as fast as a Nehalem at 2.8GHz - complete garbage, completely false, and easily demonstrated as such. It is a waste of your time and mine, and everyone else's, to post trash like that.

 

[lopoetve]

Cores to vcore depends entirely on what you're running, to be honest.

I would expect that you would probably be fine for cpu resources, but you may starve things like the SCSI queue or the like, depending on your actual usage for each system and what your vms are running and doing.

How many vms are we talking about? What applications? What's the average util. for the cpus?

Sabregen is right though - those are NUMA boxes, so 1 cpu is going to leave you very limited in terms of memory access, vs using 2 physical chips. I'd look at populating the second socket. Otherwise you may also run into some of the bugs there have been about imbalanced NUMA nodes - I've never seen someone run NUMA archetecture with a single cpu before. this would also solve any compute resource limitations you'd probably have, unless you're running a billion databases.

 

[AreEss]

Cores to vcore depends entirely on what you're running, to be honest.

I would expect that you would probably be fine for cpu resources, but you may starve things like the SCSI queue or the like, depending on your actual usage for each system and what your vms are running and doing.

How many vms are we talking about? What applications? What's the average util. for the cpus?

Sabregen is right though - those are NUMA boxes, so 1 cpu is going to leave you very limited in terms of memory access, vs using 2 physical chips. I'd look at populating the second socket. Otherwise you may also run into some of the bugs there have been about imbalanced NUMA nodes - I've never seen someone run NUMA archetecture with a single cpu before.

Actually, memory access is the absolute bottom of my concerns. They're triple-channel boxes. 6x4GB leaves me well within realistic load numbers with ~20% overhead give or take. (There's also some magic I can play with, but that's neither here nor there.) Would I rather have both sockets full? Certainly. Unfortunately, I'm working with the ugly thing called REALITY which says that it is tens of thousands of dollars more just in maintenance per year to license 12 sockets instead of 6. Hooray recession.

This is, as I said above correcting my lack before, about 80-90 environments total. I can't speak very well to the applications - there's a LOT of varied ones. It's mostly light work, as we do all the truly heavy lifting on Unix. There are a few file servers, some print queues, I think 5 MSSQL databases give or take. I'm not aware of any significant applications that chew up a lot of CPU, besides the two NFS gateways. (Those will probably be getting moved to physical blades, due to network load.)

We're not seeing many disk problems right now, but we also can't get an accurate handle on whether or not there are any. These are all running ESX on the internal drive with all VMs on the SVC. I know the SVC isn't bottlenecking, I just don't know if the individual blades are. Can you explain what 3.5's behavior with the IBM SVC is? Does it use fully active with preferred pathing correctly? If so, figure your individual blade queue numbers on 1 LUN per environment, each LUN having minimum 4 paths active for one I/O group, 4Gbit full bandwidth end to end. Backing disk performance behind the SVC is ~2200MB/s <1.9ms response.

Thanks!

 

[sabregen]

It is a waste of your time and mine, and everyone else's, to post trash like that.

Responses of this type are inflammatory, and unnecessary.

I merely represented to you the information that ESX uses to compute host resources. That's how they (VMWare) do it...and that's what we're talking about, here. I do see a major flaw in the VMWare has decided to report the processing power of dissimilar architectures (by reporting solely on CPU frequency, and not having it based on a relative index of processing power). Be that as it may, that's how they (VMWare) do it...so why are you flame bating me?

I can certainly understand the licensing aspect, and not wanting to cover additional sockets, and having to maintain the renewal of the contracts, down the road. Everyone is pinching pennies these days. If you can get away with only using 1 socket per blade, then barring any weird issues ESX has with only 1/2 populated NUMA architectures, then go for it.

I hope you get your questions answered. I sure as hell didn't mean any harm by taking time out of my morning to do something that I though would be helpful, and trying to provide you with a concise view on what you would see inside of ESX, should you choose to go with the HS22's, or divulging the potential gotcha of populating 1/2 the sockets on a NUMA blade. I'm not out to get you, I was just trying to help.

 

[QHalo]

This is why I was asking Iopeteve;

Try a PM next time.

 

[lopoetve]

Try a PM next time.

I'd actually prefer to avoid the PMs.

I'm here as an enthusiast first. I'll gladly help, but I'd rather it be all out in the open where others can benefit and ask questions too.

 

[QHalo]

I'd actually prefer to avoid the PMs.

I'm here as an enthusiast first. I'll gladly help, but I'd rather it be all out in the open where others can benefit and ask questions too.

I agree as well to keep it for everyone to see. I guess more or less I was commenting on his rather unsightly reponses to forum members. Then I took at a look at his last 20 or so posts and I can see the same 'holier than thou' trend. To ignore he goes. Sorry for the thread jack.

 

[lopoetve]

Actually, memory access is the absolute bottom of my concerns. They're triple-channel boxes. 6x4GB leaves me well within realistic load numbers with ~20% overhead give or take. (There's also some magic I can play with, but that's neither here nor there.) Would I rather have both sockets full? Certainly. Unfortunately, I'm working with the ugly thing called REALITY which says that it is tens of thousands of dollars more just in maintenance per year to license 12 sockets instead of 6. Hooray recession.

It's not the bandwidth that concerns me - it's the number of slots that can be used, and the cost for high-density sticks vs not. then again, my calculations don't include licensing fees, either from us or from IBM, so I can't truly calculate that.

Even still, I'd check with IBM about stability and issues with a half populated NUMA system - I know the older HP AMD boxes had issues with it, so I'd double check before proceeding down this route. That being said, I think you'll have the compute power to do the job. The only people that can tell for sure is VMware PSO - they don't give support access to capacity planner.

This is, as I said above correcting my lack before, about 80-90 environments total. I can't speak very well to the applications - there's a LOT of varied ones. It's mostly light work, as we do all the truly heavy lifting on Unix. There are a few file servers, some print queues, I think 5 MSSQL databases give or take. I'm not aware of any significant applications that chew up a lot of CPU, besides the two NFS gateways. (Those will probably be getting moved to physical blades, due to network load.)

We're not seeing many disk problems right now, but we also can't get an accurate handle on whether or not there are any. These are all running ESX on the internal drive with all VMs on the SVC. I know the SVC isn't bottlenecking, I just don't know if the individual blades are. Can you explain what 3.5's behavior with the IBM SVC is? Does it use fully active with preferred pathing correctly? If so, figure your individual blade queue numbers on 1 LUN per environment, each LUN having minimum 4 paths active for one I/O group, 4Gbit full bandwidth end to end. Backing disk performance behind the SVC is ~2200MB/s <1.9ms response.

Thanks!

Doesn't sound too bad for load.

SVC is not yet certified with 3.5 U4, but we support it as an active active array with a most recently used pathing policy (weird, I know) on U3 and earlier. We do not use preferred pathing that I know of - call in and open a case with storage, and we can direct you to the SVC folk here (I'm not one of them - I'm more familiar with the NetApp and EMC storage, with HP being a close third).

My only concern would be with the virtualized scsi queue, and I don't think you'll come anywhere near saturating that with 2 possible exceptions: Don't run windows update on all machines at the same time, and don't do virii scans all at the same time. These are a good best-practice anyway, but the micro-commands will overwhelm the queue in sheer numbers on their own, and since you've got fewer hosts, you have fewer queues to work with. Just stagger things and you'll avoid that whole mess.

 

[AreEss]

It's not the bandwidth that concerns me - it's the number of slots that can be used, and the cost for high-density sticks vs not. then again, my calculations don't include licensing fees, either from us or from IBM, so I can't truly calculate that.

Even still, I'd check with IBM about stability and issues with a half populated NUMA system - I know the older HP AMD boxes had issues with it, so I'd double check before proceeding down this route. That being said, I think you'll have the compute power to do the job. The only people that can tell for sure is VMware PSO - they don't give support access to capacity planner.

You'll have issues if you flip the HypeThreading switch, absolutely. On the IBMs, there's just some layout specificity that's absolutely mandatory to keep things happy. See the Product Guide over here:
ftp://ftp.software.ibm.com/common/ssi/pm/rg/n/blo03029usen/BLO03029USEN.PDF
Nutshell version; you can go up to a maximum 6 DIMMs with a single socket using sockets 1 through 6. Sockets 7 through 12 are attached to CPU1's memory controller exclusively. You actually reduce NUMA risk in this configuration; if you were to go to CPU0 + CPU1, your only safe configuration is 12 identical DIMMs. By the same token, your safe configurations with a single CPU are 1x3 and 2x3, because you cannot cross the QPI boundary. It's when you introduce the QPI boundary that things get dicey.
Can't comment on the HP issues, though. I have no experience on their AMD hardware save the DL585. The AMD8111+8131 always had some tetchiness though.

Doesn't sound too bad for load.

That's what I was hoping to hear; I am NOT a Windows guy primarily, so I don't know what kind of relative load we're talking.

SVC is not yet certified with 3.5 U4, but we support it as an active active array with a most recently used pathing policy (weird, I know) on U3 and earlier. We do not use preferred pathing that I know of - call in and open a case with storage, and we can direct you to the SVC folk here (I'm not one of them - I'm more familiar with the NetApp and EMC storage, with HP being a close third).

Argh. They told us to do that with 3.0, then told us to go call IBM. IBM told us to call VMWare. And so the cycle unquestionably continues, where the "solution" is to buy ESX at premium from IBM with IBM RPQs which just say that you call IBM, then they call you. And that the customer can't call VMware directly for any support issues whether or not they're SVC related.
Can you check with them and ask them what the hell is up with this? Full and PROPER SVC support has been being promised for two years now, and it sounds to me like development has basically decided that they don't have to follow through on that if they can find a good excuse and a bad hack.

My only concern would be with the virtualized scsi queue, and I don't think you'll come anywhere near saturating that with 2 possible exceptions: Don't run windows update on all machines at the same time, and don't do virii scans all at the same time. These are a good best-practice anyway, but the micro-commands will overwhelm the queue in sheer numbers on their own, and since you've got fewer hosts, you have fewer queues to work with. Just stagger things and you'll avoid that whole mess.

Yeah, I think that goes without saying. But thankfully, I'm Unix side, so I actually get to avoid that in it's entirety. Our Windows admin is top notch though, no doubt, so he's already taken care of all of these things. And we're cheating on backups with VCB (in theory, eventually, at 4/8Gbit x 16 ports x 8 LTO4 drives.)
My biggest concern was the CPU queues and whether or not I had a sufficient ratio of physical cores to virtual, in addition to sufficient horsepower increase per-core to make up for the loss of core count.

Thanks, Iopoetve!

 

[xphil3]

xphil3, you're absolutely right that there's an ugly memory bottleneck. But as Iopoetve pointed out, that's for VMWare server, rather than ESX.

that's for vmware server, and is way conservative especially for ESX.

Interesting.... I spoke pretty indepth with some of the lead engineers at vmworld 07 and this was what they told me as well.. I didn't even notice the link was not regarding ESX... heh... Thanks for the info lopoetve