ESXi 5.5U1 memory management?

[Zarathustra[H]]

Hey all,

I would appreciate some input here. I'm at work so I don't have the details, but I can add them later.

Scenario is as follows:

Three running guests in ESXi 5.5U1

pfSense: 256MB provisioned
Ubuntu Linux Server: 768MB Provisioned
FreeNAS: 24GB Provisioned

Server has a total of 32GB of ram.

With the system and all three guests up and running it indicates that ~5GB of RAM are unused.

Perfect I thiought. I can add my server that requires 4GB of provisioned RAM. According to the guest details it should only require 200MB of RAM overhead. Plenty of margin.

Then I try to fire it up and ESXi gives me a big fat "nope". it' cant allocate all the needed RAM.

Odd I thought, since I have ~5GB free, and that should be enough for Provision of 4GB plus 200MB of overhead...

I thought maybe I can get away with less, so I dropped it down to 3GB. Still not enough. Then 2.5GB, still not enough. At 2GB it will finally start up.


Is there a reason for this? Are there other overheads I should be aware of that I am not seeing on the guest summary page?

I know the host itself needs some RAM, but I figured that would already be accounted for when it tells me I have ~5GB free, as the host is already running.

Am I doing anything wrong here?

Is there any way possible to free up more RAM for use by my new server apart from reducing what I ahve provisioned to the other guests? (I've already brought them as low as I think is feasible)

I'd add more ram, but the FX series of chips only support 4 unbuffered slots, and there still doesn't appear to be 16gb unbuffered RAM sticks out there. (and even if there were, I have no idea if the board would support them)

Don't think I can afford a supermicro board and Opteron/Xeon right now.

I appreciate any thoughts!

Thanks,
Matt

 

[SGalbincea]

This is an old chart, but it may help you figure out the overhead you are dealing with. If I had to guess, that FreeNAS box is taking up a lot more than 24GB total.

http://pubs.vmware.com/vsp40_i/wwhe...irtual_machines.html#1_7_9_9_10_1&single=true

 

[lopoetve]

You doing passthrough?

 

[Zarathustra[H]]

This is an old chart, but it may help you figure out the overhead you are dealing with. If I had to guess, that FreeNAS box is taking up a lot more than 24GB total.

http://pubs.vmware.com/vsp40_i/wwhe...irtual_machines.html#1_7_9_9_10_1&single=true

Yes, but isn't the ram overhead captured by what is in the "summary" tab?

Let me explain.


Overall System RAM:

pfSense Guest:

Linux Server Guest:

FreeNAS Guest:

So the way I am reading this (and I may be wrong, which is why I am asking, so I understand better) is that each of the guests use their provisioned RAM, plus the overhead from the summary tab above.

In my case:

	Provisioned	Overhead	
Freenas	24576		177.82
pfSense	256		21.07
Ubuntu	768		26.51
			
	25600		225.4
			
Total Used	25825.4		
			
Total in box	32749.17		
			
diff	6923.77

So, I should have about 6.9gigs free. Now a bunch of this will be consumed by the host.

Judging by the first image that shows the total in use (with three guests running) is 27409MB I'm assuming that figure is ~1.5GB.

Either way, current state shows 32749.14MB total - 27409MB used = 5340.14MB free.

Now here is the provisioned RAM and overhead for my new guest:

You'd think that since 4096MB + 237.44MB > 5340.14MB, it ought to work, but no dice.

Any help clearing up what I am overlooking, or where my confusion here lies would be greatly appreciated.

Thanks,
Matt

 

[Zarathustra[H]]

You doing passthrough?

Yes I am doing passthrough. Does that eat RAM above what is listed as overhead in the summary section?

Each guest has at least one Direct I/O forward as follows:

pfSense: 1 dual port Intel Pro/1000 PT NIC.
Ubuntu: 1 single port Broadcom NetXtreme NIC
FreeNAS: 1 dual port Intel Pro/1000 PT NIC and 1 IBM M1015 SAS controller

My prospective guest also gets a passthrough of 1 dual port Intel Pro/1000 PT NIC.

So, how does passthrough affect RAM? And is it not included in the "overhead" on the "Summary" tab?

Thank you,
Matt

 

[SGalbincea]

I wonder if we are dealing with a heap size issue here with the passthrough. I have dealt with it when I had too many 1Gb Broadcoms in a host before, and it gives the same cannot allocate memory error you describe.

 

[lopoetve]

Yes I am doing passthrough. Does that eat RAM above what is listed as overhead in the summary section?

Each guest has at least one Direct I/O forward as follows:

pfSense: 1 dual port Intel Pro/1000 PT NIC.
Ubuntu: 1 single port Broadcom NetXtreme NIC
FreeNAS: 1 dual port Intel Pro/1000 PT NIC and 1 IBM M1015 SAS controller

My prospective guest also gets a passthrough of 1 dual port Intel Pro/1000 PT NIC.

So, how does passthrough affect RAM? And is it not included in the "overhead" on the "Summary" tab?

Thank you,
Matt

So, several people have been hitting around the mark.

1. Passthrough devices require 100% ram allocation at all times by design (so that IRQ/DMA mappings can't get moved, among other things).
2. Passthrough devices consume heap "differently" and it's hard to measure - that heap doesn't show up in memory allocation as it's not allocated, just reserved for use if needed to enable it to be passed to the guest. In a normal device, we can measure that - quick and dirty testing says that it's somewhat difficult with passthrough devices.
3. Your host burns 800MB by default for the host OS itself.
4. You can't ever actually hit 100% due to heap and some other memory reserves, even before considering item 2, although you can get stinking close.
5. Your host is effectively telling you that due to those items, you have somewhere between 2-2.5G of usable ram left - since we don't actually expect 100% (unless you're running an HPC workload, which changes the equations as well, albeit in an opposite way) utilization, it isn't displayed totally accurately in the GUI. Expected utilization is around 80% for an ESX host in production.

 

[Zarathustra[H]]

So, several people have been hitting around the mark.

1. Passthrough devices require 100% ram allocation at all times by design (so that IRQ/DMA mappings can't get moved, among other things).
2. Passthrough devices consume heap "differently" and it's hard to measure - that heap doesn't show up in memory allocation as it's not allocated, just reserved for use if needed to enable it to be passed to the guest. In a normal device, we can measure that - quick and dirty testing says that it's somewhat difficult with passthrough devices.
3. Your host burns 800MB by default for the host OS itself.
4. You can't ever actually hit 100% due to heap and some other memory reserves, even before considering item 2, although you can get stinking close.
5. Your host is effectively telling you that due to those items, you have somewhere between 2-2.5G of usable ram left - since we don't actually expect 100% (unless you're running an HPC workload, which changes the equations as well, albeit in an opposite way) utilization, it isn't displayed totally accurately in the GUI. Expected utilization is around 80% for an ESX host in production.

Ahh, That explains it then. Thank you for clearing this up!

Looks like I am going to be shopping for some of those elusive 16GB unbuffered ECC DDR3 sticks.