• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ESXi and Memory Reservation

Zarathustra[H]

Zarathustra[H]

Extremely [H]
2FA
Joined
Oct 29, 2000
Messages
42,306
Hey all,

I was just going though the settings on my VM's and noticed that by default they are all set to reserve all the RAM they have been allotted.

It would seem that this might be a little bit inefficient from the standpoint of optimal resource utilization.

How do you typically configure the memory reservation on your guests?

And how does memory reclamation really work?

Most modern OS's use all the RAM they can get. What't not in active use is instead used for caching. I presume the VMWare-tools somehow get involved here - if the full amount of RAM is not reserved and tell the OS:es when to cache and not to cache using not actively used RAM, and gives some of it back to the resource pool, but how does it decide to give it back?

For instance, one of my storage servers relies on gobs of ram for write caching, when writes temporarily overwhelm the write speed of the storage. If I don't reserve the full amount of RAM on this server, will it no longer cache my writes?

Any help in understanding this topic would be greatly appreciated.
 
To follow up on this,

I tried lowering the reservation slider (thus in my head, thin-provisioning the RAM) to half of the assigned RAM on a few different guests.

I wanted to test if it would allow me to run guests with more assigned RAM than the host, and manage it between them using thin provisioning. I guess it all doesn't work the way I thought, as upon starting the last of my test VM's it failed to start citing insufficient host RAM.

So is it possible to "thin provision" RAM between guests for its more efficient use, or am I just being silly? :p
 
I'm sure you found this out, but to my recollection, the only vm that I have that has a memory reservation is the one where I passed through my raid controller. I believe any vms that has something pass-through will auto set the memory reservation to the ram size of the vm.

Don't know anything about thin provisioning though, sorry.
 
You're looking in the wrong place. By default vSphere doesn't reserve the RAM you assign to the VM. What you are seeing is the setting that defines the amount of RAM the guest OS sees and has access to. You CAN reserve some or all of the RAM for the VM..but that's not the default.
 
NetJunkie said:
You're looking in the wrong place. By default vSphere doesn't reserve the RAM you assign to the VM. What you are seeing is the setting that defines the amount of RAM the guest OS sees and has access to. You CAN reserve some or all of the RAM for the VM..but that's not the default.
Click to expand...

It was for me.

Dreamnid, got it right above.

If you pass through a device using Direct I/O it will reserve the RAM assigned to the guest in its entirety, apparently.

I have devices passed through to all my guests.

Makes sense though if you think of it, as you don't want to accidentally give up RAM that belongs to the passed through device and have it stop working.
 
To add on to the conversation, I believe it is recommended not to specify the memory reservation as it will probably hurt than help. As you saw though, pass-through is a different animal.

For normal vms (ie. no pass-through) and no memory reservations, you should be able to start vms that exceeds the hosts ram limit to a certain degree. You could try this by creating a vm with no passthrough. I'm assuming that you don't have all the host memory reserved.
 
Yes..if you use VMDirectPath you have to reserve the entire memory amount. But...that's not used often at all so I don't consider that a default.

We do memory reservations in some cases..such as Tier 1 VMs that need guaranteed levels of performance.
 
Zarathustra[H];1039168039 said:
It was for me.

Dreamnid, got it right above.

If you pass through a device using Direct I/O it will reserve the RAM assigned to the guest in its entirety, apparently.

I have devices passed through to all my guests.

Makes sense though if you think of it, as you don't want to accidentally give up RAM that belongs to the passed through device and have it stop working.
Click to expand...

No, that'd be the limit I expect. We've never set a 100% reservation by default on any release. Ever. ~shudder~
 
Why would it automatically reserve the memory if passing thorough devices? It's not like the memory will be removed from a live guest if the host runs out occurs, it should dedupe first then swap.

I have an HBA passed through to my NAS vm and I don't see a reservation.
 
shiznit said:
Why would it automatically reserve the memory if passing thorough devices? It's not like the memory will be removed from a live guest if the host runs out occurs, it should dedupe first then swap.

I have an HBA passed through to my NAS vm and I don't see a reservation.
Click to expand...

It has to do with devices needing access to memory. If you pass a device through you're reserving memory..even if you don't realize it. Go look at the VM settings for memory reservation.
 
NetJunkie said:
It has to do with devices needing access to memory. If you pass a device through you're reserving memory..even if you don't realize it. Go look at the VM settings for memory reservation.
Click to expand...

I'll check again as soon as I get home. But I'm still wondering why the memory needs to be reserved, its not like the hypervisor steals back memory from a live guest during contention, it dedupes, baloons, and finally starts swapping to disk, but the guest OS still thinks it has the full amount it was allocated. Even if the VT-d device had memory exclusive to it, no matter what wouldn't there at least always be a pointer to a copy of the pages it was using?
 
shiznit said:
I'll check again as soon as I get home. But I'm still wondering why the memory needs to be reserved, its not like the hypervisor steals back memory from a live guest during contention, it dedupes, baloons, and finally starts swapping to disk, but the guest OS still thinks it has the full amount it was allocated. Even if the VT-d device had memory exclusive to it, no matter what wouldn't there at least always be a pointer to a copy of the pages it was using?
Click to expand...

Has to do with mapping out the exact memory ranges that the VM expects and then keeping them that way.
 
lopoetve said:
Has to do with mapping out the exact memory ranges that the VM expects and then keeping them that way.
Click to expand...

Yep. Exactly.

Bear with me here, as I am a little bit of a layman, but it works something like this:

Whenever a DMA device is installed in a computer, programs interact with the device by writing to specific memory addresses that belong to that device, typically through drivers.

When programming in old school lower level languages (C, C++, etc.) it can get very complex and tedious as the programmer has to manually keep track of all these memory address ranges. Modern higher level programming languages (Java, C#, etc.) automate this process to a certain extent, but do so at the cost of some performance.

Either way, so to programs running on the computer, an attached device is just a certain memory range, otherwise indistinguishable from other memory ranges.

If you start messing with this, and dynamically releasing and acquiring RAM, it's not difficult to see how you could quickly wind up in a bit of trouble.

It's probably not insurmountable to code around, so that you don't need to reserve the memory ranges, but that would be one hell of a project.

For maximum compatibility and stability it makes perfect sense to me why VMWare would choose to instead reserve the ram for Direct I/O Mapped VM's.
 
NetJunkie said:
Yes..if you use VMDirectPath you have to reserve the entire memory amount. But...that's not used often at all so I don't consider that a default.
Click to expand...

That's interesting.

Considering how much attention VT-d/IOMMU and Direct I/O Mapping get on these forums, I would have thought it was a lot more common than that.
 
Zarathustra[H];1039171092 said:
That's interesting.

Considering how much attention VT-d/IOMMU and Direct I/O Mapping get on these forums, I would have thought it was a lot more common than that.
Click to expand...
Only because of homebrew vitual NAS and routers, I have never seen it used in production, other than gpu for vdi, but that will change with Kepler.
 
shiznit said:
Only because of homebrew vitual NAS and routers, I have never seen it used in production, other than gpu for vdi, but that will change with Kepler.
Click to expand...

Yep. And this is why people have so much trouble with it in here. Just not many actual production use cases so it doesn't get a lot of attention.
 
You must log in or register to reply here.
Back
Top