Web server slow on ESXi

[mrgstiffler]

We have have a vm running Apache on Fedora and have noticed that if the sites on it haven't been accessed in awhile, they are extremely slow for the first 20-30 page views and then speed up. It's the same thing on 2 other VMs on the ESXi server, but not on a physical server. The server is a Dual Xeon X5460 (Quad-core 3.2GHz) with 16GB RAM. Running latest ESXi 4.0 patch.

 

[danswartz]

How much ram did you assign to the vm? Sounds like the file cache has been emptied and needs to be reloaded as sites are viewed?

 

[mrgstiffler]

How much ram did you assign to the vm? Sounds like the file cache has been emptied and needs to be reloaded as sites are viewed?

4GB for the VM. I've tried playing around with resource reservation and shares, but it hasn't made a difference.

 

[infamus]

what kind of storage is being used?

 

[mrgstiffler]

what kind of storage is being used?

We have 2 datastores. 1 on a 4 or 5 drive SAS RAID-5, and 1 on a Fiber Channel array. I've tried storing the hard drive image on both and still the same problem.

 

[Database]

vmware tools loaded I assume?

 

[mrgstiffler]

vmware tools loaded I assume?

No VMWare Tools installed. It's a server that's only accessed via SSH, so there wasn't any need for the tools.

 

[geiger]

The tools includes more than the interface plugins, you get many drivers that help the hypervisor communicate with the VM.

 

[Child of Wonder]

The tools includes more than the interface plugins, you get many drivers that help the hypervisor communicate with the VM.

There are also improved network adapter drivers (you should be using VMXNET3) and the memory balloon driver.

ALWAYS install VMware Tools if you can.

I would also disable all reservations and limits you've set on this VM and set the shares to Normal. Are there any snapshots of the VM? If so, delete them. What does the IO load on the datastore look like? Average latency high (over 50ms)?

 

[mrgstiffler]

The tools includes more than the interface plugins, you get many drivers that help the hypervisor communicate with the VM.

There are also improved network adapter drivers (you should be using VMXNET3) and the memory balloon driver.

ALWAYS install VMware Tools if you can.

I would also disable all reservations and limits you've set on this VM and set the shares to Normal. Are there any snapshots of the VM? If so, delete them. What does the IO load on the datastore look like? Average latency high (over 50ms)?

Very good to know, thanks. I'll see if I can get the tools installs.There are no snapshots of the machine. IO load generally hovers between 6MB/s-12MB/s. CPU load doesn't even come close to 33%. The biggest problem is memory usage right now. We're at around 14GB out of 16GB used. I'm going to push for an upgrade, I think we have the funds.

The network adapters for all of our VMs are set to e1000. Is that somewhat of a problem?

 

[Child of Wonder]

Very good to know, thanks. I'll see if I can get the tools installs.There are no snapshots of the machine. IO load generally hovers between 6MB/s-12MB/s. CPU load doesn't even come close to 33%. The biggest problem is memory usage right now. We're at around 14GB out of 16GB used. I'm going to push for an upgrade, I think we have the funds.

The network adapters for all of our VMs are set to e1000. Is that somewhat of a problem?

Not a problem but you'll have less CPU overheard and better network performance with VMXNET3.

Go to the performance tab for the host and look at Memory - Swap rate. Is your host swapping? If so then you either have a memory limit set on a VM or your host is out of RAM.

As for IO, look at the read and write requests and latency for your VM. See how many IOPs it's pushing and how long it's taking for them to be processed.

 

[mrgstiffler]

Not a problem but you'll have less CPU overheard and better network performance with VMXNET3.

Go to the performance tab for the host and look at Memory - Swap rate. Is your host swapping? If so then you either have a memory limit set on a VM or your host is out of RAM.

As for IO, look at the read and write requests and latency for your VM. See how many IOPs it's pushing and how long it's taking for them to be processed.

I went to performance, chose memory and adjusted the chart to show 'Swap in rate" and "Swap out rate". It should an average of 9KBps In and 12KBps Out.

I looked at the read/write requests for the VM and saw some read spikes of about 500, while the average is around 13.

 

[ChRoNo16]

How far from the physical machine is the arrays that they are stored on, as well as whats between them for a fiber switch. sorry for so many questions, but what else runs on the arrays, anything else using them?

 

[mrgstiffler]

How far from the physical machine is the arrays that they are stored on, as well as whats between them for a fiber switch. sorry for so many questions, but what else runs on the arrays, anything else using them?

Actually, scratch the fiber channel, that box is connected to an external SAS array on a PowerVault MD1000.

 

[moose517]

Wow after reading this thread tonight im gonna have to install vmeare tools. Ibmever bothered before but after reading whats been said here i need to.

 

[defuseme2k]

I noticed you said there was some swapping... do you have a memory limit set or is the host nearly out of memory to use? I'd expect poor performance across the board in that scenario... odd it would just be one VM. Memory limits can cause all kinds of issues ESPECIALLY without the VMware tools installed. If memory can't be reclaimed via the balloon driver (part of the tools) then you'll see vmkernel swapping which is bad juju and can cause weird behavior because the memory swapped could be critical to the virtual machine. Go to edit settings on the VM -> resources tab and both the memory and CPU should be unlimited.

You should never use a memory limit, I can think of no justification except a case of intentionally making a VM run like crap . I've seen them be in place unknowingly post p2v etc, but I've seen admins try to "prove" a VM doesn't need x amount of memory to some app owner. Even then a limit is a bad thing to do, because swapping could disprove your point and make you look like an idiot in the process .

And always install the tools, they aren't just there to give you a pretty icon in the tray... they are absolutely necessary to have proper drivers on the VM and you will see performance gained with them installed.

Just in case you do have a limit set: remove it, then shut the VM down so it powers off. Then power it back on to get it undone completely.

 

[Child of Wonder]

I noticed you said there was some swapping... do you have a memory limit set or is the host nearly out of memory to use? I'd expect poor performance across the board in that scenario... odd it would just be one VM. Memory limits can cause all kinds of issues ESPECIALLY without the VMware tools installed. If memory can't be reclaimed via the balloon driver (part of the tools) then you'll see vmkernel swapping which is bad juju and can cause weird behavior because the memory swapped could be critical to the virtual machine. Go to edit settings on the VM -> resources tab and both the memory and CPU should be unlimited.

You should never use a memory limit, I can think of no justification except a case of intentionally making a VM run like crap . I've seen them be in place unknowingly post p2v etc, but I've seen admins try to "prove" a VM doesn't need x amount of memory to some app owner. Even then a limit is a bad thing to do, because swapping could disprove your point and make you look like an idiot in the process .

And always install the tools, they aren't just there to give you a pretty icon in the tray... they are absolutely necessary to have proper drivers on the VM and you will see performance gained with them installed.

Just in case you do have a limit set: remove it, then shut the VM down so it powers off. Then power it back on to get it undone completely.

Couldn't agree more. I've seen memory limits in place at several client sites and they are NEVER a good idea.

Once you remove the limit you can also VMotion the VM to another host to make the change take affect.

 

[defuseme2k]

Once you remove the limit you can also VMotion the VM to another host to make the change take affect.

I've never heard that, but I'll try it. Good tip.