ESXi VMs so slow it's almost unusable

TeeJayHoward

Limpness Supreme
2FA
Joined
Feb 8, 2005
Messages
13,421
I've noticed some issues copying data to my Windows Server 2008 R2 VM recently. I seem to get a max of about 6MB/s from my laptop to the VM, across a gigabit network. This works just fine for copying most of my homework over, as it's just a bunch of text files. However, I'm re-encoding a lot of video lately, and it's obviously unacceptable for copying several gigs of data. At the time, I didn't want to screw with it, so I just decided to go back to my old Solaris-on-bare-metal setup. Unfortunately, for some odd reason, Solaris 11 doesn't want to read my NTFS-formatted drives. So, I figured, I'd just toss Solaris 11 in a VM, copy the data across ESXi's virtual network, and then flip back.

Holy cow. The virtual network is the slowest thing I have ever seen. Over 48 hours, I've managed to copy 61.3 gig of data. Not only that, but while attempting to copy the data over, remote desktop becomes so slow that it's nigh unusable. It took me about 5 minutes from the time I hit "Connect" to the time the desktop was visible. It takes 30 seconds between when I click on a tab in the vSphere Client and when the tab finally switches over. Unfortunately, VMWare hates us Mac users, so I have to use a VM (hosted on the ESXi box) to use vSphere. I've got another 2.7TB of data to copy over. At this rate, it will take a little under 100 days to do it.

Can anyone offer any advice or troubleshooting procedures I should be going through? My all-in-one configuration is as follows:

Physical: e3-1230, 8GB, 3x3TB, 1x1TB (datastore), host OS on USB drive
VM1: Server 2K8 Std, 2 cores/4GB allocated, used for vSphere Client
VM2: Server 2K8R2 Std, 2 cores/4GB allocated, used for file sharing via NFS/SMB, 2x3TB connected
VM3, Solaris 11 11/11, 8 cores/8GB allocated, destination, 1x3TB connected

I realize I'm oversubscribed, going to try and fix that now. I'm hoping that I've just chewed up all my RAM. Should take about 30 minutes with the slow speed of things. Until I do get that fixed, can anyone offer any other suggestions I could try?
 
Okay, I shut down VM1, changed the file server to 2CPU/2GB, and changed Solaris to 4CPU/4GB. I'm now reading data from the file server at a massive 1-4MB/s. It's a significant improvement, but still nowhere near the hardware's capability. I ran CrystalDiskMark on the file server, and got over 100MB/s seq. read, so from what I can tell, that VM is operating at near-native levels. My remote desktop issue seems to have disappeared as well. Running a "time dd if=/dev/zero of=/pool1/delete.me bs=1024 count=1073741" on the Solaris box now to see if that drive's operating correctly.

edit: Results:
Code:
root@Solaris:/# time dd if=/dev/zero of=/pool1/delete.me bs=1024 count=1073741
1073741+0 records in
1073741+0 records out

real    0m27.904s
user    0m0.350s
sys     0m6.296s
root@Solaris:/# ls -lh /pool1
total 2080790
-rw-r--r--   1 root     root        1.0G Jul  1 16:56 delete.me

so, 1GB in 28 seconds = 36MB/s. Much slower than I'd like, but still several times faster than the network copy. What's going on?

edit2: 1GB might be a bit too small to get a good reading. Bumped it up to 10GB.
Code:
root@Solaris:/# time dd if=/dev/zero of=/pool1/delete.me bs=1024 count=10737418
10737418+0 records in
10737418+0 records out

real    1m32.270s
user    0m3.744s
sys     1m9.915s
root@Solaris:/#ls -lh /pool1
-rw-r--r--   1 root     root         10G Jul  1 17:00 delete.me
92 seconds for 10GB, or a bit over 110MB/s. Much better.

So, I've verified that the Solaris box can write at over 100MB/s, and the Windows box can read at over 100MB/s. Why can I only transfer a file at 4MB/s?

edit3: What the deuce is going on with this machine? It's now transferring at over 50MB/s. Still slower than I'd like, but faster than I've seen in a long time.
Code:
                 extended device statistics                    tty         cpu
device    r/s    w/s   Mr/s   Mw/s wait actv  svc_t  %w  %b  tin tout  us sy wt id
sd0       0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0    0  563   9 14  0 77
sd1       0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0 
sd2       4.0    0.0    0.0    0.0  0.0  0.0    0.5   0   0 
nfs1   1940.1    0.0   54.2    0.0  0.1  2.0    1.0   4  66
 
Last edited:
Any reason you are throwing so many cores at these VMs? Unless your VMs need mondo CPU cycles, you are likely causing a problem, not solving one. Keep in mind that if you assign N cores to a VM, esxi needs to have all N cores available before it will schedule the VM. None of my VMs are that CPU intensive, so I just assign them one core each and have never come close to needing more. I have the same processor you do, and my whole host rarely goes much above 25% CPU usage (and that is with my wife's XP telecommuter VM, a ubuntu mail/web/icinga server and a pbxinaflash PBX VM.)
 
what type of network interface did you get the vm's?
Whatever the default is. Intel Pro/1000, I think.

Any reason you are throwing so many cores at these VMs? Unless your VMs need mondo CPU cycles, you are likely causing a problem, not solving one. Keep in mind that if you assign N cores to a VM, esxi needs to have all N cores available before it will schedule the VM. None of my VMs are that CPU intensive, so I just assign them one core each and have never come close to needing more. I have the same processor you do, and my whole host rarely goes much above 25% CPU usage (and that is with my wife's XP telecommuter VM, a ubuntu mail/web/icinga server and a pbxinaflash PBX VM.)
Honestly, it's just a lack of knowledge on my part. The Application Server's the only one that's actually running any real programs, but I keep hearing about how hardware-hungry ZFS is. I figured, Hey, I've got the cores, why not give 'em out?
 
Whatever the default is. Intel Pro/1000, I think.


Honestly, it's just a lack of knowledge on my part. The Application Server's the only one that's actually running any real programs, but I keep hearing about how hardware-hungry ZFS is. I figured, Hey, I've got the cores, why not give 'em out?


Why not use the VMX3? Also, are you configuring the hdd's as ide or sas or what?
 
Well, try what I suggested. Change each VM to one core each and see. Your performance really blows - and you have the same CPU I do, so I'm suspicious of the multi-core config.
 
Keep in mind that if you assign N cores to a VM, esxi needs to have all N cores available before it will schedule the VM.

That was true on 3.5 but the CPU Scheduler was relaxed in vSphere 4 a little and then even more in vSphere 5 to allow a multiple vCPU VMs to not have to wait for the same number of cores before it can get pCPU time. But I do agree that the OP has drastically oversubscribed his CPU. He could get them all down to 1 vCPU no problem.

However, that may not be the root issue.

OP, since you're oversubscribing your RAM your host may be swapping like crazy which is causing the slow disk issues.

SSH into your host and run "esxtop" then press the "m" key. Do you see any values under "SWCUR," "SWTGT," "SWR/s," and "SWW/s?" If so, that means you're swapping your VM's memory to disk. If you're not familiar with activating and accessing the command line, go to the Performance tab on your host, switch the view to "Memory," and see what appears under "Swap used."

Since you're seeing better performance after trimming back the RAM on a VM, I suspect this might be your problem.
 
Last edited:
Why not use the VMX3? Also, are you configuring the hdd's as ide or sas or what?
I guess I can swap the NIC over to VMXNET 3. It's currently set to "E1000". What are the advantages of doing that? I'll definitely give it a shot if I encounter any additional issues copying stuff. As for the HDDs:

Hard disk 3: Mapped Raw LUN
Physical LUN and Datastore Mapping File are on the local datastore. It's a .vmdk for a local SATA drive.
Virtual Device Node: SCSI (0:2) Hard disk 3
Compatibility Mode: Physical


But I do agree that the OP has drastically oversubscribed his CPU. He could get them all down to 1 vCPU no problem.
I'll do that next time I shut down one of the VMs.

OP, since you're oversubscribing your RAM your host may be swapping like crazy which is causing the slow disk issues.

SSH into your host and run "esxtop" then press the "m" key. Do you see any values under "SWCUR," "SWTGT," "SWR/s," and "SWW/s?" If so, that means you're swapping your VM's memory to disk. If you're not familiar with activating and accessing the command line, go to the Performance tab on your host, switch the view to "Memory," and see what appears under "Swap used."

Since you're seeing better performance after trimming back the RAM on a VM, I suspect this might be your problem.
I'm familiar with top, but not esxtop. I gave it a shot... All that's showing up is a ton of numbers. "M" doesn't do anything. Example output below:
Code:
"0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.60","0.20","0.40","0.01","0.00","9.25","0.01","9.26","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","0","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.99","0.20","0.80","0.01","0.00","5.61","0.01","5.62","0.00","31","0","0","0","0.00","0.00","28.43","0.40","27.63","0.00","5.86","3.76","0.00","3.76","0.00","28.23","0.40","27.44","0.00","5.66","3.76","0.00","3.76","0.06","31","32","0","0","0.00","0.00","28.43","0.40","27.63","0.00","5.86","3.76","0.00","3.76","0.00","28.43","0.40","27.63","0.00","5.86","3.76","0.00","3.76","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","0","0","0","0.00","0.00","482.52","482.52","0.00","56.75","0.00","0.47","0.00","0.48","0.00","482.52","482.52","0.00","56.75","0.00","0.47","0.00","0.48","0.00","31","32","0","0","0.00","0.00","482.52","482.52","0.00","56.75","0.00","0.47","0.00","0.48","0.00","482.52","482.52","0.00","56.75","0.00","0.47","0.00","0.48","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","0","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","31","32","0","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0","0.00","0.00","0.00","0.00","0.00","0.00","0","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","6.56","0.08","6.96","0.00","0.00","0.00","4.77","0.07","5.17","0.00","0.00","0.00","2430.27","461.48","22907.47","14.68","0.00","0.00","22909.26","13.99","42655.82","483.04","0.00","48.10","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","471.58","5.00","0.00","0.00","278.93","191.06","0.00","0.00","1.59","0.00","12.72","1.00","0.00","1.59","0.00","1.59","0.00","6.36","3.18","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","2361.09","0.00","466.01","163.22","357.86","210.54","364.22","234.00","364.42","200.80","49.50","0.00","12.72","0.00","7.95","1.59","3.18","17.69","0.00","6.36","63.82","0.00","8.15","3.18","1.59","17.50","1.59","20.68","0.00","11.13","3.78","0.00","0.00","0.20","0.00","0.40","1.59","0.00","1.59","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","83.06","16.94","0.00","89.70","0.20","0.00","0.00","0.00","0.00","0.00","0.00","0.00","10.11","5.53","94.47","0.00","88.67","0.20","0.00","0.00","0.00","0.00","0.00","0.00","0.20","10.94","66.76","33.24","0.00","84.03","1.59","1.61","1.39","0.60","0.20","0.40","0.00","0.20","9.99","13.14","86.86","0.00","86.03","1.46","1.59","1.19","0.80","0.33","0.00","0.20","0.20","8.20","70.94","29.06","0.00","87.10","0.40","0.40","0.80","0.20","0.20","0.20","0.20","0.00","10.51","15.39","84.61","0.00","92.21","0.60","0.40","0.64","0.20","0.10","0.00","0.00","0.00","5.85","75.60","24.40","0.00","89.15","0.00","0.60","0.20","0.00","0.20","0.20","0.00","0.00","9.66","9.64","90.36","0.00","93.84","0.00","0.60","0.20","0.00","0.00","0.20","0.00","0.00","5.17"

Little bit ridiculous, and not very useful. I do have a value under "swap used", listed below, but the line on the chart is sitting at 0. I think I can safely ignore the 11M swap that's being used. Active memory is less than Granted memory right now, even if it is somewhat close to the line. The oversubscription was most likely my issue.

The Summary tab shows that I'm using 7372MB of my 8169.72MB memory. I've got 4GB allocated to one VM, and 2GB allocated to another. I can only assume that the extra 1GB is being used by the host OS.

It's odd... I love activity monitor / perfmon when working on OSX/Windows boxes. I didn't even think to see if ESXi had anything similar.
 
Last edited:
Run this during your next copy:

esxtop -a -b -d 5 -n 60 > /tmp/performance.csv

(syntax off the top of my head). Upload that. We'll pull it apart.
 
Run this during your next copy:

esxtop -a -b -d 5 -n 60 > /tmp/performance.csv

(syntax off the top of my head). Upload that. We'll pull it apart.
I'm currently transferring data, so I ran the above command for a couple seconds, then Ctrl+C'd out.
Code:
Reiko:~ tim$ ssh 192.168.0.200
Password: 
The time and date of this login have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
~ $ su -
Password: 
~ # esxtop -a -b -d 5 -n 60 > /tmp/performance.csv
~ # cat /tmp/performance.csv 
<SNIPPED, JUST CHECKING!>
~ # exit
~ $ exit
Connection to 192.168.0.200 closed.
Reiko:~ tim$ scp root@192.168.0.200:/tmp/performance.csv ./
Password: 
performance.csv                               100% 1548KB 387.1KB/s   00:04
I'm registering for a rapidshare account now so that I can get the file to ya. Taking 'em forever to send me the authentication code.
 
"That was true on 3.5 but the CPU Scheduler was relaxed in vSphere 4 a little and then even more in vSphere 5 to allow a multiple vCPU VMs to not have to wait for the same number of cores before it can get pCPU time."

Ah, nice to know, thanks.
 
I'm currently transferring data, so I ran the above command for a couple seconds, then Ctrl+C'd out.
Code:
Reiko:~ tim$ ssh 192.168.0.200
Password: 
The time and date of this login have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
~ $ su -
Password: 
~ # esxtop -a -b -d 5 -n 60 > /tmp/performance.csv
~ # cat /tmp/performance.csv 
<SNIPPED, JUST CHECKING!>
~ # exit
~ $ exit
Connection to 192.168.0.200 closed.
Reiko:~ tim$ scp root@192.168.0.200:/tmp/performance.csv ./
Password: 
performance.csv                               100% 1548KB 387.1KB/s   00:04
I'm registering for a rapidshare account now so that I can get the file to ya. Taking 'em forever to send me the authentication code.

No no no - let that run the whole time. It'll go for 5 minutes by design (based on the parameters I gave it). Will need that for valid datas.
 
"That was true on 3.5 but the CPU Scheduler was relaxed in vSphere 4 a little and then even more in vSphere 5 to allow a multiple vCPU VMs to not have to wait for the same number of cores before it can get pCPU time."

Ah, nice to know, thanks.

*ish.

We can only flex the cpu so far before we HAVE to catch up the other threads - result, stinky performance. Otherwise, the guest would die horribly.
 
No no no - let that run the whole time. It'll go for 5 minutes by design (based on the parameters I gave it). Will need that for valid datas.
Okay, done. New file is located here.

Am I correct in assuming that Solaris 11 would benefit from more RAM in this situation?
Code:
root@Solaris:~# nice -10 echo "::memstat"|mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     270447              1056   26%
ZFS File Data              587929              2296   56%
Anon                        32718               127    3%
Exec and libs                1085                 4    0%
Page cache                 129399               505   12%
Free (cachelist)            24696                96    2%
Free (freelist)               140                 0    0%

Total                     1046414              4087
Physical                  1046413              4087
 
Last edited:
*ish.

We can only flex the cpu so far before we HAVE to catch up the other threads - result, stinky performance. Otherwise, the guest would die horribly.

Yes. The rule of thumb to not give VMs more vCPUs than absolutely necessary still applies. The performance hit just isn't as great anymore because of the changes to the scheduler, but that's still no reason to give out more vCPUs than needed especially 8 in the OP's case!
 
Yes. The rule of thumb to not give VMs more vCPUs than absolutely necessary still applies. The performance hit just isn't as great anymore because of the changes to the scheduler, but that's still no reason to give out more vCPUs than needed especially 8 in the OP's case!

And, if you go too far, it'll eventually kaputt itself the same old way. It's just a bit more flexible :)

Nahalem works better for it too, based on some improvements in that architecture.

I'll look at the file once I get to work (OP).


btw, did I ever email you back (Child of Wonder)? I know I intended to, but this summer's been insane.
 
And, if you go too far, it'll eventually kaputt itself the same old way. It's just a bit more flexible :)

Nahalem works better for it too, based on some improvements in that architecture.

I'll look at the file once I get to work (OP).


btw, did I ever email you back (Child of Wonder)? I know I intended to, but this summer's been insane.

Yeah, you did. Will there be a Colorado left by the time we get there???
 
I went to the BOF @ USENIX LISA last year for VMware and there were LOTS of people still annoyed about multi-vCPU performance in 4 and 5. By far and away the biggest complaint, but the VMware Engineers basically said they've done nearly all they can without CPU architecture changes.

>1 vCPU provides rapidly diminishing returns and can and often does slow the system down if it doesn't really need the treads. Of course this can be mitigated by setting affinity or not oversubscribing vCPUs, but you lose some of the benefits to going virtual.
 
Back
Top